[PATCH v2 00/39] Enable build of full Xen for RISC-V

[PATCH v2 00/39] Enable build of full Xen for RISC-V

[Oleksii Kurochko]

This patch series performs all of the additions necessary to drop the
build overrides for RISCV and enable the full Xen build. Except in cases
where compatibile implementations already exist (e.g. atomic.h and
bitops.h), the newly added definitions are simple.

The patch series is based on the following patch series:
- [PATCH v2] xen: remove <asm/delay.h> [1]
- [PATCH v3 00/14] Introduce generic headers [2]

[1] https://lore.kernel.org/xen-devel/3d55bce44bd6ab9973cbe0ea2fc136cc44d35df2.1698759633.git.oleksii.kurochko@gmail.com/
[2] https://lore.kernel.org/xen-devel/cover.1700221559.git.oleksii.kurochko@gmail.com/

---
Changes in V2:
  - Drop the following patches as they are the part of [2]:
      [PATCH v1 06/57] xen/riscv: introduce paging.h
      [PATCH v1 08/57] xen/riscv: introduce asm/device.h
      [PATCH v1 10/57] xen/riscv: introduce asm/grant_table.h
      [PATCH v1 12/57] xen/riscv: introduce asm/hypercall.h
      [PATCH v1 13/57] xen/riscv: introduce asm/iocap.h
      [PATCH v1 15/57] xen/riscv: introduce asm/mem_access.h
      [PATCH v1 18/57] xen/riscv: introduce asm/random.h
      [PATCH v1 21/57] xen/riscv: introduce asm/xenoprof.h
      [PATCH v1 24/57] xen/riscv: introduce asm/percpu.h
      [PATCH v1 29/57] xen/riscv: introduce asm/hardirq.h
      [PATCH v1 33/57] xen/riscv: introduce asm/altp2m.h
      [PATCH v1 38/57] xen/riscv: introduce asm/monitor.h
      [PATCH v1 39/57] xen/riscv: introduce asm/numa.h
      [PATCH v1 42/57] xen/riscv: introduce asm/softirq.h
  - xen/lib.h in most of the cases were changed to xen/bug.h as
    mostly functionilty of bug.h is used.
  - align arch-riscv.h with Arm's version of it.
  - change the Author of commit with introduction of asm/atomic.h.
  - update some definition from spinlock.h.
  - code style changes.
---

Bobby Eshleman (1):
  xen/riscv: introduce asm/atomic.h

Oleksii Kurochko (38):
  xen/riscv: disable unnecessary configs
  xen/riscv: use some asm-generic headers
  xen/riscv:introduce asm/byteorder.h
  xen/riscv: add public arch-riscv.h
  xen/riscv: introduce spinlock.h
  xen/riscv: introduce fence.h
  xen/riscv: introduce arch-riscv/hvm/save.h
  xen/riscv: introduce asm/cpufeature.h
  xen/riscv: introduce asm/guest_atomics.h
  xen/riscv: introduce asm/iommu.h
  xen/riscv: introduce asm/nospec.h
  xen/riscv: introduce asm/setup.h
  xen/riscv: introduce asm/system.h
  xen/riscv: introduce bitops.h
  xen/riscv: introduce flushtlb.h
  xen/riscv: introduce asm/smp.h
  xen/riscv: introduce cmpxchg.h
  xen/riscv: introduce asm/io.h
  xen/riscv: define bug frame tables in xen.lds.S
  xen/riscv: introduce bit operations
  xen/riscv: introduce asm/domain.h
  xen/riscv: introduce asm/guest_access.h
  xen/riscv: introduce asm/irq.h
  xen/riscv: introduce asm/p2m.h
  xen/riscv: introduce asm/regs.h
  xen/riscv: introduce asm/time.h
  xen/riscv: introduce asm/event.h
  xen/riscv: add definition of __read_mostly
  xen/riscv: define an address of frame table
  xen/riscv: add required things to asm/current.h
  xen/riscv: add minimal stuff to asm/page.h to build full Xen
  xen/riscv: add minimal stuff to asm/processor.h to build full Xen
  xen: add RISCV support for pmu.h
  xen: add necessary headers to common to build full Xen for RISC-V
  xen/riscv: add minimal stuff to asm/mm.h to build full Xen
  xen/rirscv: add minimal amount of stubs to build full Xen
  xen/riscv: enable full Xen build
  xen: fix compilation issue of serial.c

 automation/gitlab-ci/build.yaml            | 122 ++++++
 automation/scripts/build                   |   4 +-
 xen/arch/riscv/Makefile                    |  18 +-
 xen/arch/riscv/arch.mk                     |   4 -
 xen/arch/riscv/configs/tiny64_defconfig    |  19 +-
 xen/arch/riscv/early_printk.c              | 168 --------
 xen/arch/riscv/include/asm/Makefile        |  14 +
 xen/arch/riscv/include/asm/atomic.h        | 375 ++++++++++++++++++
 xen/arch/riscv/include/asm/bitops.h        | 288 ++++++++++++++
 xen/arch/riscv/include/asm/byteorder.h     |  16 +
 xen/arch/riscv/include/asm/cache.h         |   2 +
 xen/arch/riscv/include/asm/cmpxchg.h       | 379 ++++++++++++++++++
 xen/arch/riscv/include/asm/config.h        |  21 +
 xen/arch/riscv/include/asm/cpufeature.h    |  22 ++
 xen/arch/riscv/include/asm/current.h       |  20 +
 xen/arch/riscv/include/asm/domain.h        |  43 +++
 xen/arch/riscv/include/asm/event.h         |  34 ++
 xen/arch/riscv/include/asm/fence.h         |  12 +
 xen/arch/riscv/include/asm/flushtlb.h      |  31 ++
 xen/arch/riscv/include/asm/guest_access.h  |  29 ++
 xen/arch/riscv/include/asm/guest_atomics.h |  48 +++
 xen/arch/riscv/include/asm/io.h            | 134 +++++++
 xen/arch/riscv/include/asm/iommu.h         |   7 +
 xen/arch/riscv/include/asm/irq.h           |  37 ++
 xen/arch/riscv/include/asm/mm.h            | 270 ++++++++++++-
 xen/arch/riscv/include/asm/nospec.h        |  25 ++
 xen/arch/riscv/include/asm/p2m.h           | 105 +++++
 xen/arch/riscv/include/asm/page.h          |  21 +
 xen/arch/riscv/include/asm/processor.h     |  15 +
 xen/arch/riscv/include/asm/regs.h          |  26 ++
 xen/arch/riscv/include/asm/setup.h         |   6 +
 xen/arch/riscv/include/asm/smp.h           |  23 ++
 xen/arch/riscv/include/asm/spinlock.h      |  15 +
 xen/arch/riscv/include/asm/system.h        |  79 ++++
 xen/arch/riscv/include/asm/time.h          |  19 +
 xen/arch/riscv/lib/Makefile                |   1 +
 xen/arch/riscv/lib/find_next_bit.c         | 278 ++++++++++++++
 xen/arch/riscv/mm.c                        |  52 ++-
 xen/arch/riscv/setup.c                     |   9 +-
 xen/arch/riscv/stubs.c                     | 426 +++++++++++++++++++++
 xen/arch/riscv/traps.c                     |  25 ++
 xen/arch/riscv/xen.lds.S                   |  10 +
 xen/drivers/char/serial.c                  |   2 +
 xen/include/public/arch-riscv.h            |  93 +++++
 xen/include/public/arch-riscv/hvm/save.h   |  20 +
 xen/include/public/hvm/save.h              |   2 +
 xen/include/public/pmu.h                   |   2 +
 xen/include/public/xen.h                   |   2 +
 xen/include/xen/domain.h                   |   1 +
 49 files changed, 3194 insertions(+), 180 deletions(-)
 create mode 100644 xen/arch/riscv/include/asm/Makefile
 create mode 100644 xen/arch/riscv/include/asm/atomic.h
 create mode 100644 xen/arch/riscv/include/asm/bitops.h
 create mode 100644 xen/arch/riscv/include/asm/byteorder.h
 create mode 100644 xen/arch/riscv/include/asm/cmpxchg.h
 create mode 100644 xen/arch/riscv/include/asm/cpufeature.h
 create mode 100644 xen/arch/riscv/include/asm/domain.h
 create mode 100644 xen/arch/riscv/include/asm/event.h
 create mode 100644 xen/arch/riscv/include/asm/fence.h
 create mode 100644 xen/arch/riscv/include/asm/flushtlb.h
 create mode 100644 xen/arch/riscv/include/asm/guest_access.h
 create mode 100644 xen/arch/riscv/include/asm/guest_atomics.h
 create mode 100644 xen/arch/riscv/include/asm/io.h
 create mode 100644 xen/arch/riscv/include/asm/iommu.h
 create mode 100644 xen/arch/riscv/include/asm/irq.h
 create mode 100644 xen/arch/riscv/include/asm/nospec.h
 create mode 100644 xen/arch/riscv/include/asm/p2m.h
 create mode 100644 xen/arch/riscv/include/asm/regs.h
 create mode 100644 xen/arch/riscv/include/asm/setup.h
 create mode 100644 xen/arch/riscv/include/asm/smp.h
 create mode 100644 xen/arch/riscv/include/asm/spinlock.h
 create mode 100644 xen/arch/riscv/include/asm/system.h
 create mode 100644 xen/arch/riscv/include/asm/time.h
 create mode 100644 xen/arch/riscv/lib/Makefile
 create mode 100644 xen/arch/riscv/lib/find_next_bit.c
 create mode 100644 xen/arch/riscv/stubs.c
 create mode 100644 xen/include/public/arch-riscv.h
 create mode 100644 xen/include/public/arch-riscv/hvm/save.h

-- 
2.42.0

[PATCH v2 01/39] xen/riscv: disable unnecessary configs

[Oleksii Kurochko]

The patch also fixes the build script as conf util expects
to have each config on separate line.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - update the commit message.
 - remove xen/arch/riscv/Kconfig changes.
---
 automation/gitlab-ci/build.yaml         | 122 ++++++++++++++++++++++++
 automation/scripts/build                |   4 +-
 xen/arch/riscv/configs/tiny64_defconfig |  18 ++++
 3 files changed, 142 insertions(+), 2 deletions(-)

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 32af30cced..f4ef81839a 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -522,6 +522,38 @@ archlinux-current-gcc-riscv64:
     CONTAINER: archlinux:current-riscv64
     KBUILD_DEFCONFIG: tiny64_defconfig
     HYPERVISOR_ONLY: y
+    EXTRA_XEN_CONFIG:
+      CONFIG_COVERAGE=n
+      CONFIG_GRANT_TABLE=n
+      CONFIG_SCHED_CREDIT=n
+      CONFIG_SCHED_CREDIT2=n
+      CONFIG_SCHED_RTDS=n
+      CONFIG_SCHED_NULL=n
+      CONFIG_SCHED_ARINC653=n
+      CONFIG_TRACEBUFFER=n
+      CONFIG_HYPFS=n
+      CONFIG_GRANT_TABLE=n
+      CONFIG_SPECULATIVE_HARDEN_ARRAY=n
+      CONFIG_ARGO=n
+      CONFIG_HYPFS_CONFIG=n
+      CONFIG_CORE_PARKING=n
+      CONFIG_DEBUG_TRACE=n
+      CONFIG_IOREQ_SERVER=n
+      CONFIG_CRASH_DEBUG=n
+      CONFIG_KEXEC=n
+      CONFIG_LIVEPATCH=n
+      CONFIG_MEM_ACCESS=n
+      CONFIG_NUMA=n
+      CONFIG_PERF_COUNTERS=n
+      CONFIG_HAS_PMAP=n
+      CONFIG_TRACEBUFFER=n
+      CONFIG_XENOPROF=n
+      CONFIG_COMPAT=n
+      CONFIG_COVERAGE=n
+      CONFIG_UBSAN=n
+      CONFIG_NEEDS_LIBELF=n
+      CONFIG_XSM=n
+
 
 archlinux-current-gcc-riscv64-debug:
   extends: .gcc-riscv64-cross-build-debug
@@ -529,6 +561,38 @@ archlinux-current-gcc-riscv64-debug:
     CONTAINER: archlinux:current-riscv64
     KBUILD_DEFCONFIG: tiny64_defconfig
     HYPERVISOR_ONLY: y
+    EXTRA_XEN_CONFIG:
+      CONFIG_COVERAGE=n
+      CONFIG_GRANT_TABLE=n
+      CONFIG_SCHED_CREDIT=n
+      CONFIG_SCHED_CREDIT2=n
+      CONFIG_SCHED_RTDS=n
+      CONFIG_SCHED_NULL=n
+      CONFIG_SCHED_ARINC653=n
+      CONFIG_TRACEBUFFER=n
+      CONFIG_HYPFS=n
+      CONFIG_GRANT_TABLE=n
+      CONFIG_SPECULATIVE_HARDEN_ARRAY=n
+      CONFIG_ARGO=n
+      CONFIG_HYPFS_CONFIG=n
+      CONFIG_CORE_PARKING=n
+      CONFIG_DEBUG_TRACE=n
+      CONFIG_IOREQ_SERVER=n
+      CONFIG_CRASH_DEBUG=n
+      CONFIG_KEXEC=n
+      CONFIG_LIVEPATCH=n
+      CONFIG_MEM_ACCESS=n
+      CONFIG_NUMA=n
+      CONFIG_PERF_COUNTERS=n
+      CONFIG_HAS_PMAP=n
+      CONFIG_TRACEBUFFER=n
+      CONFIG_XENOPROF=n
+      CONFIG_COMPAT=n
+      CONFIG_COVERAGE=n
+      CONFIG_UBSAN=n
+      CONFIG_NEEDS_LIBELF=n
+      CONFIG_XSM=n
+
 
 archlinux-current-gcc-riscv64-randconfig:
   extends: .gcc-riscv64-cross-build
@@ -538,6 +602,35 @@ archlinux-current-gcc-riscv64-randconfig:
     RANDCONFIG: y
     EXTRA_FIXED_RANDCONFIG:
       CONFIG_COVERAGE=n
+      CONFIG_GRANT_TABLE=n
+      CONFIG_SCHED_CREDIT=n
+      CONFIG_SCHED_CREDIT2=n
+      CONFIG_SCHED_RTDS=n
+      CONFIG_SCHED_NULL=n
+      CONFIG_SCHED_ARINC653=n
+      CONFIG_TRACEBUFFER=n
+      CONFIG_HYPFS=n
+      CONFIG_GRANT_TABLE=n
+      CONFIG_SPECULATIVE_HARDEN_ARRAY=n
+      CONFIG_ARGO=n
+      CONFIG_HYPFS_CONFIG=n
+      CONFIG_CORE_PARKING=n
+      CONFIG_DEBUG_TRACE=n
+      CONFIG_IOREQ_SERVER=n
+      CONFIG_CRASH_DEBUG=n
+      CONFIG_KEXEC=n
+      CONFIG_LIVEPATCH=n
+      CONFIG_MEM_ACCESS=n
+      CONFIG_NUMA=n
+      CONFIG_PERF_COUNTERS=n
+      CONFIG_HAS_PMAP=n
+      CONFIG_TRACEBUFFER=n
+      CONFIG_XENOPROF=n
+      CONFIG_COMPAT=n
+      CONFIG_COVERAGE=n
+      CONFIG_UBSAN=n
+      CONFIG_NEEDS_LIBELF=n
+      CONFIG_XSM=n
 
 archlinux-current-gcc-riscv64-debug-randconfig:
   extends: .gcc-riscv64-cross-build-debug
@@ -547,6 +640,35 @@ archlinux-current-gcc-riscv64-debug-randconfig:
     RANDCONFIG: y
     EXTRA_FIXED_RANDCONFIG:
       CONFIG_COVERAGE=n
+      CONFIG_GRANT_TABLE=n
+      CONFIG_SCHED_CREDIT=n
+      CONFIG_SCHED_CREDIT2=n
+      CONFIG_SCHED_RTDS=n
+      CONFIG_SCHED_NULL=n
+      CONFIG_SCHED_ARINC653=n
+      CONFIG_TRACEBUFFER=n
+      CONFIG_HYPFS=n
+      CONFIG_GRANT_TABLE=n
+      CONFIG_SPECULATIVE_HARDEN_ARRAY=n
+      CONFIG_ARGO=n
+      CONFIG_HYPFS_CONFIG=n
+      CONFIG_CORE_PARKING=n
+      CONFIG_DEBUG_TRACE=n
+      CONFIG_IOREQ_SERVER=n
+      CONFIG_CRASH_DEBUG=n
+      CONFIG_KEXEC=n
+      CONFIG_LIVEPATCH=n
+      CONFIG_MEM_ACCESS=n
+      CONFIG_NUMA=n
+      CONFIG_PERF_COUNTERS=n
+      CONFIG_HAS_PMAP=n
+      CONFIG_TRACEBUFFER=n
+      CONFIG_XENOPROF=n
+      CONFIG_COMPAT=n
+      CONFIG_COVERAGE=n
+      CONFIG_UBSAN=n
+      CONFIG_NEEDS_LIBELF=n
+      CONFIG_XSM=n
 
 # Power cross-build
 debian-bullseye-gcc-ppc64le:
diff --git a/automation/scripts/build b/automation/scripts/build
index b3c71fb6fb..88287b9c46 100755
--- a/automation/scripts/build
+++ b/automation/scripts/build
@@ -14,7 +14,7 @@ if [[ "${RANDCONFIG}" == "y" ]]; then
 
     # Append job-specific fixed configuration
     if [[ -n "${EXTRA_FIXED_RANDCONFIG}" ]]; then
-        echo "${EXTRA_FIXED_RANDCONFIG}" >> xen/tools/kconfig/allrandom.config
+        sed "s/ /\n/g" <<< "${EXTRA_FIXED_RANDCONFIG}" > xen/tools/kconfig/allrandom.config
     fi
 
     make -j$(nproc) -C xen KCONFIG_ALLCONFIG=tools/kconfig/allrandom.config randconfig
@@ -28,7 +28,7 @@ else
     echo "CONFIG_DEBUG=${debug}" >> xen/.config
 
     if [[ -n "${EXTRA_XEN_CONFIG}" ]]; then
-        echo "${EXTRA_XEN_CONFIG}" >> xen/.config
+        sed "s/ /\n/g" <<< "${EXTRA_XEN_CONFIG}" >> xen/.config
     fi
 
     make -j$(nproc) -C xen olddefconfig
diff --git a/xen/arch/riscv/configs/tiny64_defconfig b/xen/arch/riscv/configs/tiny64_defconfig
index 3c9a2ff941..ce37c5ae00 100644
--- a/xen/arch/riscv/configs/tiny64_defconfig
+++ b/xen/arch/riscv/configs/tiny64_defconfig
@@ -6,6 +6,24 @@
 # CONFIG_HYPFS is not set
 # CONFIG_GRANT_TABLE is not set
 # CONFIG_SPECULATIVE_HARDEN_ARRAY is not set
+# CONFIG_ARGO is not set
+# CONFIG_HYPFS_CONFIG is not set
+# CONFIG_CORE_PARKING is not set
+# CONFIG_DEBUG_TRACE is not set
+# CONFIG_IOREQ_SERVER is not set
+# CONFIG_CRASH_DEBUG is not setz
+# CONFIG_KEXEC is not set
+# CONFIG_LIVEPATCH is not set
+# CONFIG_MEM_ACCESS is not set
+# CONFIG_NUMA is not set
+# CONFIG_PERF_COUNTERS is not set
+# CONFIG_HAS_PMAP is not set
+# CONFIG_TRACEBUFFER is not set
+# CONFIG_XENOPROF is not set
+# CONFIG_COMPAT is not set
+# CONFIG_COVERAGE is not set
+# CONFIG_UBSAN is not set
+# CONFIG_NEEDS_LIBELF is not set
 
 CONFIG_RISCV_64=y
 CONFIG_DEBUG=y
-- 
2.42.0

[PATCH v2 02/39] xen/riscv: use some asm-generic headers

[Oleksii Kurochko]

Some headers are the same as asm-generic verions of them
so use them instead of arch-specific headers.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - New commit introduced in V2.
---
 xen/arch/riscv/include/asm/Makefile | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/Makefile

diff --git a/xen/arch/riscv/include/asm/Makefile b/xen/arch/riscv/include/asm/Makefile
new file mode 100644
index 0000000000..4da0c70cc0
--- /dev/null
+++ b/xen/arch/riscv/include/asm/Makefile
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: GPL-2.0-only
+generic-y += altp2m.h
+generic-y += device.h
+generic-y += div64.h
+generic-y += hardirq.h
+generic-y += hypercall.h
+generic-y += iocap.h
+generic-y += monitor.h
+generic-y += numa.h
+generic-y += paging.h
+generic-y += percpu.h
+generic-y += random.h
+generic-y += softirq.h
+generic-y += vm_event.h
-- 
2.42.0

[PATCH v2 03/39] xen/riscv:introduce asm/byteorder.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/byteorder.h | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/byteorder.h

diff --git a/xen/arch/riscv/include/asm/byteorder.h b/xen/arch/riscv/include/asm/byteorder.h
new file mode 100644
index 0000000000..320a03c88f
--- /dev/null
+++ b/xen/arch/riscv/include/asm/byteorder.h
@@ -0,0 +1,16 @@
+#ifndef __ASM_RISCV_BYTEORDER_H__
+#define __ASM_RISCV_BYTEORDER_H__
+
+#define __BYTEORDER_HAS_U64__
+
+#include <xen/byteorder/little_endian.h>
+
+#endif /* __ASM_RISCV_BYTEORDER_H__ */
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
-- 
2.42.0

[PATCH v2 04/39] xen/riscv: add public arch-riscv.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
    - align with arch-arm.h.
---

 xen/include/public/arch-riscv.h | 93 +++++++++++++++++++++++++++++++++
 xen/include/public/xen.h        |  2 +
 2 files changed, 95 insertions(+)
 create mode 100644 xen/include/public/arch-riscv.h

diff --git a/xen/include/public/arch-riscv.h b/xen/include/public/arch-riscv.h
new file mode 100644
index 0000000000..168263b920
--- /dev/null
+++ b/xen/include/public/arch-riscv.h
@@ -0,0 +1,93 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Guest OS interface to RISC-V Xen.
+ * Initially based on the ARM implementation.
+ */
+
+#ifndef __XEN_PUBLIC_ARCH_RISCV_H__
+#define __XEN_PUBLIC_ARCH_RISCV_H__
+
+#if defined(__XEN__) || defined(__XEN_TOOLS__) || defined(__GNUC__)
+#define  int64_aligned_t  int64_t __attribute__((__aligned__(8)))
+#define uint64_aligned_t uint64_t __attribute__((__aligned__(8)))
+#endif
+
+#ifndef __ASSEMBLY__
+#define ___DEFINE_XEN_GUEST_HANDLE(name, type)                  \
+    typedef union { type *p; unsigned long q; }                 \
+        __guest_handle_ ## name;                                \
+    typedef union { type *p; uint64_aligned_t q; }              \
+        __guest_handle_64_ ## name
+
+/*
+ * XEN_GUEST_HANDLE represents a guest pointer, when passed as a field
+ * in a struct in memory. On RISCV is always 8 bytes sizes and 8 bytes
+ * aligned.
+ * XEN_GUEST_HANDLE_PARAM represents a guest pointer, when passed as an
+ * hypercall argument. It is 4 bytes on riscv32 and 8 bytes on riscv64.
+ */
+#define __DEFINE_XEN_GUEST_HANDLE(name, type) \
+    ___DEFINE_XEN_GUEST_HANDLE(name, type);   \
+    ___DEFINE_XEN_GUEST_HANDLE(const_##name, const type)
+#define DEFINE_XEN_GUEST_HANDLE(name)   __DEFINE_XEN_GUEST_HANDLE(name, name)
+#define __XEN_GUEST_HANDLE(name)        __guest_handle_64_ ## name
+#define XEN_GUEST_HANDLE(name)          __XEN_GUEST_HANDLE(name)
+#define XEN_GUEST_HANDLE_PARAM(name)    __guest_handle_ ## name
+#define set_xen_guest_handle_raw(hnd, val)                  \
+    do {                                                    \
+        typeof(&(hnd)) sxghr_tmp_ = &(hnd);                 \
+        sxghr_tmp_->q = 0;                                  \
+        sxghr_tmp_->p = (val);                              \
+    } while ( 0 )
+#define set_xen_guest_handle(hnd, val) set_xen_guest_handle_raw(hnd, val)
+
+typedef uint64_t xen_pfn_t;
+#define PRI_xen_pfn PRIx64
+#define PRIu_xen_pfn PRIu64
+
+typedef uint64_t xen_ulong_t;
+#define PRI_xen_ulong PRIx64
+
+#if defined(__XEN__) || defined(__XEN_TOOLS__)
+
+struct vcpu_guest_context {
+};
+typedef struct vcpu_guest_context vcpu_guest_context_t;
+DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
+
+struct xen_arch_domainconfig {
+};
+
+#endif
+
+/* TODO:  add a placeholder entry if no real ones surface */
+struct arch_vcpu_info {
+};
+typedef struct arch_vcpu_info arch_vcpu_info_t;
+
+/* TODO:  add a placeholder entry if no real ones surface */
+struct arch_shared_info {
+};
+typedef struct arch_shared_info arch_shared_info_t;
+
+/*
+ * Maximum number of virtual CPUs in legacy multi-processor guests.
+ * Only one. All other VCPUS must use VCPUOP_register_vcpu_info.
+ */
+#define XEN_LEGACY_MAX_VCPUS 1
+
+/* Stub definition of PMU structure */
+typedef struct xen_pmu_arch { uint8_t dummy; } xen_pmu_arch_t;
+#endif
+
+#endif /*  __XEN_PUBLIC_ARCH_RISCV_H__ */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h
index b812a0a324..b47d48d0e2 100644
--- a/xen/include/public/xen.h
+++ b/xen/include/public/xen.h
@@ -18,6 +18,8 @@
 #include "arch-arm.h"
 #elif defined(__powerpc64__)
 #include "arch-ppc.h"
+#elif defined(__riscv)
+#include "arch-riscv.h"
 #else
 #error "Unsupported architecture"
 #endif
-- 
2.42.0

[PATCH v2 05/39] xen/riscv: introduce spinlock.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - update definition of arch_lock_signal() to not violate cod style.
 - update definition of arch_lock_signal_wmb() to use WMB aspect.
---
 xen/arch/riscv/include/asm/spinlock.h | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/spinlock.h

diff --git a/xen/arch/riscv/include/asm/spinlock.h b/xen/arch/riscv/include/asm/spinlock.h
new file mode 100644
index 0000000000..bf45a7f005
--- /dev/null
+++ b/xen/arch/riscv/include/asm/spinlock.h
@@ -0,0 +1,15 @@
+#ifndef __ASM_RISCV_SPINLOCK_H
+#define __ASM_RISCV_SPINLOCK_H
+
+#define arch_lock_acquire_barrier() smp_mb()
+#define arch_lock_release_barrier() smp_mb()
+
+#define arch_lock_relax() cpu_relax()
+#define arch_lock_signal() ((void)0)
+#define arch_lock_signal_wmb()      \
+({                                  \
+    smp_wmb();                      \
+    arch_lock_signal();             \
+})
+
+#endif /* __ASM_RISCV_SPINLOCK_H */
-- 
2.42.0

[PATCH v2 06/39] xen/riscv: introduce fence.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/fence.h | 12 ++++++++++++
 1 file changed, 12 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/fence.h

diff --git a/xen/arch/riscv/include/asm/fence.h b/xen/arch/riscv/include/asm/fence.h
new file mode 100644
index 0000000000..2b443a3a48
--- /dev/null
+++ b/xen/arch/riscv/include/asm/fence.h
@@ -0,0 +1,12 @@
+#ifndef _ASM_RISCV_FENCE_H
+#define _ASM_RISCV_FENCE_H
+
+#ifdef CONFIG_SMP
+#define RISCV_ACQUIRE_BARRIER		"\tfence r , rw\n"
+#define RISCV_RELEASE_BARRIER		"\tfence rw,  w\n"
+#else
+#define RISCV_ACQUIRE_BARRIER
+#define RISCV_RELEASE_BARRIER
+#endif
+
+#endif	/* _ASM_RISCV_FENCE_H */
-- 
2.42.0

[PATCH v2 07/39] xen/riscv: introduce arch-riscv/hvm/save.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - remove copyright an the top of hvm/save.h as the header write now is a newly
   introduced empty header.
---
 xen/include/public/arch-riscv/hvm/save.h | 20 ++++++++++++++++++++
 xen/include/public/hvm/save.h            |  2 ++
 2 files changed, 22 insertions(+)
 create mode 100644 xen/include/public/arch-riscv/hvm/save.h

diff --git a/xen/include/public/arch-riscv/hvm/save.h b/xen/include/public/arch-riscv/hvm/save.h
new file mode 100644
index 0000000000..1eb037830d
--- /dev/null
+++ b/xen/include/public/arch-riscv/hvm/save.h
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Structure definitions for HVM state that is held by Xen and must
+ * be saved along with the domain's memory and device-model state.
+ */
+
+#ifndef __XEN_PUBLIC_HVM_SAVE_RISCV_H__
+#define __XEN_PUBLIC_HVM_SAVE_RISCV_H__
+
+#endif
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/public/hvm/save.h b/xen/include/public/hvm/save.h
index 2cf4238daa..80328c3216 100644
--- a/xen/include/public/hvm/save.h
+++ b/xen/include/public/hvm/save.h
@@ -91,6 +91,8 @@ DECLARE_HVM_SAVE_TYPE(END, 0, struct hvm_save_end);
 #include "../arch-arm/hvm/save.h"
 #elif defined(__powerpc64__)
 #include "../arch-ppc.h"
+#elif defined(__riscv)
+#include "../arch-riscv/hvm/save.h"
 #else
 #error "unsupported architecture"
 #endif
-- 
2.42.0

[PATCH v2 08/39] xen/riscv: introduce asm/cpufeature.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/cpufeature.h | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/cpufeature.h

diff --git a/xen/arch/riscv/include/asm/cpufeature.h b/xen/arch/riscv/include/asm/cpufeature.h
new file mode 100644
index 0000000000..211827f816
--- /dev/null
+++ b/xen/arch/riscv/include/asm/cpufeature.h
@@ -0,0 +1,22 @@
+#ifndef __ASM_RISCV_CPUFEATURE_H
+#define __ASM_RISCV_CPUFEATURE_H
+
+#ifndef __ASSEMBLY__
+
+static inline int cpu_nr_siblings(unsigned int cpu)
+{
+    return 1;
+}
+
+#endif /* __ASSEMBLY__ */
+
+#endif
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
+
-- 
2.42.0

[PATCH v2 09/39] xen/riscv: introduce asm/guest_atomics.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/guest_atomics.h | 48 ++++++++++++++++++++++
 1 file changed, 48 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/guest_atomics.h

diff --git a/xen/arch/riscv/include/asm/guest_atomics.h b/xen/arch/riscv/include/asm/guest_atomics.h
new file mode 100644
index 0000000000..9f14f76e92
--- /dev/null
+++ b/xen/arch/riscv/include/asm/guest_atomics.h
@@ -0,0 +1,48 @@
+#ifndef __ASM_RISCV_GUEST_ATOMICS_H
+#define __ASM_RISCV_GUEST_ATOMICS_H
+
+/*
+ * TODO: implement guest atomics
+ */
+
+#define guest_testop(name)                                                  \
+static inline int guest_##name(struct domain *d, int nr, volatile void *p)  \
+{                                                                           \
+    (void) d;                                                               \
+    (void) nr;                                                              \
+    (void) p;                                                               \
+                                                                            \
+    return 0;                                                               \
+}
+
+#define guest_bitop(name)                                                   \
+static inline void guest_##name(struct domain *d, int nr, volatile void *p) \
+{                                                                           \
+    (void) d;                                                               \
+    (void) nr;                                                              \
+    (void) p;                                                               \
+}
+
+guest_bitop(set_bit)
+guest_bitop(clear_bit)
+guest_bitop(change_bit)
+
+#undef guest_bitop
+
+guest_testop(test_and_set_bit)
+guest_testop(test_and_clear_bit)
+guest_testop(test_and_change_bit)
+
+#undef guest_testop
+
+#define guest_test_bit(d, nr, p) ((void)(d), test_bit(nr, p))
+
+#endif /* __ASM_RISCV_GUEST_ATOMICS_H */
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
-- 
2.42.0

[PATCH v2 10/39] xen/riscv: introduce asm/iommu.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/iommu.h | 7 +++++++
 1 file changed, 7 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/iommu.h

diff --git a/xen/arch/riscv/include/asm/iommu.h b/xen/arch/riscv/include/asm/iommu.h
new file mode 100644
index 0000000000..648b6fb922
--- /dev/null
+++ b/xen/arch/riscv/include/asm/iommu.h
@@ -0,0 +1,7 @@
+#ifndef __ASM_RISCV_IOMMU_H__
+#define __ASM_RISCV_IOMMU_H__
+
+struct arch_iommu {
+};
+
+#endif /* __ASM_IOMMU_H__ */
-- 
2.42.0

[PATCH v2 11/39] xen/riscv: introduce asm/nospec.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/nospec.h | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/nospec.h

diff --git a/xen/arch/riscv/include/asm/nospec.h b/xen/arch/riscv/include/asm/nospec.h
new file mode 100644
index 0000000000..55087fa831
--- /dev/null
+++ b/xen/arch/riscv/include/asm/nospec.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. */
+
+#ifndef _ASM_RISCV_NOSPEC_H
+#define _ASM_RISCV_NOSPEC_H
+
+static inline bool evaluate_nospec(bool condition)
+{
+    return condition;
+}
+
+static inline void block_speculation(void)
+{
+}
+
+#endif /* _ASM_RISCV_NOSPEC_H */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
-- 
2.42.0

[PATCH v2 12/39] xen/riscv: introduce asm/setup.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/setup.h | 6 ++++++
 1 file changed, 6 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/setup.h

diff --git a/xen/arch/riscv/include/asm/setup.h b/xen/arch/riscv/include/asm/setup.h
new file mode 100644
index 0000000000..a1adb2be0d
--- /dev/null
+++ b/xen/arch/riscv/include/asm/setup.h
@@ -0,0 +1,6 @@
+#ifndef __ASM_RISCV_SETUP_H__
+#define __ASM_RISCV_SETUP_H__
+
+#define max_init_domid (0)
+
+#endif /* __ASM_RISCV_SETUP_H__ */
-- 
2.42.0

[PATCH v2 13/39] xen/riscv: introduce asm/system.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/system.h | 79 +++++++++++++++++++++++++++++
 1 file changed, 79 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/system.h

diff --git a/xen/arch/riscv/include/asm/system.h b/xen/arch/riscv/include/asm/system.h
new file mode 100644
index 0000000000..45bfed4e2b
--- /dev/null
+++ b/xen/arch/riscv/include/asm/system.h
@@ -0,0 +1,79 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef _ASM_RISCV_BARRIER_H
+#define _ASM_RISCV_BARRIER_H
+
+#include <asm/csr.h>
+
+#ifndef __ASSEMBLY__
+
+#define RISCV_FENCE(p, s) \
+    __asm__ __volatile__ ("fence " #p "," #s : : : "memory")
+
+/* These barriers need to enforce ordering on both devices or memory. */
+#define mb()                    RISCV_FENCE(iorw,iorw)
+#define rmb()                   RISCV_FENCE(ir,ir)
+#define wmb()                   RISCV_FENCE(ow,ow)
+
+/* These barriers do not need to enforce ordering on devices, just memory. */
+#define smp_mb()                RISCV_FENCE(rw,rw)
+#define smp_rmb()               RISCV_FENCE(r,r)
+#define smp_wmb()               RISCV_FENCE(w,w)
+#define smp_mb__before_atomic() smp_mb()
+#define smp_mb__after_atomic()  smp_mb()
+
+/*
+#define __smp_store_release(p, v)       \
+do {                                    \
+	compiletime_assert_atomic_type(*p); \
+	RISCV_FENCE(rw,w);                  \
+	WRITE_ONCE(*p, v);                  \
+} while (0)
+
+#define __smp_load_acquire(p)           \
+({                                      \
+    typeof(*p) ___p1 = READ_ONCE(*p);   \
+    compiletime_assert_atomic_type(*p); \
+    RISCV_FENCE(r,rw);                  \
+    ___p1;                              \
+})
+*/
+
+static inline unsigned long local_save_flags(void)
+{
+    return csr_read(sstatus);
+}
+
+static inline void local_irq_enable(void)
+{
+    csr_set(sstatus, SSTATUS_SIE);
+}
+
+static inline void local_irq_disable(void)
+{
+    csr_clear(sstatus, SSTATUS_SIE);
+}
+
+#define local_irq_save(x)                           \
+({                                                  \
+    x = csr_read_clear(CSR_SSTATUS, SSTATUS_SIE);   \
+    local_irq_disable();                            \
+})
+
+static inline void local_irq_restore(unsigned long flags)
+{
+	csr_set(CSR_SSTATUS, flags & SSTATUS_SIE);
+}
+
+static inline int local_irq_is_enabled(void)
+{
+    unsigned long flags = local_save_flags();
+
+    return flags & SSTATUS_SIE;
+}
+
+#define arch_fetch_and_add(x, v) __sync_fetch_and_add(x, v)
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* _ASM_RISCV_BARRIER_H */
-- 
2.42.0

[PATCH v2 14/39] xen/riscv: introduce bitops.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/bitops.h | 288 ++++++++++++++++++++++++++++
 1 file changed, 288 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/bitops.h

diff --git a/xen/arch/riscv/include/asm/bitops.h b/xen/arch/riscv/include/asm/bitops.h
new file mode 100644
index 0000000000..24a49c499b
--- /dev/null
+++ b/xen/arch/riscv/include/asm/bitops.h
@@ -0,0 +1,288 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (C) 2012 Regents of the University of California */
+
+#ifndef _ASM_RISCV_BITOPS_H
+#define _ASM_RISCV_BITOPS_H
+
+#include <asm/system.h>
+
+#define BITOP_BITS_PER_WORD     32
+#define BITOP_MASK(nr)	        (1UL << ((nr) % BITOP_BITS_PER_WORD))
+#define BITOP_WORD(nr)	        ((nr) / BITOP_BITS_PER_WORD)
+#define BITS_PER_BYTE	        8
+
+#define __set_bit(n,p)          set_bit(n,p)
+#define __clear_bit(n,p)        clear_bit(n,p)
+
+#define __AMO(op)	"amo" #op ".w"
+
+#define __test_and_op_bit_ord(op, mod, nr, addr, ord)		\
+({								\
+	unsigned long __res, __mask;				\
+	__mask = BITOP_MASK(nr);					\
+	__asm__ __volatile__ (					\
+		__AMO(op) #ord " %0, %2, %1"			\
+		: "=r" (__res), "+A" (addr[BITOP_WORD(nr)])	\
+		: "r" (mod(__mask))				\
+		: "memory");					\
+	((__res & __mask) != 0);				\
+})
+
+#define __op_bit_ord(op, mod, nr, addr, ord)			\
+	__asm__ __volatile__ (					\
+		__AMO(op) #ord " zero, %1, %0"			\
+		: "+A" (addr[BITOP_WORD(nr)])			\
+		: "r" (mod(BITOP_MASK(nr)))			\
+		: "memory");
+
+#define __test_and_op_bit(op, mod, nr, addr) 			\
+	__test_and_op_bit_ord(op, mod, nr, addr, .aqrl)
+
+#define __op_bit(op, mod, nr, addr)				\
+	__op_bit_ord(op, mod, nr, addr, )
+
+/* Bitmask modifiers */
+#define __NOP(x)	(x)
+#define __NOT(x)	(~(x))
+
+/**
+ * __test_and_set_bit - Set a bit and return its old value
+ * @nr: Bit to set
+ * @addr: Address to count from
+ *
+ * This operation may be reordered on other architectures than x86.
+ */
+static inline int __test_and_set_bit(int nr, volatile void *p)
+{
+	volatile uint32_t *addr = p;
+
+	return __test_and_op_bit(or, __NOP, nr, addr);
+}
+
+/**
+ * __test_and_clear_bit - Clear a bit and return its old value
+ * @nr: Bit to clear
+ * @addr: Address to count from
+ *
+ * This operation can be reordered on other architectures other than x86.
+ */
+static inline int __test_and_clear_bit(int nr, volatile void *p)
+{
+	volatile uint32_t *addr = p;
+
+	return __test_and_op_bit(and, __NOT, nr, addr);
+}
+
+/**
+ * set_bit - Atomically set a bit in memory
+ * @nr: the bit to set
+ * @addr: the address to start counting from
+ *
+ * Note: there are no guarantees that this function will not be reordered
+ * on non x86 architectures, so if you are writing portable code,
+ * make sure not to rely on its reordering guarantees.
+ *
+ * Note that @nr may be almost arbitrarily large; this function is not
+ * restricted to acting on a single-word quantity.
+ */
+static inline void set_bit(int nr, volatile void *p)
+{
+	volatile uint32_t *addr = p;
+
+	__op_bit(or, __NOP, nr, addr);
+}
+
+/**
+ * clear_bit - Clears a bit in memory
+ * @nr: Bit to clear
+ * @addr: Address to start counting from
+ *
+ * Note: there are no guarantees that this function will not be reordered
+ * on non x86 architectures, so if you are writing portable code,
+ * make sure not to rely on its reordering guarantees.
+ */
+static inline void clear_bit(int nr, volatile void *p)
+{
+	volatile uint32_t *addr = p;
+
+	__op_bit(and, __NOT, nr, addr);
+}
+
+static inline int test_bit(int nr, const volatile void *p)
+{
+	const volatile uint32_t *addr = (const volatile uint32_t *)p;
+
+	return 1UL & (addr[BITOP_WORD(nr)] >> (nr & (BITOP_BITS_PER_WORD-1)));
+}
+
+#undef __test_and_op_bit
+#undef __op_bit
+#undef __NOP
+#undef __NOT
+#undef __AMO
+
+static inline int fls(unsigned int x)
+{
+    return generic_fls(x);
+}
+
+static inline int flsl(unsigned long x)
+{
+    return generic_flsl(x);
+}
+
+#define test_and_set_bit   __test_and_set_bit
+#define test_and_clear_bit __test_and_clear_bit
+
+/* Based on linux/include/asm-generic/bitops/find.h */
+
+#ifndef find_next_bit
+/**
+ * find_next_bit - find the next set bit in a memory region
+ * @addr: The address to base the search on
+ * @offset: The bitnumber to start searching at
+ * @size: The bitmap size in bits
+ */
+extern unsigned long find_next_bit(const unsigned long *addr, unsigned long
+		size, unsigned long offset);
+#endif
+
+#ifndef find_next_zero_bit
+/**
+ * find_next_zero_bit - find the next cleared bit in a memory region
+ * @addr: The address to base the search on
+ * @offset: The bitnumber to start searching at
+ * @size: The bitmap size in bits
+ */
+extern unsigned long find_next_zero_bit(const unsigned long *addr, unsigned
+		long size, unsigned long offset);
+#endif
+
+/**
+ * find_first_bit - find the first set bit in a memory region
+ * @addr: The address to start the search at
+ * @size: The maximum size to search
+ *
+ * Returns the bit number of the first set bit.
+ */
+extern unsigned long find_first_bit(const unsigned long *addr,
+				    unsigned long size);
+
+/**
+ * find_first_zero_bit - find the first cleared bit in a memory region
+ * @addr: The address to start the search at
+ * @size: The maximum size to search
+ *
+ * Returns the bit number of the first cleared bit.
+ */
+extern unsigned long find_first_zero_bit(const unsigned long *addr,
+					 unsigned long size);
+
+#define ffs(x) ({ unsigned int __t = (x); fls(__t & -__t); })
+
+/**
+ * ffs - find first bit in word.
+ * @word: The word to search
+ *
+ * Returns 0 if no bit exists, otherwise returns 1-indexed bit location.
+ */
+static inline unsigned long __ffs(unsigned long word)
+{
+	int num = 0;
+
+#if BITS_PER_LONG == 64
+	if ((word & 0xffffffff) == 0) {
+		num += 32;
+		word >>= 32;
+	}
+#endif
+	if ((word & 0xffff) == 0) {
+		num += 16;
+		word >>= 16;
+	}
+	if ((word & 0xff) == 0) {
+		num += 8;
+		word >>= 8;
+	}
+	if ((word & 0xf) == 0) {
+		num += 4;
+		word >>= 4;
+	}
+	if ((word & 0x3) == 0) {
+		num += 2;
+		word >>= 2;
+	}
+	if ((word & 0x1) == 0)
+		num += 1;
+	return num;
+}
+
+/**
+ * ffsl - find first bit in long.
+ * @word: The word to search
+ *
+ * Returns 0 if no bit exists, otherwise returns 1-indexed bit location.
+ */
+static inline unsigned int ffsl(unsigned long word)
+{
+    int num = 1;
+
+    if (!word)
+        return 0;
+
+#if BITS_PER_LONG == 64
+    if ((word & 0xffffffff) == 0) {
+        num += 32;
+        word >>= 32;
+    }
+#endif
+    if ((word & 0xffff) == 0) {
+        num += 16;
+        word >>= 16;
+    }
+    if ((word & 0xff) == 0) {
+        num += 8;
+        word >>= 8;
+    }
+    if ((word & 0xf) == 0) {
+        num += 4;
+        word >>= 4;
+    }
+    if ((word & 0x3) == 0) {
+        num += 2;
+        word >>= 2;
+    }
+    if ((word & 0x1) == 0)
+        num += 1;
+    return num;
+}
+
+/*
+ * ffz - find first zero in word.
+ * @word: The word to search
+ *
+ * Undefined if no zero exists, so code should check against ~0UL first.
+ */
+#define ffz(x)  __ffs(~(x))
+
+/**
+ * find_first_set_bit - find the first set bit in @word
+ * @word: the word to search
+ *
+ * Returns the bit-number of the first set bit (first bit being 0).
+ * The input must *not* be zero.
+ */
+static inline unsigned int find_first_set_bit(unsigned long word)
+{
+        return ffsl(word) - 1;
+}
+
+/**
+ * hweightN - returns the hamming weight of a N-bit word
+ * @x: the word to weigh
+ *
+ * The Hamming Weight of a number is the total number of bits set in it.
+ */
+#define hweight64(x) generic_hweight64(x)
+
+#endif /* _ASM_RISCV_BITOPS_H */
-- 
2.42.0

[PATCH v2 15/39] xen/riscv: introduce flushtlb.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/flushtlb.h | 31 +++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/flushtlb.h

diff --git a/xen/arch/riscv/include/asm/flushtlb.h b/xen/arch/riscv/include/asm/flushtlb.h
new file mode 100644
index 0000000000..7059a53823
--- /dev/null
+++ b/xen/arch/riscv/include/asm/flushtlb.h
@@ -0,0 +1,31 @@
+#ifndef __ASM_RISCV_FLUSHTLB_H__
+#define __ASM_RISCV_FLUSHTLB_H__
+
+#include <xen/cpumask.h>
+
+/*
+ * Filter the given set of CPUs, removing those that definitely flushed their
+ * TLB since @page_timestamp.
+ */
+/* XXX lazy implementation just doesn't clear anything.... */
+static inline void tlbflush_filter(cpumask_t *mask, uint32_t page_timestamp) {}
+
+#define tlbflush_current_time() (0)
+
+static inline void page_set_tlbflush_timestamp(struct page_info *page)
+{
+    BUG();
+}
+
+/* Flush specified CPUs' TLBs */
+void arch_flush_tlb_mask(const cpumask_t *mask);
+
+#endif /* __ASM_RISCV_FLUSHTLB_H__ */
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
-- 
2.42.0

[PATCH v2 16/39] xen/riscv: introduce asm/smp.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/smp.h | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/smp.h

diff --git a/xen/arch/riscv/include/asm/smp.h b/xen/arch/riscv/include/asm/smp.h
new file mode 100644
index 0000000000..a7b4f94ce4
--- /dev/null
+++ b/xen/arch/riscv/include/asm/smp.h
@@ -0,0 +1,23 @@
+#ifndef __ASM_RISCV_SMP_H
+#define __ASM_RISCV_SMP_H
+
+#ifndef __ASSEMBLY__
+#include <xen/cpumask.h>
+#include <xen/percpu.h>
+#endif
+
+DECLARE_PER_CPU(cpumask_var_t, cpu_sibling_mask);
+DECLARE_PER_CPU(cpumask_var_t, cpu_core_mask);
+
+#define cpu_is_offline(cpu) unlikely(!cpu_online(cpu))
+
+/*
+ * Do we, for platform reasons, need to actually keep CPUs online when we
+ * would otherwise prefer them to be off?
+ */
+#define park_offline_cpus false
+
+/* TODO: need to be implemeted */
+#define smp_processor_id() (0)
+
+#endif
\ No newline at end of file
-- 
2.42.0

[PATCH v2 17/39] xen/riscv: introduce asm/atomic.h

[Oleksii Kurochko]

From: Bobby Eshleman <bobbyeshleman@gmail.com>

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Change an author of commit. I got this header from Bobby's old repo.
---
 xen/arch/riscv/include/asm/atomic.h | 375 ++++++++++++++++++++++++++++
 1 file changed, 375 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/atomic.h

diff --git a/xen/arch/riscv/include/asm/atomic.h b/xen/arch/riscv/include/asm/atomic.h
new file mode 100644
index 0000000000..d4849cf282
--- /dev/null
+++ b/xen/arch/riscv/include/asm/atomic.h
@@ -0,0 +1,375 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Taken and modified from Linux.
+ * 
+ * Copyright (C) 2007 Red Hat, Inc. All Rights Reserved.
+ * Copyright (C) 2012 Regents of the University of California
+ * Copyright (C) 2017 SiFive
+ * Copyright (C) 2021 Vates SAS
+ */
+
+#ifndef _ASM_RISCV_ATOMIC_H
+#define _ASM_RISCV_ATOMIC_H
+
+#include <xen/atomic.h>
+#include <asm/cmpxchg.h>
+#include <asm/fence.h>
+#include <asm/io.h>
+#include <asm/system.h>
+
+void __bad_atomic_size(void);
+
+static always_inline void read_atomic_size(const volatile void *p,
+                                           void *res,
+                                           unsigned int size)
+{
+    switch ( size ) {
+    case 1: *(uint8_t *)res = readb((uint8_t *)p); break;
+    case 2: *(uint16_t *)res = readw((uint16_t *)p); break;
+    case 4: *(uint32_t *)res = readl((uint32_t *)p); break;
+    case 8: *(uint32_t *)res  = readq((uint64_t *)p); break;
+    default: __bad_atomic_size(); break;
+    }
+}
+
+#define read_atomic(p) ({                                               \
+    union { typeof(*p) val; char c[0]; } x_;                            \
+    read_atomic_size(p, x_.c, sizeof(*p));                              \
+    x_.val;                                                             \
+})
+
+
+#define write_atomic(p, x) ({                                           \
+    typeof(*p) __x = (x);                                               \
+    switch ( sizeof(*p) ) {                                             \
+    case 1: writeb((uint8_t)__x,  (uint8_t *)  p); break;              \
+    case 2: writew((uint16_t)__x, (uint16_t *) p); break;              \
+    case 4: writel((uint32_t)__x, (uint32_t *) p); break;              \
+    case 8: writeq((uint64_t)__x, (uint64_t *) p); break;              \
+    default: __bad_atomic_size(); break;                                \
+    }                                                                   \
+    __x;                                                                \
+})
+
+/* TODO: Fix this */
+#define add_sized(p, x) ({                                              \
+    typeof(*(p)) __x = (x);                                             \
+    switch ( sizeof(*(p)) )                                             \
+    {                                                                   \
+    case 1: writeb(read_atomic(p) + __x, (uint8_t *)(p)); break;        \
+    case 2: writew(read_atomic(p) + __x, (uint16_t *)(p)); break;       \
+    case 4: writel(read_atomic(p) + __x, (uint32_t *)(p)); break;       \
+    default: __bad_atomic_size(); break;                                \
+    }                                                                   \
+})
+
+/*
+ *  __unqual_scalar_typeof(x) - Declare an unqualified scalar type, leaving
+ *               non-scalar types unchanged.
+ *
+ * Prefer C11 _Generic for better compile-times and simpler code. Note: 'char'
+ * is not type-compatible with 'signed char', and we define a separate case.
+ */
+#define __scalar_type_to_expr_cases(type)               \
+    unsigned type:  (unsigned type)0,                   \
+    signed type:    (signed type)0
+
+#define __unqual_scalar_typeof(x) typeof(               \
+    _Generic((x),                                       \
+        char:  (char)0,                                 \
+        __scalar_type_to_expr_cases(char),              \
+        __scalar_type_to_expr_cases(short),             \
+        __scalar_type_to_expr_cases(int),               \
+        __scalar_type_to_expr_cases(long),              \
+        __scalar_type_to_expr_cases(long long),         \
+        default: (x)))
+
+#define READ_ONCE(x)  (*(const volatile __unqual_scalar_typeof(x) *)&(x))
+#define WRITE_ONCE(x, val)                                      \
+    do {                                                        \
+            *(volatile typeof(x) *)&(x) = (val);                \
+    } while (0)
+
+#define __atomic_acquire_fence()					\
+	__asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory")
+
+#define __atomic_release_fence()					\
+	__asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory");
+
+static inline int atomic_read(const atomic_t *v)
+{
+	return READ_ONCE(v->counter);
+}
+
+static inline int _atomic_read(atomic_t v)
+{
+    return v.counter;
+}
+
+static inline void atomic_set(atomic_t *v, int i)
+{
+	WRITE_ONCE(v->counter, i);
+}
+
+static inline void _atomic_set(atomic_t *v, int i)
+{
+    v->counter = i;
+}
+
+static inline int atomic_sub_and_test(int i, atomic_t *v)
+{
+    return atomic_sub_return(i, v) == 0;
+}
+
+static inline void atomic_inc(atomic_t *v)
+{
+    atomic_add(1, v);
+}
+
+static inline int atomic_inc_return(atomic_t *v)
+{
+    return atomic_add_return(1, v);
+}
+
+static inline void atomic_dec(atomic_t *v)
+{
+    atomic_sub(1, v);
+}
+
+static inline int atomic_dec_return(atomic_t *v)
+{
+    return atomic_sub_return(1, v);
+}
+
+
+static inline int atomic_dec_and_test(atomic_t *v)
+{
+    return atomic_sub_return(1, v) == 0;
+}
+
+static inline int atomic_add_negative(int i, atomic_t *v)
+{
+    return atomic_add_return(i, v) < 0;
+}
+
+static inline int atomic_inc_and_test(atomic_t *v)
+{
+    return atomic_add_return(1, v) == 0;
+}
+
+/*
+ * First, the atomic ops that have no ordering constraints and therefor don't
+ * have the AQ or RL bits set.  These don't return anything, so there's only
+ * one version to worry about.
+ */
+#define ATOMIC_OP(op, asm_op, I, asm_type, c_type, prefix)		\
+static inline							\
+void atomic##prefix##_##op(c_type i, atomic##prefix##_t *v)		\
+{									\
+	__asm__ __volatile__ (						\
+		"	amo" #asm_op "." #asm_type " zero, %1, %0"	\
+		: "+A" (v->counter)					\
+		: "r" (I)						\
+		: "memory");						\
+}									\
+
+#define ATOMIC_OPS(op, asm_op, I)					\
+        ATOMIC_OP (op, asm_op, I, w, int,   )
+
+ATOMIC_OPS(add, add,  i)
+ATOMIC_OPS(sub, add, -i)
+ATOMIC_OPS(and, and,  i)
+ATOMIC_OPS( or,  or,  i)
+ATOMIC_OPS(xor, xor,  i)
+
+#undef ATOMIC_OP
+#undef ATOMIC_OPS
+
+/*
+ * Atomic ops that have ordered, relaxed, acquire, and release variants.
+ * There's two flavors of these: the arithmatic ops have both fetch and return
+ * versions, while the logical ops only have fetch versions.
+ */
+#define ATOMIC_FETCH_OP(op, asm_op, I, asm_type, c_type, prefix)	\
+static inline							\
+c_type atomic##prefix##_fetch_##op##_relaxed(c_type i,			\
+					     atomic##prefix##_t *v)	\
+{									\
+	register c_type ret;						\
+	__asm__ __volatile__ (						\
+		"	amo" #asm_op "." #asm_type " %1, %2, %0"	\
+		: "+A" (v->counter), "=r" (ret)				\
+		: "r" (I)						\
+		: "memory");						\
+	return ret;							\
+}									\
+static inline							\
+c_type atomic##prefix##_fetch_##op(c_type i, atomic##prefix##_t *v)	\
+{									\
+	register c_type ret;						\
+	__asm__ __volatile__ (						\
+		"	amo" #asm_op "." #asm_type ".aqrl  %1, %2, %0"	\
+		: "+A" (v->counter), "=r" (ret)				\
+		: "r" (I)						\
+		: "memory");						\
+	return ret;							\
+}
+
+#define ATOMIC_OP_RETURN(op, asm_op, c_op, I, asm_type, c_type, prefix)	\
+static inline							\
+c_type atomic##prefix##_##op##_return_relaxed(c_type i,			\
+					      atomic##prefix##_t *v)	\
+{									\
+        return atomic##prefix##_fetch_##op##_relaxed(i, v) c_op I;	\
+}									\
+static inline							\
+c_type atomic##prefix##_##op##_return(c_type i, atomic##prefix##_t *v)	\
+{									\
+        return atomic##prefix##_fetch_##op(i, v) c_op I;		\
+}
+
+#define ATOMIC_OPS(op, asm_op, c_op, I)					\
+        ATOMIC_FETCH_OP( op, asm_op,       I, w, int,   )		\
+        ATOMIC_OP_RETURN(op, asm_op, c_op, I, w, int,   )
+
+ATOMIC_OPS(add, add, +,  i)
+ATOMIC_OPS(sub, add, +, -i)
+
+#define atomic_add_return_relaxed	atomic_add_return_relaxed
+#define atomic_sub_return_relaxed	atomic_sub_return_relaxed
+#define atomic_add_return		atomic_add_return
+#define atomic_sub_return		atomic_sub_return
+
+#define atomic_fetch_add_relaxed	atomic_fetch_add_relaxed
+#define atomic_fetch_sub_relaxed	atomic_fetch_sub_relaxed
+#define atomic_fetch_add		atomic_fetch_add
+#define atomic_fetch_sub		atomic_fetch_sub
+
+#undef ATOMIC_OPS
+
+#define ATOMIC_OPS(op, asm_op, I)					\
+        ATOMIC_FETCH_OP(op, asm_op, I, w, int,   )
+
+ATOMIC_OPS(and, and, i)
+ATOMIC_OPS( or,  or, i)
+ATOMIC_OPS(xor, xor, i)
+
+#define atomic_fetch_and_relaxed	atomic_fetch_and_relaxed
+#define atomic_fetch_or_relaxed		atomic_fetch_or_relaxed
+#define atomic_fetch_xor_relaxed	atomic_fetch_xor_relaxed
+#define atomic_fetch_and		atomic_fetch_and
+#define atomic_fetch_or			atomic_fetch_or
+#define atomic_fetch_xor		atomic_fetch_xor
+
+#undef ATOMIC_OPS
+
+#undef ATOMIC_FETCH_OP
+#undef ATOMIC_OP_RETURN
+
+/* This is required to provide a full barrier on success. */
+static inline int atomic_add_unless(atomic_t *v, int a, int u)
+{
+       int prev, rc;
+
+	__asm__ __volatile__ (
+		"0:	lr.w     %[p],  %[c]\n"
+		"	beq      %[p],  %[u], 1f\n"
+		"	add      %[rc], %[p], %[a]\n"
+		"	sc.w.rl  %[rc], %[rc], %[c]\n"
+		"	bnez     %[rc], 0b\n"
+		"	fence    rw, rw\n"
+		"1:\n"
+		: [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter)
+		: [a]"r" (a), [u]"r" (u)
+		: "memory");
+	return prev;
+}
+#define atomic_fetch_add_unless atomic_fetch_add_unless
+
+/*
+ * atomic_{cmp,}xchg is required to have exactly the same ordering semantics as
+ * {cmp,}xchg and the operations that return, so they need a full barrier.
+ */
+#define ATOMIC_OP(c_t, prefix, size)					\
+static inline							\
+c_t atomic##prefix##_xchg_relaxed(atomic##prefix##_t *v, c_t n)		\
+{									\
+	return __xchg_relaxed(&(v->counter), n, size);			\
+}									\
+static inline							\
+c_t atomic##prefix##_xchg_acquire(atomic##prefix##_t *v, c_t n)		\
+{									\
+	return __xchg_acquire(&(v->counter), n, size);			\
+}									\
+static inline							\
+c_t atomic##prefix##_xchg_release(atomic##prefix##_t *v, c_t n)		\
+{									\
+	return __xchg_release(&(v->counter), n, size);			\
+}									\
+static inline							\
+c_t atomic##prefix##_xchg(atomic##prefix##_t *v, c_t n)			\
+{									\
+	return __xchg(&(v->counter), n, size);				\
+}									\
+static inline							\
+c_t atomic##prefix##_cmpxchg_relaxed(atomic##prefix##_t *v,		\
+				     c_t o, c_t n)			\
+{									\
+	return __cmpxchg_relaxed(&(v->counter), o, n, size);		\
+}									\
+static inline							\
+c_t atomic##prefix##_cmpxchg_acquire(atomic##prefix##_t *v,		\
+				     c_t o, c_t n)			\
+{									\
+	return __cmpxchg_acquire(&(v->counter), o, n, size);		\
+}									\
+static inline							\
+c_t atomic##prefix##_cmpxchg_release(atomic##prefix##_t *v,		\
+				     c_t o, c_t n)			\
+{									\
+	return __cmpxchg_release(&(v->counter), o, n, size);		\
+}									\
+static inline							\
+c_t atomic##prefix##_cmpxchg(atomic##prefix##_t *v, c_t o, c_t n)	\
+{									\
+	return __cmpxchg(&(v->counter), o, n, size);			\
+}
+
+#define ATOMIC_OPS()							\
+	ATOMIC_OP(int,   , 4)
+
+ATOMIC_OPS()
+
+#define atomic_xchg_relaxed atomic_xchg_relaxed
+#define atomic_xchg_acquire atomic_xchg_acquire
+#define atomic_xchg_release atomic_xchg_release
+#define atomic_xchg atomic_xchg
+#define atomic_cmpxchg_relaxed atomic_cmpxchg_relaxed
+#define atomic_cmpxchg_acquire atomic_cmpxchg_acquire
+#define atomic_cmpxchg_release atomic_cmpxchg_release
+#define atomic_cmpxchg atomic_cmpxchg
+
+#undef ATOMIC_OPS
+#undef ATOMIC_OP
+
+static inline int atomic_sub_if_positive(atomic_t *v, int offset)
+{
+       int prev, rc;
+
+	__asm__ __volatile__ (
+		"0:	lr.w     %[p],  %[c]\n"
+		"	sub      %[rc], %[p], %[o]\n"
+		"	bltz     %[rc], 1f\n"
+		"	sc.w.rl  %[rc], %[rc], %[c]\n"
+		"	bnez     %[rc], 0b\n"
+		"	fence    rw, rw\n"
+		"1:\n"
+		: [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter)
+		: [o]"r" (offset)
+		: "memory");
+	return prev - offset;
+}
+
+#define atomic_dec_if_positive(v)	atomic_sub_if_positive(v, 1)
+
+#endif /* _ASM_RISCV_ATOMIC_H */
-- 
2.42.0

[PATCH v2 18/39] xen/riscv: introduce cmpxchg.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
	- update the comment at the top of the header.
	- change xen/lib.h to xen/bug.h.
	- sort inclusion of headers properly.
---
 xen/arch/riscv/include/asm/cmpxchg.h | 379 +++++++++++++++++++++++++++
 1 file changed, 379 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/cmpxchg.h

diff --git a/xen/arch/riscv/include/asm/cmpxchg.h b/xen/arch/riscv/include/asm/cmpxchg.h
new file mode 100644
index 0000000000..c535bc9660
--- /dev/null
+++ b/xen/arch/riscv/include/asm/cmpxchg.h
@@ -0,0 +1,379 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ *  Taken and modified from Linux.
+ *
+ *  Copyright (C) 2014 Regents of the University of California
+ */
+
+#ifndef _ASM_RISCV_CMPXCHG_H
+#define _ASM_RISCV_CMPXCHG_H
+
+#include <xen/bug.h>
+
+#include <asm/fence.h>
+#include <asm/system.h>
+
+#define __xchg_relaxed(ptr, new, size)					\
+({									\
+	__typeof__(ptr) __ptr = (ptr);					\
+	__typeof__(new) __new = (new);					\
+	__typeof__(*(ptr)) __ret;					\
+	switch (size) {							\
+	case 4:								\
+		__asm__ __volatile__ (					\
+			"	amoswap.w %0, %2, %1\n"			\
+			: "=r" (__ret), "+A" (*__ptr)			\
+			: "r" (__new)					\
+			: "memory");					\
+		break;							\
+	case 8:								\
+		__asm__ __volatile__ (					\
+			"	amoswap.d %0, %2, %1\n"			\
+			: "=r" (__ret), "+A" (*__ptr)			\
+			: "r" (__new)					\
+			: "memory");					\
+		break;							\
+	default:							\
+		ASSERT_UNREACHABLE();					\
+	}								\
+	__ret;								\
+})
+
+#define xchg_relaxed(ptr, x)						\
+({									\
+	__typeof__(*(ptr)) _x_ = (x);					\
+	(__typeof__(*(ptr))) __xchg_relaxed((ptr),			\
+					    _x_, sizeof(*(ptr)));	\
+})
+
+#define __xchg_acquire(ptr, new, size)					\
+({									\
+	__typeof__(ptr) __ptr = (ptr);					\
+	__typeof__(new) __new = (new);					\
+	__typeof__(*(ptr)) __ret;					\
+	switch (size) {							\
+	case 4:								\
+		__asm__ __volatile__ (					\
+			"	amoswap.w %0, %2, %1\n"			\
+			RISCV_ACQUIRE_BARRIER				\
+			: "=r" (__ret), "+A" (*__ptr)			\
+			: "r" (__new)					\
+			: "memory");					\
+		break;							\
+	case 8:								\
+		__asm__ __volatile__ (					\
+			"	amoswap.d %0, %2, %1\n"			\
+			RISCV_ACQUIRE_BARRIER				\
+			: "=r" (__ret), "+A" (*__ptr)			\
+			: "r" (__new)					\
+			: "memory");					\
+		break;							\
+	default:							\
+		ASSERT_UNREACHABLE();					\
+	}								\
+	__ret;								\
+})
+
+#define xchg_acquire(ptr, x)						\
+({									\
+	__typeof__(*(ptr)) _x_ = (x);					\
+	(__typeof__(*(ptr))) __xchg_acquire((ptr),			\
+					    _x_, sizeof(*(ptr)));	\
+})
+
+#define __xchg_release(ptr, new, size)					\
+({									\
+	__typeof__(ptr) __ptr = (ptr);					\
+	__typeof__(new) __new = (new);					\
+	__typeof__(*(ptr)) __ret;					\
+	switch (size) {							\
+	case 4:								\
+		__asm__ __volatile__ (					\
+			RISCV_RELEASE_BARRIER				\
+			"	amoswap.w %0, %2, %1\n"			\
+			: "=r" (__ret), "+A" (*__ptr)			\
+			: "r" (__new)					\
+			: "memory");					\
+		break;							\
+	case 8:								\
+		__asm__ __volatile__ (					\
+			RISCV_RELEASE_BARRIER				\
+			"	amoswap.d %0, %2, %1\n"			\
+			: "=r" (__ret), "+A" (*__ptr)			\
+			: "r" (__new)					\
+			: "memory");					\
+		break;							\
+	default:							\
+		ASSERT_UNREACHABLE();					\
+	}								\
+	__ret;								\
+})
+
+#define xchg_release(ptr, x)						\
+({									\
+	__typeof__(*(ptr)) _x_ = (x);					\
+	(__typeof__(*(ptr))) __xchg_release((ptr),			\
+					    _x_, sizeof(*(ptr)));	\
+})
+
+#define __xchg(ptr, new, size)						\
+({									\
+	__typeof__(ptr) __ptr = (ptr);					\
+	__typeof__(new) __new = (new);					\
+	__typeof__(*(ptr)) __ret;					\
+	switch (size) {							\
+	case 4:								\
+		__asm__ __volatile__ (					\
+			"	amoswap.w.aqrl %0, %2, %1\n"		\
+			: "=r" (__ret), "+A" (*__ptr)			\
+			: "r" (__new)					\
+			: "memory");					\
+		break;							\
+	case 8:								\
+		__asm__ __volatile__ (					\
+			"	amoswap.d.aqrl %0, %2, %1\n"		\
+			: "=r" (__ret), "+A" (*__ptr)			\
+			: "r" (__new)					\
+			: "memory");					\
+		break;							\
+	default:							\
+		__ret = 0; \
+		ASSERT_UNREACHABLE();					\
+	}								\
+	__ret;								\
+})
+
+#define xchg(ptr, x)							\
+({									\
+	__typeof__(*(ptr)) _x_ = (x);					\
+	(__typeof__(*(ptr))) __xchg((ptr), _x_, sizeof(*(ptr)));	\
+})
+
+#define xchg32(ptr, x)							\
+({									\
+	BUILD_BUG_ON(sizeof(*(ptr)) != 4);				\
+	xchg((ptr), (x));						\
+})
+
+#define xchg64(ptr, x)							\
+({									\
+	BUILD_BUG_ON(sizeof(*(ptr)) != 8);				\
+	xchg((ptr), (x));						\
+})
+
+/*
+ * Atomic compare and exchange.  Compare OLD with MEM, if identical,
+ * store NEW in MEM.  Return the initial value in MEM.  Success is
+ * indicated by comparing RETURN with OLD.
+ */
+#define __cmpxchg_relaxed(ptr, old, new, size)				\
+({									\
+	__typeof__(ptr) __ptr = (ptr);					\
+	__typeof__(*(ptr)) __old = (old);				\
+	__typeof__(*(ptr)) __new = (new);				\
+	__typeof__(*(ptr)) __ret;					\
+	register unsigned int __rc;					\
+	switch (size) {							\
+	case 4:								\
+		__asm__ __volatile__ (					\
+			"0:	lr.w %0, %2\n"				\
+			"	bne  %0, %z3, 1f\n"			\
+			"	sc.w %1, %z4, %2\n"			\
+			"	bnez %1, 0b\n"				\
+			"1:\n"						\
+			: "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr)	\
+			: "rJ" (__old), "rJ" (__new)			\
+			: "memory");					\
+		break;							\
+	case 8:								\
+		__asm__ __volatile__ (					\
+			"0:	lr.d %0, %2\n"				\
+			"	bne %0, %z3, 1f\n"			\
+			"	sc.d %1, %z4, %2\n"			\
+			"	bnez %1, 0b\n"				\
+			"1:\n"						\
+			: "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr)	\
+			: "rJ" (__old), "rJ" (__new)			\
+			: "memory");					\
+		break;							\
+	default:							\
+		ASSERT_UNREACHABLE();					\
+	}								\
+	__ret;								\
+})
+
+#define cmpxchg_relaxed(ptr, o, n)					\
+({									\
+	__typeof__(*(ptr)) _o_ = (o);					\
+	__typeof__(*(ptr)) _n_ = (n);					\
+	(__typeof__(*(ptr))) __cmpxchg_relaxed((ptr),			\
+					_o_, _n_, sizeof(*(ptr)));	\
+})
+
+#define __cmpxchg_acquire(ptr, old, new, size)				\
+({									\
+	__typeof__(ptr) __ptr = (ptr);					\
+	__typeof__(*(ptr)) __old = (old);				\
+	__typeof__(*(ptr)) __new = (new);				\
+	__typeof__(*(ptr)) __ret;					\
+	register unsigned int __rc;					\
+	switch (size) {							\
+	case 4:								\
+		__asm__ __volatile__ (					\
+			"0:	lr.w %0, %2\n"				\
+			"	bne  %0, %z3, 1f\n"			\
+			"	sc.w %1, %z4, %2\n"			\
+			"	bnez %1, 0b\n"				\
+			RISCV_ACQUIRE_BARRIER				\
+			"1:\n"						\
+			: "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr)	\
+			: "rJ" (__old), "rJ" (__new)			\
+			: "memory");					\
+		break;							\
+	case 8:								\
+		__asm__ __volatile__ (					\
+			"0:	lr.d %0, %2\n"				\
+			"	bne %0, %z3, 1f\n"			\
+			"	sc.d %1, %z4, %2\n"			\
+			"	bnez %1, 0b\n"				\
+			RISCV_ACQUIRE_BARRIER				\
+			"1:\n"						\
+			: "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr)	\
+			: "rJ" (__old), "rJ" (__new)			\
+			: "memory");					\
+		break;							\
+	default:							\
+		ASSERT_UNREACHABLE();					\
+	}								\
+	__ret;								\
+})
+
+#define cmpxchg_acquire(ptr, o, n)					\
+({									\
+	__typeof__(*(ptr)) _o_ = (o);					\
+	__typeof__(*(ptr)) _n_ = (n);					\
+	(__typeof__(*(ptr))) __cmpxchg_acquire((ptr),			\
+					_o_, _n_, sizeof(*(ptr)));	\
+})
+
+#define __cmpxchg_release(ptr, old, new, size)				\
+({									\
+	__typeof__(ptr) __ptr = (ptr);					\
+	__typeof__(*(ptr)) __old = (old);				\
+	__typeof__(*(ptr)) __new = (new);				\
+	__typeof__(*(ptr)) __ret;					\
+	register unsigned int __rc;					\
+	switch (size) {							\
+	case 4:								\
+		__asm__ __volatile__ (					\
+			RISCV_RELEASE_BARRIER				\
+			"0:	lr.w %0, %2\n"				\
+			"	bne  %0, %z3, 1f\n"			\
+			"	sc.w %1, %z4, %2\n"			\
+			"	bnez %1, 0b\n"				\
+			"1:\n"						\
+			: "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr)	\
+			: "rJ" (__old), "rJ" (__new)			\
+			: "memory");					\
+		break;							\
+	case 8:								\
+		__asm__ __volatile__ (					\
+			RISCV_RELEASE_BARRIER				\
+			"0:	lr.d %0, %2\n"				\
+			"	bne %0, %z3, 1f\n"			\
+			"	sc.d %1, %z4, %2\n"			\
+			"	bnez %1, 0b\n"				\
+			"1:\n"						\
+			: "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr)	\
+			: "rJ" (__old), "rJ" (__new)			\
+			: "memory");					\
+		break;							\
+	default:							\
+		ASSERT_UNREACHABLE();					\
+	}								\
+	__ret;								\
+})
+
+#define cmpxchg_release(ptr, o, n)					\
+({									\
+	__typeof__(*(ptr)) _o_ = (o);					\
+	__typeof__(*(ptr)) _n_ = (n);					\
+	(__typeof__(*(ptr))) __cmpxchg_release((ptr),			\
+					_o_, _n_, sizeof(*(ptr)));	\
+})
+
+#define __cmpxchg(ptr, old, new, size)					\
+({									\
+	__typeof__(ptr) __ptr = (ptr);					\
+	__typeof__(*(ptr)) __old = (__typeof__(*(ptr)))(old);				\
+	__typeof__(*(ptr)) __new = (__typeof__(*(ptr)))(new);				\
+	__typeof__(*(ptr)) __ret;					\
+	register unsigned int __rc;					\
+	switch (size) {							\
+	case 4:								\
+		__asm__ __volatile__ (					\
+			"0:	lr.w %0, %2\n"				\
+			"	bne  %0, %z3, 1f\n"			\
+			"	sc.w.rl %1, %z4, %2\n"			\
+			"	bnez %1, 0b\n"				\
+			"	fence rw, rw\n"				\
+			"1:\n"						\
+			: "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr)	\
+			: "rJ" (__old), "rJ" (__new)			\
+			: "memory");					\
+		break;							\
+	case 8:								\
+		__asm__ __volatile__ (					\
+			"0:	lr.d %0, %2\n"				\
+			"	bne %0, %z3, 1f\n"			\
+			"	sc.d.rl %1, %z4, %2\n"			\
+			"	bnez %1, 0b\n"				\
+			"	fence rw, rw\n"				\
+			"1:\n"						\
+			: "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr)	\
+			: "rJ" (__old), "rJ" (__new)			\
+			: "memory");					\
+		break;							\
+	default:							\
+		__ret = 0; \
+		ASSERT_UNREACHABLE();					\
+	}								\
+	__ret;								\
+})
+
+#define cmpxchg(ptr, o, n)						\
+({									\
+	__typeof__(*(ptr)) _o_ = (o);					\
+	__typeof__(*(ptr)) _n_ = (n);					\
+	(__typeof__(*(ptr))) __cmpxchg((ptr),				\
+				       _o_, _n_, sizeof(*(ptr)));	\
+})
+
+#define cmpxchg_local(ptr, o, n)					\
+	(__cmpxchg_relaxed((ptr), (o), (n), sizeof(*(ptr))))
+
+#define cmpxchg32(ptr, o, n)						\
+({									\
+	BUILD_BUG_ON(sizeof(*(ptr)) != 4);				\
+	cmpxchg((ptr), (o), (n));					\
+})
+
+#define cmpxchg32_local(ptr, o, n)					\
+({									\
+	BUILD_BUG_ON(sizeof(*(ptr)) != 4);				\
+	cmpxchg_relaxed((ptr), (o), (n))				\
+})
+
+#define cmpxchg64(ptr, o, n)						\
+({									\
+	BUILD_BUG_ON(sizeof(*(ptr)) != 8);				\
+	cmpxchg((ptr), (o), (n));					\
+})
+
+#define cmpxchg64_local(ptr, o, n)					\
+({									\
+	BUILD_BUG_ON(sizeof(*(ptr)) != 8);				\
+	cmpxchg_relaxed((ptr), (o), (n));				\
+})
+
+#endif /* _ASM_RISCV_CMPXCHG_H */
-- 
2.42.0

[PATCH v2 19/39] xen/riscv: introduce asm/io.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/io.h | 134 ++++++++++++++++++++++++++++++++
 1 file changed, 134 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/io.h

diff --git a/xen/arch/riscv/include/asm/io.h b/xen/arch/riscv/include/asm/io.h
new file mode 100644
index 0000000000..987fddf902
--- /dev/null
+++ b/xen/arch/riscv/include/asm/io.h
@@ -0,0 +1,134 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Taken and modified from Linux.
+ *
+ * {read,write}{b,w,l,q} based on arch/arm64/include/asm/io.h
+ *   which was based on arch/arm/include/io.h
+ *
+ * Copyright (C) 1996-2000 Russell King
+ * Copyright (C) 2012 ARM Ltd.
+ * Copyright (C) 2014 Regents of the University of California
+ */
+
+#ifndef _ASM_RISCV_IO_H
+#define _ASM_RISCV_IO_H
+
+#include <asm/byteorder.h>
+
+/*
+ * The RISC-V ISA doesn't yet specify how to query or modify PMAs, so we can't
+ * change the properties of memory regions.  This should be fixed by the
+ * upcoming platform spec.
+ */
+#define ioremap_nocache(addr, size) ioremap((addr), (size))
+#define ioremap_wc(addr, size) ioremap((addr), (size))
+#define ioremap_wt(addr, size) ioremap((addr), (size))
+
+/* Generic IO read/write.  These perform native-endian accesses. */
+#define __raw_writeb __raw_writeb
+static inline void __raw_writeb(u8 val, volatile void __iomem *addr)
+{
+	asm volatile("sb %0, 0(%1)" : : "r" (val), "r" (addr));
+}
+
+#define __raw_writew __raw_writew
+static inline void __raw_writew(u16 val, volatile void __iomem *addr)
+{
+	asm volatile("sh %0, 0(%1)" : : "r" (val), "r" (addr));
+}
+
+#define __raw_writel __raw_writel
+static inline void __raw_writel(u32 val, volatile void __iomem *addr)
+{
+	asm volatile("sw %0, 0(%1)" : : "r" (val), "r" (addr));
+}
+
+#ifdef CONFIG_64BIT
+#define __raw_writeq __raw_writeq
+static inline void __raw_writeq(u64 val, volatile void __iomem *addr)
+{
+	asm volatile("sd %0, 0(%1)" : : "r" (val), "r" (addr));
+}
+#endif
+
+#define __raw_readb __raw_readb
+static inline u8 __raw_readb(const volatile void __iomem *addr)
+{
+	u8 val;
+
+	asm volatile("lb %0, 0(%1)" : "=r" (val) : "r" (addr));
+	return val;
+}
+
+#define __raw_readw __raw_readw
+static inline u16 __raw_readw(const volatile void __iomem *addr)
+{
+	u16 val;
+
+	asm volatile("lh %0, 0(%1)" : "=r" (val) : "r" (addr));
+	return val;
+}
+
+#define __raw_readl __raw_readl
+static inline u32 __raw_readl(const volatile void __iomem *addr)
+{
+	u32 val;
+
+	asm volatile("lw %0, 0(%1)" : "=r" (val) : "r" (addr));
+	return val;
+}
+
+#ifdef CONFIG_64BIT
+#define __raw_readq __raw_readq
+static inline u64 __raw_readq(const volatile void __iomem *addr)
+{
+	u64 val;
+
+	asm volatile("ld %0, 0(%1)" : "=r" (val) : "r" (addr));
+	return val;
+}
+#endif
+
+/*
+ * Unordered I/O memory access primitives.  These are even more relaxed than
+ * the relaxed versions, as they don't even order accesses between successive
+ * operations to the I/O regions.
+ */
+#define readb_cpu(c)		({ u8  __r = __raw_readb(c); __r; })
+#define readw_cpu(c)		({ u16 __r = le16_to_cpu((__force __le16)__raw_readw(c)); __r; })
+#define readl_cpu(c)		({ u32 __r = le32_to_cpu((__force __le32)__raw_readl(c)); __r; })
+
+#define writeb_cpu(v,c)		((void)__raw_writeb((v),(c)))
+#define writew_cpu(v,c)		((void)__raw_writew((__force u16)cpu_to_le16(v),(c)))
+#define writel_cpu(v,c)		((void)__raw_writel((__force u32)cpu_to_le32(v),(c)))
+
+#ifdef CONFIG_64BIT
+#define readq_cpu(c)		({ u64 __r = le64_to_cpu((__force __le64)__raw_readq(c)); __r; })
+#define writeq_cpu(v,c)		((void)__raw_writeq((__force u64)cpu_to_le64(v),(c)))
+#endif
+
+/*
+ * I/O memory access primitives. Reads are ordered relative to any
+ * following Normal memory access. Writes are ordered relative to any prior
+ * Normal memory access.  The memory barriers here are necessary as RISC-V
+ * doesn't define any ordering between the memory space and the I/O space.
+ */
+#define __io_br()	do {} while (0)
+#define __io_ar(v)	__asm__ __volatile__ ("fence i,r" : : : "memory");
+#define __io_bw()	__asm__ __volatile__ ("fence w,o" : : : "memory");
+#define __io_aw()	do { } while (0)
+
+#define readb(c)	({ u8  __v; __io_br(); __v = readb_cpu(c); __io_ar(__v); __v; })
+#define readw(c)	({ u16 __v; __io_br(); __v = readw_cpu(c); __io_ar(__v); __v; })
+#define readl(c)	({ u32 __v; __io_br(); __v = readl_cpu(c); __io_ar(__v); __v; })
+
+#define writeb(v,c)	({ __io_bw(); writeb_cpu((v),(c)); __io_aw(); })
+#define writew(v,c)	({ __io_bw(); writew_cpu((v),(c)); __io_aw(); })
+#define writel(v,c)	({ __io_bw(); writel_cpu((v),(c)); __io_aw(); })
+
+#ifdef CONFIG_64BIT
+#define readq(c)	({ u64 __v; __io_br(); __v = readq_cpu(c); __io_ar(__v); __v; })
+#define writeq(v,c)	({ __io_bw(); writeq_cpu((v),(c)); __io_aw(); })
+#endif
+
+#endif /* _ASM_RISCV_IO_H */
-- 
2.42.0

[PATCH v2 20/39] xen/riscv: define bug frame tables in xen.lds.S

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/xen.lds.S | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/xen/arch/riscv/xen.lds.S b/xen/arch/riscv/xen.lds.S
index 3fa7db3bf9..a10e0ad87c 100644
--- a/xen/arch/riscv/xen.lds.S
+++ b/xen/arch/riscv/xen.lds.S
@@ -45,6 +45,16 @@ SECTIONS
     . = ALIGN(PAGE_SIZE);
     .rodata : {
         _srodata = .;          /* Read-only data */
+        /* Bug frames table */
+        __start_bug_frames = .;
+        *(.bug_frames.0)
+        __stop_bug_frames_0 = .;
+        *(.bug_frames.1)
+        __stop_bug_frames_1 = .;
+        *(.bug_frames.2)
+        __stop_bug_frames_2 = .;
+        *(.bug_frames.3)
+        __stop_bug_frames_3 = .;
         *(.rodata)
         *(.rodata.*)
         *(.data.rel.ro)
-- 
2.42.0

[PATCH v2 21/39] xen/riscv: introduce bit operations

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/Makefile            |   1 +
 xen/arch/riscv/lib/Makefile        |   1 +
 xen/arch/riscv/lib/find_next_bit.c | 278 +++++++++++++++++++++++++++++
 3 files changed, 280 insertions(+)
 create mode 100644 xen/arch/riscv/lib/Makefile
 create mode 100644 xen/arch/riscv/lib/find_next_bit.c

diff --git a/xen/arch/riscv/Makefile b/xen/arch/riscv/Makefile
index 2fefe14e7c..b08048948f 100644
--- a/xen/arch/riscv/Makefile
+++ b/xen/arch/riscv/Makefile
@@ -1,5 +1,6 @@
 obj-$(CONFIG_EARLY_PRINTK) += early_printk.o
 obj-y += entry.o
+obj-y += lib/
 obj-y += mm.o
 obj-$(CONFIG_RISCV_64) += riscv64/
 obj-y += sbi.o
diff --git a/xen/arch/riscv/lib/Makefile b/xen/arch/riscv/lib/Makefile
new file mode 100644
index 0000000000..6fae6a1f10
--- /dev/null
+++ b/xen/arch/riscv/lib/Makefile
@@ -0,0 +1 @@
+obj-y += find_next_bit.o
diff --git a/xen/arch/riscv/lib/find_next_bit.c b/xen/arch/riscv/lib/find_next_bit.c
new file mode 100644
index 0000000000..ac5a4687e3
--- /dev/null
+++ b/xen/arch/riscv/lib/find_next_bit.c
@@ -0,0 +1,278 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * find_next_bit.c: fallback find next bit implementation
+ *
+ * Copyright (C) 2004 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells (dhowells@redhat.com)
+ */
+#include <xen/bitops.h>
+#include <asm/types.h>
+#include <asm/byteorder.h>
+
+#ifndef find_next_bit
+/*
+ * Find the next set bit in a memory region.
+ */
+unsigned long find_next_bit(const unsigned long *addr, unsigned long size,
+			    unsigned long offset)
+{
+	const unsigned long *p = addr + BIT_WORD(offset);
+	unsigned long result = offset & ~(BITS_PER_LONG-1);
+	unsigned long tmp;
+
+	if (offset >= size)
+		return size;
+	size -= result;
+	offset %= BITS_PER_LONG;
+	if (offset) {
+		tmp = *(p++);
+		tmp &= (~0UL << offset);
+		if (size < BITS_PER_LONG)
+			goto found_first;
+		if (tmp)
+			goto found_middle;
+		size -= BITS_PER_LONG;
+		result += BITS_PER_LONG;
+	}
+	while (size & ~(BITS_PER_LONG-1)) {
+		if ((tmp = *(p++)))
+			goto found_middle;
+		result += BITS_PER_LONG;
+		size -= BITS_PER_LONG;
+	}
+	if (!size)
+		return result;
+	tmp = *p;
+
+found_first:
+	tmp &= (~0UL >> (BITS_PER_LONG - size));
+	if (tmp == 0UL)		/* Are any bits set? */
+		return result + size;	/* Nope. */
+found_middle:
+	return result + __ffs(tmp);
+}
+EXPORT_SYMBOL(find_next_bit);
+#endif
+
+#ifndef find_next_zero_bit
+/*
+ * This implementation of find_{first,next}_zero_bit was stolen from
+ * Linus' asm-alpha/bitops.h.
+ */
+unsigned long find_next_zero_bit(const unsigned long *addr, unsigned long size,
+				 unsigned long offset)
+{
+	const unsigned long *p = addr + BIT_WORD(offset);
+	unsigned long result = offset & ~(BITS_PER_LONG-1);
+	unsigned long tmp;
+
+	if (offset >= size)
+		return size;
+	size -= result;
+	offset %= BITS_PER_LONG;
+	if (offset) {
+		tmp = *(p++);
+		tmp |= ~0UL >> (BITS_PER_LONG - offset);
+		if (size < BITS_PER_LONG)
+			goto found_first;
+		if (~tmp)
+			goto found_middle;
+		size -= BITS_PER_LONG;
+		result += BITS_PER_LONG;
+	}
+	while (size & ~(BITS_PER_LONG-1)) {
+		if (~(tmp = *(p++)))
+			goto found_middle;
+		result += BITS_PER_LONG;
+		size -= BITS_PER_LONG;
+	}
+	if (!size)
+		return result;
+	tmp = *p;
+
+found_first:
+	tmp |= ~0UL << size;
+	if (tmp == ~0UL)	/* Are any bits zero? */
+		return result + size;	/* Nope. */
+found_middle:
+	return result + ffz(tmp);
+}
+EXPORT_SYMBOL(find_next_zero_bit);
+#endif
+
+#ifndef find_first_bit
+/*
+ * Find the first set bit in a memory region.
+ */
+unsigned long find_first_bit(const unsigned long *addr, unsigned long size)
+{
+	const unsigned long *p = addr;
+	unsigned long result = 0;
+	unsigned long tmp;
+
+	while (size & ~(BITS_PER_LONG-1)) {
+		if ((tmp = *(p++)))
+			goto found;
+		result += BITS_PER_LONG;
+		size -= BITS_PER_LONG;
+	}
+	if (!size)
+		return result;
+
+	tmp = (*p) & (~0UL >> (BITS_PER_LONG - size));
+	if (tmp == 0UL)		/* Are any bits set? */
+		return result + size;	/* Nope. */
+found:
+	return result + __ffs(tmp);
+}
+EXPORT_SYMBOL(find_first_bit);
+#endif
+
+#ifndef find_first_zero_bit
+/*
+ * Find the first cleared bit in a memory region.
+ */
+unsigned long find_first_zero_bit(const unsigned long *addr, unsigned long size)
+{
+	const unsigned long *p = addr;
+	unsigned long result = 0;
+	unsigned long tmp;
+
+	while (size & ~(BITS_PER_LONG-1)) {
+		if (~(tmp = *(p++)))
+			goto found;
+		result += BITS_PER_LONG;
+		size -= BITS_PER_LONG;
+	}
+	if (!size)
+		return result;
+
+	tmp = (*p) | (~0UL << size);
+	if (tmp == ~0UL)	/* Are any bits zero? */
+		return result + size;	/* Nope. */
+found:
+	return result + ffz(tmp);
+}
+EXPORT_SYMBOL(find_first_zero_bit);
+#endif
+
+#ifdef __BIG_ENDIAN
+
+/* include/linux/byteorder does not support "unsigned long" type */
+static inline unsigned long ext2_swabp(const unsigned long * x)
+{
+#if BITS_PER_LONG == 64
+	return (unsigned long) __swab64p((u64 *) x);
+#elif BITS_PER_LONG == 32
+	return (unsigned long) __swab32p((u32 *) x);
+#else
+#error BITS_PER_LONG not defined
+#endif
+}
+
+/* include/linux/byteorder doesn't support "unsigned long" type */
+static inline unsigned long ext2_swab(const unsigned long y)
+{
+#if BITS_PER_LONG == 64
+	return (unsigned long) __swab64((u64) y);
+#elif BITS_PER_LONG == 32
+	return (unsigned long) __swab32((u32) y);
+#else
+#error BITS_PER_LONG not defined
+#endif
+}
+
+#ifndef find_next_zero_bit_le
+unsigned long find_next_zero_bit_le(const void *addr, unsigned
+		long size, unsigned long offset)
+{
+	const unsigned long *p = addr;
+	unsigned long result = offset & ~(BITS_PER_LONG - 1);
+	unsigned long tmp;
+
+	if (offset >= size)
+		return size;
+	p += BIT_WORD(offset);
+	size -= result;
+	offset &= (BITS_PER_LONG - 1UL);
+	if (offset) {
+		tmp = ext2_swabp(p++);
+		tmp |= (~0UL >> (BITS_PER_LONG - offset));
+		if (size < BITS_PER_LONG)
+			goto found_first;
+		if (~tmp)
+			goto found_middle;
+		size -= BITS_PER_LONG;
+		result += BITS_PER_LONG;
+	}
+
+	while (size & ~(BITS_PER_LONG - 1)) {
+		if (~(tmp = *(p++)))
+			goto found_middle_swap;
+		result += BITS_PER_LONG;
+		size -= BITS_PER_LONG;
+	}
+	if (!size)
+		return result;
+	tmp = ext2_swabp(p);
+found_first:
+	tmp |= ~0UL << size;
+	if (tmp == ~0UL)	/* Are any bits zero? */
+		return result + size; /* Nope. Skip ffz */
+found_middle:
+	return result + ffz(tmp);
+
+found_middle_swap:
+	return result + ffz(ext2_swab(tmp));
+}
+EXPORT_SYMBOL(find_next_zero_bit_le);
+#endif
+
+#ifndef find_next_bit_le
+unsigned long find_next_bit_le(const void *addr, unsigned
+		long size, unsigned long offset)
+{
+	const unsigned long *p = addr;
+	unsigned long result = offset & ~(BITS_PER_LONG - 1);
+	unsigned long tmp;
+
+	if (offset >= size)
+		return size;
+	p += BIT_WORD(offset);
+	size -= result;
+	offset &= (BITS_PER_LONG - 1UL);
+	if (offset) {
+		tmp = ext2_swabp(p++);
+		tmp &= (~0UL << offset);
+		if (size < BITS_PER_LONG)
+			goto found_first;
+		if (tmp)
+			goto found_middle;
+		size -= BITS_PER_LONG;
+		result += BITS_PER_LONG;
+	}
+
+	while (size & ~(BITS_PER_LONG - 1)) {
+		tmp = *(p++);
+		if (tmp)
+			goto found_middle_swap;
+		result += BITS_PER_LONG;
+		size -= BITS_PER_LONG;
+	}
+	if (!size)
+		return result;
+	tmp = ext2_swabp(p);
+found_first:
+	tmp &= (~0UL >> (BITS_PER_LONG - size));
+	if (tmp == 0UL)		/* Are any bits set? */
+		return result + size; /* Nope. */
+found_middle:
+	return result + __ffs(tmp);
+
+found_middle_swap:
+	return result + __ffs(ext2_swab(tmp));
+}
+EXPORT_SYMBOL(find_next_bit_le);
+#endif
+
+#endif /* __BIG_ENDIAN */
-- 
2.42.0

[PATCH v2 22/39] xen/riscv: introduce asm/domain.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/domain.h | 43 +++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/domain.h

diff --git a/xen/arch/riscv/include/asm/domain.h b/xen/arch/riscv/include/asm/domain.h
new file mode 100644
index 0000000000..f2dccf370f
--- /dev/null
+++ b/xen/arch/riscv/include/asm/domain.h
@@ -0,0 +1,43 @@
+#ifndef __ASM_RISCV_DOMAIN_H__
+#define __ASM_RISCV_DOMAIN_H__
+
+#include <xen/xmalloc.h>
+#include <public/hvm/params.h>
+
+struct hvm_domain
+{
+    uint64_t              params[HVM_NR_PARAMS];
+};
+
+#define is_domain_direct_mapped(d) ((void)(d), 0)
+
+struct arch_vcpu_io {
+};
+
+struct arch_vcpu {
+};
+
+struct arch_domain {
+    struct hvm_domain hvm;
+};
+
+#include <xen/sched.h>
+
+static inline struct vcpu_guest_context *alloc_vcpu_guest_context(void)
+{
+    return xmalloc(struct vcpu_guest_context);
+}
+
+static inline void free_vcpu_guest_context(struct vcpu_guest_context *vgc)
+{
+    xfree(vgc);
+}
+
+struct guest_memory_policy {};
+static inline void update_guest_memory_policy(struct vcpu *v,
+                                              struct guest_memory_policy *gmp)
+{}
+
+static inline void arch_vcpu_block(struct vcpu *v) {}
+
+#endif /* __ASM_RISCV_DOMAIN_H__ */
-- 
2.42.0

[PATCH v2 23/39] xen/riscv: introduce asm/guest_access.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - change xen/mm.h to xen/types.h
---
 xen/arch/riscv/include/asm/guest_access.h | 29 +++++++++++++++++++++++
 1 file changed, 29 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/guest_access.h

diff --git a/xen/arch/riscv/include/asm/guest_access.h b/xen/arch/riscv/include/asm/guest_access.h
new file mode 100644
index 0000000000..122ecdc048
--- /dev/null
+++ b/xen/arch/riscv/include/asm/guest_access.h
@@ -0,0 +1,29 @@
+#ifndef __ASM_RISCV_GUEST_ACCESS_H__
+#define __ASM_RISCV_GUEST_ACCESS_H__
+
+#include <xen/types.h>
+
+unsigned long raw_copy_to_guest(void *to, const void *from, unsigned len);
+unsigned long raw_copy_from_guest(void *to, const void *from, unsigned len);
+
+#define __raw_copy_to_guest raw_copy_to_guest
+#define __raw_copy_from_guest raw_copy_from_guest
+
+#define guest_handle_okay(hnd, nr) (1)
+#define guest_handle_subrange_okay(hnd, first, last) (1)
+
+struct domain;
+unsigned long copy_to_guest_phys(struct domain *d,
+                                 paddr_t gpa,
+                                 void *buf,
+                                 unsigned int len);
+
+#endif /* __ASM_RISCV_GUEST_ACCESS_H__ */
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
-- 
2.42.0

[PATCH v2 24/39] xen/riscv: introduce asm/irq.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
	- add ifdef CONFIG_HAS_DEVICE_TREE for things that shouldn't be
      in case !CONFIG_HAS_DEVICE_TREE
	- use proper includes.
---
 xen/arch/riscv/include/asm/irq.h | 37 ++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/irq.h

diff --git a/xen/arch/riscv/include/asm/irq.h b/xen/arch/riscv/include/asm/irq.h
new file mode 100644
index 0000000000..ca49d3f9b4
--- /dev/null
+++ b/xen/arch/riscv/include/asm/irq.h
@@ -0,0 +1,37 @@
+#ifndef __ASM_RISCV_IRQ_H__
+#define __ASM_RISCV_IRQ_H__
+
+#include <xen/bug.h>
+#ifdef CONFIG_HAS_DEVICE_TREE
+#include <xen/device_tree.h>
+#endif
+
+/* TODO */
+#define nr_irqs 0U
+#define nr_static_irqs 0
+#define arch_hwdom_irqs(domid) 0U
+
+#define domain_pirq_to_irq(d, pirq) (pirq)
+
+#define arch_evtchn_bind_pirq(d, pirq) ((void)((d) + (pirq)))
+
+struct arch_pirq {
+};
+
+struct arch_irq_desc {
+    unsigned int type;
+};
+
+static inline void arch_move_irqs(struct vcpu *v)
+{
+    BUG();
+}
+
+#ifdef CONFIG_HAS_DEVICE_TREE
+static inline int platform_get_irq(const struct dt_device_node *device, int index)
+{
+    BUG();
+}
+#endif
+
+#endif /* __ASM_RISCV_IRQ_H__ */
-- 
2.42.0

[PATCH v2 25/39] xen/riscv: introduce asm/p2m.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/p2m.h | 105 +++++++++++++++++++++++++++++++
 1 file changed, 105 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/p2m.h

diff --git a/xen/arch/riscv/include/asm/p2m.h b/xen/arch/riscv/include/asm/p2m.h
new file mode 100644
index 0000000000..993aec6d2c
--- /dev/null
+++ b/xen/arch/riscv/include/asm/p2m.h
@@ -0,0 +1,105 @@
+#ifndef __ASM_RISCV_P2M_H__
+#define __ASM_RISCV_P2M_H__
+
+#include <asm/page-bits.h>
+
+#define paddr_bits PADDR_BITS
+
+/*
+ * List of possible type for each page in the p2m entry.
+ * The number of available bit per page in the pte for this purpose is 4 bits.
+ * So it's possible to only have 16 fields. If we run out of value in the
+ * future, it's possible to use higher value for pseudo-type and don't store
+ * them in the p2m entry.
+ */
+typedef enum {
+    p2m_invalid = 0,    /* Nothing mapped here */
+    p2m_ram_rw,         /* Normal read/write guest RAM */
+    p2m_ram_ro,         /* Read-only; writes are silently dropped */
+    p2m_mmio_direct_dev,/* Read/write mapping of genuine Device MMIO area */
+    p2m_mmio_direct_nc, /* Read/write mapping of genuine MMIO area non-cacheable */
+    p2m_mmio_direct_c,  /* Read/write mapping of genuine MMIO area cacheable */
+    p2m_map_foreign_rw, /* Read/write RAM pages from foreign domain */
+    p2m_map_foreign_ro, /* Read-only RAM pages from foreign domain */
+    p2m_grant_map_rw,   /* Read/write grant mapping */
+    p2m_grant_map_ro,   /* Read-only grant mapping */
+    /* The types below are only used to decide the page attribute in the P2M */
+    p2m_iommu_map_rw,   /* Read/write iommu mapping */
+    p2m_iommu_map_ro,   /* Read-only iommu mapping */
+    p2m_max_real_type,  /* Types after this won't be store in the p2m */
+} p2m_type_t;
+
+#include <xen/p2m-common.h>
+
+static inline int get_page_and_type(struct page_info *page,
+                                    struct domain *domain,
+                                    unsigned long type)
+{
+    BUG();
+    return 1;
+}
+
+/* Look up a GFN and take a reference count on the backing page. */
+typedef unsigned int p2m_query_t;
+#define P2M_ALLOC    (1u<<0)   /* Populate PoD and paged-out entries */
+#define P2M_UNSHARE  (1u<<1)   /* Break CoW sharing */
+
+static inline struct page_info *get_page_from_gfn(
+    struct domain *d, unsigned long gfn, p2m_type_t *t, p2m_query_t q)
+{
+    BUG();
+    return NULL;
+}
+
+static inline void memory_type_changed(struct domain *d)
+{
+    BUG();
+}
+
+
+static inline int guest_physmap_mark_populate_on_demand(struct domain *d, unsigned long gfn,
+                                                        unsigned int order)
+{
+    BUG();
+    return 1;
+}
+
+static inline int guest_physmap_add_entry(struct domain *d,
+                            gfn_t gfn,
+                            mfn_t mfn,
+                            unsigned long page_order,
+                            p2m_type_t t)
+{
+    BUG();
+    return 1;
+}
+
+/* Untyped version for RAM only, for compatibility */
+static inline int __must_check
+guest_physmap_add_page(struct domain *d, gfn_t gfn, mfn_t mfn,
+                       unsigned int page_order)
+{
+    return guest_physmap_add_entry(d, gfn, mfn, page_order, p2m_ram_rw);
+}
+
+static inline mfn_t gfn_to_mfn(struct domain *d, gfn_t gfn)
+{
+    BUG();
+    return _mfn(0);
+}
+
+static inline bool arch_acquire_resource_check(struct domain *d)
+{
+    /*
+     * The reference counting of foreign entries in set_foreign_p2m_entry()
+     * is supported on RISCV.
+     */
+    return true;
+}
+
+static inline void p2m_altp2m_check(struct vcpu *v, uint16_t idx)
+{
+    /* Not supported on RISCV. */
+}
+
+#endif /* __ASM_RISCV_P2M_H__ */
\ No newline at end of file
-- 
2.42.0

[PATCH v2 26/39] xen/riscv: introduce asm/regs.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - change xen/lib.h to xen/bug.h
 - remove unnecessary empty line
---
 xen/arch/riscv/include/asm/regs.h | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/regs.h

diff --git a/xen/arch/riscv/include/asm/regs.h b/xen/arch/riscv/include/asm/regs.h
new file mode 100644
index 0000000000..8556837c65
--- /dev/null
+++ b/xen/arch/riscv/include/asm/regs.h
@@ -0,0 +1,26 @@
+#ifndef __ARM_RISCV_REGS_H__
+#define __ARM_RISCV_REGS_H__
+
+#ifndef __ASSEMBLY__
+
+#include <xen/bug.h>
+#include <asm/current.h>
+
+#define hyp_mode(r)     (0)
+
+static inline bool guest_mode(const struct cpu_user_regs *r)
+{
+    BUG();
+}
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* __ARM_RISCV_REGS_H__ */
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
-- 
2.42.0

[PATCH v2 27/39] xen/riscv: introduce asm/time.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 -  change xen/lib.h to xen/bug.h
 - remove inclusion of <asm/processor.h> as it's not needed.
---
 xen/arch/riscv/include/asm/time.h | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/time.h

diff --git a/xen/arch/riscv/include/asm/time.h b/xen/arch/riscv/include/asm/time.h
new file mode 100644
index 0000000000..3ce848deac
--- /dev/null
+++ b/xen/arch/riscv/include/asm/time.h
@@ -0,0 +1,19 @@
+#ifndef __ASM_RISCV_TIME_H__
+#define __ASM_RISCV_TIME_H__
+
+#include <xen/bug.h>
+#include <asm/csr.h>
+
+struct vcpu;
+
+/* TODO: implement */
+static inline void force_update_vcpu_system_time(struct vcpu *v) { BUG(); }
+
+typedef unsigned long cycles_t;
+
+static inline cycles_t get_cycles(void)
+{
+	return csr_read(CSR_TIME);
+}
+
+#endif /* __ASM_RISCV_TIME_H__ */
\ No newline at end of file
-- 
2.42.0

[PATCH v2 28/39] xen/riscv: introduce asm/event.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/event.h | 34 ++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
 create mode 100644 xen/arch/riscv/include/asm/event.h

diff --git a/xen/arch/riscv/include/asm/event.h b/xen/arch/riscv/include/asm/event.h
new file mode 100644
index 0000000000..48630c77b3
--- /dev/null
+++ b/xen/arch/riscv/include/asm/event.h
@@ -0,0 +1,34 @@
+#ifndef __ASM_RISCV_EVENT_H__
+#define __ASM_RISCV_EVENT_H__
+
+void vcpu_mark_events_pending(struct vcpu *v);
+
+static inline int vcpu_event_delivery_is_enabled(struct vcpu *v)
+{
+    return 0;
+}
+
+static inline int local_events_need_delivery(void)
+{
+    return 0;
+}
+
+static inline void local_event_delivery_enable(void)
+{
+}
+
+/* No arch specific virq definition now. Default to global. */
+static inline bool arch_virq_is_global(unsigned int virq)
+{
+    return true;
+}
+
+#endif
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
-- 
2.42.0

[PATCH v2 29/39] xen/riscv: add definition of __read_mostly

[Oleksii Kurochko]

The definition of __read_mostly should be removed in:
https://lore.kernel.org/xen-devel/f25eb5c9-7c14-6e23-8535-2c66772b333e@suse.com/

The patch introduces it in arch-specific header to not
block enabling of full Xen build for RISC-V.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
 - update the commit message
---
 xen/arch/riscv/include/asm/cache.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/xen/arch/riscv/include/asm/cache.h b/xen/arch/riscv/include/asm/cache.h
index 69573eb051..94bd94db53 100644
--- a/xen/arch/riscv/include/asm/cache.h
+++ b/xen/arch/riscv/include/asm/cache.h
@@ -3,4 +3,6 @@
 #ifndef _ASM_RISCV_CACHE_H
 #define _ASM_RISCV_CACHE_H
 
+#define __read_mostly __section(".data.read_mostly")
+
 #endif /* _ASM_RISCV_CACHE_H */
-- 
2.42.0

[PATCH v2 30/39] xen/riscv: define an address of frame table

[Oleksii Kurochko]

Also the patchs adds some helpful macros.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/config.h | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/xen/arch/riscv/include/asm/config.h b/xen/arch/riscv/include/asm/config.h
index f0544c6a20..8d2103b3ce 100644
--- a/xen/arch/riscv/include/asm/config.h
+++ b/xen/arch/riscv/include/asm/config.h
@@ -77,12 +77,31 @@
   name:
 #endif
 
+#define VPN_BITS    (9)
+#define OFFSET_BITS (12)
+
 #ifdef CONFIG_RISCV_64
+
+#define SLOTN_ENTRY_BITS        (HYP_PT_ROOT_LEVEL * VPN_BITS + OFFSET_BITS)
+#define SLOTN(slot)             (_AT(vaddr_t,slot) << SLOTN_ENTRY_BITS)
+#define SLOTN_ENTRY_SIZE        SLOTN(1)
+
 #define XEN_VIRT_START 0xFFFFFFFFC0000000 /* (_AC(-1, UL) + 1 - GB(1)) */
+
+#define FRAMETABLE_VIRT_START   SLOTN(196)
+#define FRAMETABLE_SIZE         GB(3)
+#define FRAMETABLE_NR           (FRAMETABLE_SIZE / sizeof(*frame_table))
+#define FRAMETABLE_VIRT_END     (FRAMETABLE_VIRT_START + FRAMETABLE_SIZE - 1)
+
+#define VMAP_VIRT_START         SLOTN(194)
+#define VMAP_VIRT_SIZE          GB(1)
+
 #else
 #error "RV32 isn't supported"
 #endif
 
+#define HYPERVISOR_VIRT_START XEN_VIRT_START
+
 #define SMP_CACHE_BYTES (1 << 6)
 
 #define STACK_SIZE PAGE_SIZE
@@ -95,6 +114,8 @@
 #define RV_STAGE1_MODE SATP_MODE_SV32
 #endif
 
+#define HYP_PT_ROOT_LEVEL (CONFIG_PAGING_LEVELS - 1)
+
 #define IDENT_AREA_SIZE 64
 
 #endif /* __RISCV_CONFIG_H__ */
-- 
2.42.0

[PATCH v2 31/39] xen/riscv: add required things to asm/current.h

[Oleksii Kurochko]

Add minimal requied things to be able to build full Xen.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/current.h | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/xen/arch/riscv/include/asm/current.h b/xen/arch/riscv/include/asm/current.h
index d84f15dc50..e105329bf3 100644
--- a/xen/arch/riscv/include/asm/current.h
+++ b/xen/arch/riscv/include/asm/current.h
@@ -3,6 +3,22 @@
 #ifndef __ASM_CURRENT_H
 #define __ASM_CURRENT_H
 
+#include <xen/percpu.h>
+#include <asm/processor.h>
+
+#ifndef __ASSEMBLY__
+
+struct vcpu;
+
+/* Which VCPU is "current" on this PCPU. */
+DECLARE_PER_CPU(struct vcpu *, curr_vcpu);
+
+#define current            (this_cpu(curr_vcpu))
+#define set_current(vcpu)  do { current = (vcpu); } while (0)
+#define get_cpu_current(cpu)  (per_cpu(curr_vcpu, cpu))
+
+#define guest_cpu_user_regs() (0)
+
 #define switch_stack_and_jump(stack, fn) do {               \
     asm volatile (                                          \
             "mv sp, %0\n"                                   \
@@ -10,4 +26,8 @@
     unreachable();                                          \
 } while ( false )
 
+#define get_per_cpu_offset() __per_cpu_offset[get_processor_id()]
+
+#endif /* __ASSEMBLY__ */
+
 #endif /* __ASM_CURRENT_H */
-- 
2.42.0

[PATCH v2 32/39] xen/riscv: add minimal stuff to asm/page.h to build full Xen

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/page.h | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/xen/arch/riscv/include/asm/page.h b/xen/arch/riscv/include/asm/page.h
index 95074e29b3..abbae75aaf 100644
--- a/xen/arch/riscv/include/asm/page.h
+++ b/xen/arch/riscv/include/asm/page.h
@@ -6,6 +6,7 @@
 #ifndef __ASSEMBLY__
 
 #include <xen/const.h>
+#include <xen/bug.h>
 #include <xen/types.h>
 
 #include <asm/mm.h>
@@ -32,6 +33,9 @@
 #define PTE_LEAF_DEFAULT            (PTE_VALID | PTE_READABLE | PTE_WRITABLE)
 #define PTE_TABLE                   (PTE_VALID)
 
+/* TODO */
+#define PAGE_HYPERVISOR 0
+
 /* Calculate the offsets into the pagetables for a given VA */
 #define pt_linear_offset(lvl, va)   ((va) >> XEN_PT_LEVEL_SHIFT(lvl))
 
@@ -46,6 +50,9 @@ typedef struct {
 #endif
 } pte_t;
 
+#define pfn_to_paddr(pfn) ((paddr_t)(pfn) << PAGE_SHIFT)
+#define paddr_to_pfn(pa)  ((unsigned long)((pa) >> PAGE_SHIFT))
+
 static inline pte_t paddr_to_pte(paddr_t paddr,
                                  unsigned int permissions)
 {
@@ -62,6 +69,20 @@ static inline bool pte_is_valid(pte_t p)
     return p.pte & PTE_VALID;
 }
 
+static inline void invalidate_icache(void)
+{
+    BUG();
+}
+
+#define clear_page(page) memset((void *)(page), 0, PAGE_SIZE)
+#define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE)
+
+/* TODO: Flush the dcache for an entire page. */
+static inline void flush_page_to_ram(unsigned long mfn, bool sync_icache)
+{
+    BUG();
+}
+
 #endif /* __ASSEMBLY__ */
 
 #endif /* _ASM_RISCV_PAGE_H */
-- 
2.42.0

[PATCH v2 33/39] xen/riscv: add minimal stuff to asm/processor.h to build full Xen

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/include/asm/processor.h | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/xen/arch/riscv/include/asm/processor.h b/xen/arch/riscv/include/asm/processor.h
index 6db681d805..b6218a00a7 100644
--- a/xen/arch/riscv/include/asm/processor.h
+++ b/xen/arch/riscv/include/asm/processor.h
@@ -12,6 +12,9 @@
 
 #ifndef __ASSEMBLY__
 
+/* TODO: need to be implemeted */
+#define get_processor_id() 0
+
 /* On stack VCPU state */
 struct cpu_user_regs
 {
@@ -53,6 +56,18 @@ struct cpu_user_regs
     unsigned long pregs;
 };
 
+/* TODO: need to implement */
+#define cpu_to_core(_cpu)   (0)
+#define cpu_to_socket(_cpu) (0)
+
+static inline void cpu_relax(void)
+{
+	int dummy;
+	/* In lieu of a halt instruction, induce a long-latency stall. */
+	__asm__ __volatile__ ("div %0, %0, zero" : "=r" (dummy));
+	barrier();
+}
+
 static inline void wfi(void)
 {
     __asm__ __volatile__ ("wfi");
-- 
2.42.0

[PATCH v2 34/39] xen: add RISCV support for pmu.h

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/include/public/pmu.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/xen/include/public/pmu.h b/xen/include/public/pmu.h
index 5a176b6ac3..af8b7babdd 100644
--- a/xen/include/public/pmu.h
+++ b/xen/include/public/pmu.h
@@ -13,6 +13,8 @@
 #include "arch-arm.h"
 #elif defined (__powerpc64__)
 #include "arch-ppc.h"
+#elif defined(__riscv)
+#include "arch-riscv.h"
 #else
 #error "Unsupported architecture"
 #endif
-- 
2.42.0

[PATCH v2 35/39] xen: add necessary headers to common to build full Xen for RISC-V

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/include/xen/domain.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/xen/include/xen/domain.h b/xen/include/xen/domain.h
index 54d88bf5e3..a4e11342e3 100644
--- a/xen/include/xen/domain.h
+++ b/xen/include/xen/domain.h
@@ -4,6 +4,7 @@
 
 #include <xen/types.h>
 
+#include <public/domctl.h>
 #include <public/xen.h>
 
 struct guest_area {
-- 
2.42.0

[PATCH v2 36/39] xen/riscv: add minimal stuff to asm/mm.h to build full Xen

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - define stub for arch_get_dma_bitsize(void)
---
 xen/arch/riscv/include/asm/mm.h | 270 +++++++++++++++++++++++++++++++-
 1 file changed, 268 insertions(+), 2 deletions(-)

diff --git a/xen/arch/riscv/include/asm/mm.h b/xen/arch/riscv/include/asm/mm.h
index 07c7a0abba..68460f6659 100644
--- a/xen/arch/riscv/include/asm/mm.h
+++ b/xen/arch/riscv/include/asm/mm.h
@@ -3,10 +3,271 @@
 #ifndef _ASM_RISCV_MM_H
 #define _ASM_RISCV_MM_H
 
+#include <public/xen.h>
+#include <xen/pdx.h>
+#include <xen/types.h>
+
+#include <asm/page.h>
 #include <asm/page-bits.h>
 
-#define pfn_to_paddr(pfn) ((paddr_t)(pfn) << PAGE_SHIFT)
-#define paddr_to_pfn(pa)  ((unsigned long)((pa) >> PAGE_SHIFT))
+#define paddr_to_pdx(pa)    mfn_to_pdx(maddr_to_mfn(pa))
+#define gfn_to_gaddr(gfn)   pfn_to_paddr(gfn_x(gfn))
+#define gaddr_to_gfn(ga)    _gfn(paddr_to_pfn(ga))
+#define mfn_to_maddr(mfn)   pfn_to_paddr(mfn_x(mfn))
+#define maddr_to_mfn(ma)    _mfn(paddr_to_pfn(ma))
+#define vmap_to_mfn(va)     maddr_to_mfn(virt_to_maddr((vaddr_t)va))
+#define vmap_to_page(va)    mfn_to_page(vmap_to_mfn(va))
+#define paddr_to_pdx(pa)    mfn_to_pdx(maddr_to_mfn(pa))
+#define gfn_to_gaddr(gfn)   pfn_to_paddr(gfn_x(gfn))
+#define gaddr_to_gfn(ga)    _gfn(paddr_to_pfn(ga))
+#define mfn_to_maddr(mfn)   pfn_to_paddr(mfn_x(mfn))
+#define maddr_to_mfn(ma)    _mfn(paddr_to_pfn(ma))
+#define vmap_to_mfn(va)     maddr_to_mfn(virt_to_maddr((vaddr_t)va))
+#define vmap_to_page(va)    mfn_to_page(vmap_to_mfn(va))
+
+#define virt_to_maddr(va) ((paddr_t)((vaddr_t)(va) & PADDR_MASK))
+#define maddr_to_virt(pa) ((void *)((paddr_t)(pa) | XEN_VIRT_START))
+
+/* Convert between Xen-heap virtual addresses and machine frame numbers. */
+#define __virt_to_mfn(va) (virt_to_maddr(va) >> PAGE_SHIFT)
+#define __mfn_to_virt(mfn) (maddr_to_virt((paddr_t)(mfn) << PAGE_SHIFT))
+
+/* Convert between Xen-heap virtual addresses and page-info structures. */
+static inline struct page_info *virt_to_page(const void *v)
+{
+    BUG();
+    return NULL;
+}
+
+/*
+ * We define non-underscored wrappers for above conversion functions.
+ * These are overriden in various source files while underscored version
+ * remain intact.
+ */
+#define virt_to_mfn(va)     __virt_to_mfn(va)
+#define mfn_to_virt(mfn)    __mfn_to_virt(mfn)
+
+struct page_info
+{
+    /* Each frame can be threaded onto a doubly-linked list. */
+    struct page_list_entry list;
+
+    /* Reference count and various PGC_xxx flags and fields. */
+    unsigned long count_info;
+
+    /* Context-dependent fields follow... */
+    union {
+        /* Page is in use: ((count_info & PGC_count_mask) != 0). */
+        struct {
+            /* Type reference count and various PGT_xxx flags and fields. */
+            unsigned long type_info;
+        } inuse;
+        /* Page is on a free list: ((count_info & PGC_count_mask) == 0). */
+        union {
+            struct {
+                /*
+                 * Index of the first *possibly* unscrubbed page in the buddy.
+                 * One more bit than maximum possible order to accommodate
+                 * INVALID_DIRTY_IDX.
+                 */
+#define INVALID_DIRTY_IDX ((1UL << (MAX_ORDER + 1)) - 1)
+                unsigned long first_dirty:MAX_ORDER + 1;
+
+                /* Do TLBs need flushing for safety before next page use? */
+                bool need_tlbflush:1;
+
+#define BUDDY_NOT_SCRUBBING    0
+#define BUDDY_SCRUBBING        1
+#define BUDDY_SCRUB_ABORT      2
+                unsigned long scrub_state:2;
+            };
+
+            unsigned long val;
+            } free;
+
+    } u;
+
+    union {
+        /* Page is in use, but not as a shadow. */
+        struct {
+            /* Owner of this page (zero if page is anonymous). */
+            struct domain *domain;
+        } inuse;
+
+        /* Page is on a free list. */
+        struct {
+            /* Order-size of the free chunk this page is the head of. */
+            unsigned int order;
+        } free;
+
+    } v;
+
+    union {
+        /*
+         * Timestamp from 'TLB clock', used to avoid extra safety flushes.
+         * Only valid for: a) free pages, and b) pages with zero type count
+         */
+        u32 tlbflush_timestamp;
+    };
+    u64 pad;
+};
+
+#define frame_table ((struct page_info *)FRAMETABLE_VIRT_START)
+
+/* PDX of the first page in the frame table. */
+extern unsigned long frametable_base_pdx;
+
+/* Convert between machine frame numbers and page-info structures. */
+#define mfn_to_page(mfn)                                            \
+    (frame_table + (mfn_to_pdx(mfn) - frametable_base_pdx))
+#define page_to_mfn(pg)                                             \
+    pdx_to_mfn((unsigned long)((pg) - frame_table) + frametable_base_pdx)
+
+static inline void *page_to_virt(const struct page_info *pg)
+{
+    return mfn_to_virt(mfn_x(page_to_mfn(pg)));
+}
+
+/*
+ * Common code requires get_page_type and put_page_type.
+ * We don't care about typecounts so we just do the minimum to make it
+ * happy.
+ */
+static inline int get_page_type(struct page_info *page, unsigned long type)
+{
+    return 1;
+}
+
+static inline void put_page_type(struct page_info *page)
+{
+    return;
+}
+
+/* TODO */
+static inline bool get_page_nr(struct page_info *page, const struct domain *domain,
+                        unsigned long nr)
+{
+    BUG();
+}
+static inline void put_page_nr(struct page_info *page, unsigned long nr)
+{
+    BUG();
+}
+
+static inline void put_page_and_type(struct page_info *page)
+{
+    put_page_type(page);
+    put_page(page);
+}
+
+/*
+ * RISCV does not have an M2P, but common code expects a handful of
+ * M2P-related defines and functions. Provide dummy versions of these.
+ */
+#define INVALID_M2P_ENTRY        (~0UL)
+#define SHARED_M2P_ENTRY         (~0UL - 1UL)
+#define SHARED_M2P(_e)           ((_e) == SHARED_M2P_ENTRY)
+
+/* Xen always owns P2M on PPC */
+#define set_gpfn_from_mfn(mfn, pfn) do { (void) (mfn), (void)(pfn); } while (0)
+#define mfn_to_gfn(d, mfn) ((void)(d), _gfn(mfn_x(mfn)))
+
+#define PDX_GROUP_SHIFT (16 + 5)
+
+static inline unsigned long domain_get_maximum_gpfn(struct domain *d)
+{
+    BUG();
+    return 0;
+}
+
+static inline long arch_memory_op(int op, XEN_GUEST_HANDLE_PARAM(void) arg)
+{
+    BUG();
+    return 0;
+}
+
+/*
+ * On RISCV, all the RAM is currently direct mapped in Xen.
+ * Hence return always true.
+ */
+static inline bool arch_mfns_in_directmap(unsigned long mfn, unsigned long nr)
+{
+    return true;
+}
+
+#define PG_shift(idx)   (BITS_PER_LONG - (idx))
+#define PG_mask(x, idx) (x ## UL << PG_shift(idx))
+
+#define PGT_none          PG_mask(0, 1)  /* no special uses of this page   */
+#define PGT_writable_page PG_mask(1, 1)  /* has writable mappings?         */
+#define PGT_type_mask     PG_mask(1, 1)  /* Bits 31 or 63.                 */
+
+ /* Count of uses of this frame as its current type. */
+#define PGT_count_width   PG_shift(2)
+#define PGT_count_mask    ((1UL<<PGT_count_width)-1)
+
+/*
+ * Page needs to be scrubbed. Since this bit can only be set on a page that is
+ * free (i.e. in PGC_state_free) we can reuse PGC_allocated bit.
+ */
+#define _PGC_need_scrub   _PGC_allocated
+#define PGC_need_scrub    PGC_allocated
+
+//  /* Cleared when the owning guest 'frees' this page. */
+#define _PGC_allocated    PG_shift(1)
+#define PGC_allocated     PG_mask(1, 1)
+  /* Page is Xen heap? */
+#define _PGC_xen_heap     PG_shift(2)
+#define PGC_xen_heap      PG_mask(1, 2)
+#ifdef CONFIG_STATIC_MEMORY
+/* Page is static memory */
+#define _PGC_static    PG_shift(3)
+#define PGC_static     PG_mask(1, 3)
+#else
+#define PGC_static     0
+#endif
+/* ... */
+/* Page is broken? */
+#define _PGC_broken       PG_shift(7)
+#define PGC_broken        PG_mask(1, 7)
+ /* Mutually-exclusive page states: { inuse, offlining, offlined, free }. */
+#define PGC_state         PG_mask(3, 9)
+#define PGC_state_inuse   PG_mask(0, 9)
+#define PGC_state_offlining PG_mask(1, 9)
+#define PGC_state_offlined PG_mask(2, 9)
+#define PGC_state_free    PG_mask(3, 9)
+// #define page_state_is(pg, st) (((pg)->count_info&PGC_state) == PGC_state_##st)
+
+/* Count of references to this frame. */
+#define PGC_count_width   PG_shift(9)
+#define PGC_count_mask    ((1UL<<PGC_count_width)-1)
+
+#define page_state_is(pg, st) (((pg)->count_info&PGC_state) == PGC_state_##st)
+
+#define _PGC_extra        PG_shift(10)
+#define PGC_extra         PG_mask(1, 10)
+
+#define is_xen_heap_page(page) ((page)->count_info & PGC_xen_heap)
+#define is_xen_heap_mfn(mfn) \
+    (mfn_valid(mfn) && is_xen_heap_page(mfn_to_page(mfn)))
+
+#define is_xen_fixed_mfn(mfn)                                   \
+    ((mfn_to_maddr(mfn) >= virt_to_maddr(&_start)) &&           \
+     (mfn_to_maddr(mfn) <= virt_to_maddr((vaddr_t)_end - 1)))
+
+#define page_get_owner(_p)    (_p)->v.inuse.domain
+#define page_set_owner(_p,_d) ((_p)->v.inuse.domain = (_d))
+
+/* TODO: implement */
+#define mfn_valid(mfn) ({ (void) (mfn); 0; })
+// #define max_page (0UL)
+
+#define mfn_to_gfn(d, mfn) ((void)(d), _gfn(mfn_x(mfn)))
+
+#define domain_set_alloc_bitsize(d) ((void)0)
+#define domain_clamp_alloc_bitsize(d, b) (b)
+
+#define PFN_ORDER(_pfn) ((_pfn)->v.free.order)
 
 extern unsigned char cpu0_boot_stack[];
 
@@ -20,4 +281,9 @@ unsigned long calc_phys_offset(void);
 
 void turn_on_mmu(unsigned long ra);
 
+static inline unsigned int arch_get_dma_bitsize(void)
+{
+    return 32; /* TODO */
+}
+
 #endif /* _ASM_RISCV_MM_H */
-- 
2.42.0

[PATCH v2 37/39] xen/rirscv: add minimal amount of stubs to build full Xen

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - define udelay stub
 - remove 'select HAS_PDX' from RISC-V Kconfig because of
   https://lore.kernel.org/xen-devel/20231006144405.1078260-1-andrew.cooper3@citrix.com/
---
 xen/arch/riscv/Makefile       |   1 +
 xen/arch/riscv/early_printk.c | 168 --------------
 xen/arch/riscv/mm.c           |  52 ++++-
 xen/arch/riscv/setup.c        |   9 +-
 xen/arch/riscv/stubs.c        | 426 ++++++++++++++++++++++++++++++++++
 xen/arch/riscv/traps.c        |  25 ++
 6 files changed, 511 insertions(+), 170 deletions(-)
 create mode 100644 xen/arch/riscv/stubs.c

diff --git a/xen/arch/riscv/Makefile b/xen/arch/riscv/Makefile
index b08048948f..50c09469a0 100644
--- a/xen/arch/riscv/Makefile
+++ b/xen/arch/riscv/Makefile
@@ -5,6 +5,7 @@ obj-y += mm.o
 obj-$(CONFIG_RISCV_64) += riscv64/
 obj-y += sbi.o
 obj-y += setup.o
+obj-y += stubs.o
 obj-y += traps.o
 
 $(TARGET): $(TARGET)-syms
diff --git a/xen/arch/riscv/early_printk.c b/xen/arch/riscv/early_printk.c
index 60742a042d..610c814f54 100644
--- a/xen/arch/riscv/early_printk.c
+++ b/xen/arch/riscv/early_printk.c
@@ -40,171 +40,3 @@ void early_printk(const char *str)
         str++;
     }
 }
-
-/*
- * The following #if 1 ... #endif should be removed after printk
- * and related stuff are ready.
- */
-#if 1
-
-#include <xen/stdarg.h>
-#include <xen/string.h>
-
-/**
- * strlen - Find the length of a string
- * @s: The string to be sized
- */
-size_t (strlen)(const char * s)
-{
-    const char *sc;
-
-    for (sc = s; *sc != '\0'; ++sc)
-        /* nothing */;
-    return sc - s;
-}
-
-/**
- * memcpy - Copy one area of memory to another
- * @dest: Where to copy to
- * @src: Where to copy from
- * @count: The size of the area.
- *
- * You should not use this function to access IO space, use memcpy_toio()
- * or memcpy_fromio() instead.
- */
-void *(memcpy)(void *dest, const void *src, size_t count)
-{
-    char *tmp = (char *) dest, *s = (char *) src;
-
-    while (count--)
-        *tmp++ = *s++;
-
-    return dest;
-}
-
-int vsnprintf(char* str, size_t size, const char* format, va_list args)
-{
-    size_t i = 0; /* Current position in the output string */
-    size_t written = 0; /* Total number of characters written */
-    char* dest = str;
-
-    while ( format[i] != '\0' && written < size - 1 )
-    {
-        if ( format[i] == '%' )
-        {
-            i++;
-
-            if ( format[i] == '\0' )
-                break;
-
-            if ( format[i] == '%' )
-            {
-                if ( written < size - 1 )
-                {
-                    dest[written] = '%';
-                    written++;
-                }
-                i++;
-                continue;
-            }
-
-            /*
-             * Handle format specifiers.
-             * For simplicity, only %s and %d are implemented here.
-             */
-
-            if ( format[i] == 's' )
-            {
-                char* arg = va_arg(args, char*);
-                size_t arglen = strlen(arg);
-
-                size_t remaining = size - written - 1;
-
-                if ( arglen > remaining )
-                    arglen = remaining;
-
-                memcpy(dest + written, arg, arglen);
-
-                written += arglen;
-                i++;
-            }
-            else if ( format[i] == 'd' )
-            {
-                int arg = va_arg(args, int);
-
-                /* Convert the integer to string representation */
-                char numstr[32]; /* Assumes a maximum of 32 digits */
-                int numlen = 0;
-                int num = arg;
-                size_t remaining;
-
-                if ( arg < 0 )
-                {
-                    if ( written < size - 1 )
-                    {
-                        dest[written] = '-';
-                        written++;
-                    }
-
-                    num = -arg;
-                }
-
-                do
-                {
-                    numstr[numlen] = '0' + num % 10;
-                    num = num / 10;
-                    numlen++;
-                } while ( num > 0 );
-
-                /* Reverse the string */
-                for (int j = 0; j < numlen / 2; j++)
-                {
-                    char tmp = numstr[j];
-                    numstr[j] = numstr[numlen - 1 - j];
-                    numstr[numlen - 1 - j] = tmp;
-                }
-
-                remaining = size - written - 1;
-
-                if ( numlen > remaining )
-                    numlen = remaining;
-
-                memcpy(dest + written, numstr, numlen);
-
-                written += numlen;
-                i++;
-            }
-        }
-        else
-        {
-            if ( written < size - 1 )
-            {
-                dest[written] = format[i];
-                written++;
-            }
-            i++;
-        }
-    }
-
-    if ( size > 0 )
-        dest[written] = '\0';
-
-    return written;
-}
-
-void printk(const char *format, ...)
-{
-    static char buf[1024];
-
-    va_list args;
-    va_start(args, format);
-
-    (void)vsnprintf(buf, sizeof(buf), format, args);
-
-    early_printk(buf);
-
-    va_end(args);
-}
-
-#endif
-
diff --git a/xen/arch/riscv/mm.c b/xen/arch/riscv/mm.c
index 053f043a3d..f0eddab56c 100644
--- a/xen/arch/riscv/mm.c
+++ b/xen/arch/riscv/mm.c
@@ -1,19 +1,23 @@
 /* SPDX-License-Identifier: GPL-2.0-only */
 
+#include <xen/bug.h>
 #include <xen/cache.h>
 #include <xen/compiler.h>
 #include <xen/init.h>
 #include <xen/kernel.h>
 #include <xen/macros.h>
+#include <xen/mm.h>
 #include <xen/pfn.h>
 
 #include <asm/early_printk.h>
 #include <asm/csr.h>
 #include <asm/current.h>
-#include <asm/mm.h>
 #include <asm/page.h>
 #include <asm/processor.h>
 
+unsigned long frametable_base_pdx __read_mostly;
+unsigned long frametable_virt_end __read_mostly;
+
 struct mmu_desc {
     unsigned int num_levels;
     unsigned int pgtbl_count;
@@ -294,3 +298,49 @@ unsigned long __init calc_phys_offset(void)
     phys_offset = load_start - XEN_VIRT_START;
     return phys_offset;
 }
+
+void put_page(struct page_info *page)
+{
+    assert_failed(__func__);
+}
+
+unsigned long get_upper_mfn_bound(void)
+{
+    /* No memory hotplug yet, so current memory limit is the final one. */
+    return max_page - 1;
+}
+
+void arch_dump_shared_mem_info(void)
+{
+    WARN();
+}
+
+int populate_pt_range(unsigned long virt, unsigned long nr_mfns)
+{
+    assert_failed(__func__);
+    return -1;
+}
+
+int xenmem_add_to_physmap_one(struct domain *d, unsigned int space,
+                              union add_to_physmap_extra extra,
+                              unsigned long idx, gfn_t gfn)
+{
+    WARN();
+
+    return 0;
+}
+
+int destroy_xen_mappings(unsigned long s, unsigned long e)
+{
+    assert_failed(__func__);
+    return -1;
+}
+
+int map_pages_to_xen(unsigned long virt,
+                     mfn_t mfn,
+                     unsigned long nr_mfns,
+                     unsigned int flags)
+{
+    assert_failed(__func__);
+    return -1;
+}
diff --git a/xen/arch/riscv/setup.c b/xen/arch/riscv/setup.c
index 6593f601c1..8944e0ecfe 100644
--- a/xen/arch/riscv/setup.c
+++ b/xen/arch/riscv/setup.c
@@ -2,9 +2,16 @@
 
 #include <xen/compile.h>
 #include <xen/init.h>
+#include <xen/mm.h>
+
+#include <public/version.h>
 
 #include <asm/early_printk.h>
-#include <asm/mm.h>
+
+void arch_get_xen_caps(xen_capabilities_info_t *info)
+{
+    assert_failed("need to be implemented");
+}
 
 /* Xen stack for bringing up the first CPU. */
 unsigned char __initdata cpu0_boot_stack[STACK_SIZE]
diff --git a/xen/arch/riscv/stubs.c b/xen/arch/riscv/stubs.c
new file mode 100644
index 0000000000..2031ef6bde
--- /dev/null
+++ b/xen/arch/riscv/stubs.c
@@ -0,0 +1,426 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#include <xen/cpumask.h>
+#include <xen/domain.h>
+#include <xen/irq.h>
+#include <xen/nodemask.h>
+#include <xen/time.h>
+#include <public/domctl.h>
+#include <public/vm_event.h>
+
+#include <asm/current.h>
+
+/* smpboot.c */
+
+cpumask_t cpu_online_map;
+cpumask_t cpu_present_map;
+cpumask_t cpu_possible_map;
+
+/* ID of the PCPU we're running on */
+DEFINE_PER_CPU(unsigned int, cpu_id);
+/* XXX these seem awfully x86ish... */
+/* representing HT siblings of each logical CPU */
+DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_mask);
+/* representing HT and core siblings of each logical CPU */
+DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_core_mask);
+
+nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
+
+/* time.c */
+
+unsigned long __read_mostly cpu_khz;  /* CPU clock frequency in kHz. */
+
+s_time_t get_s_time(void)
+{
+    BUG();
+}
+
+int reprogram_timer(s_time_t timeout)
+{
+    BUG();
+}
+
+void send_timer_event(struct vcpu *v)
+{
+    BUG();
+}
+
+void domain_set_time_offset(struct domain *d, int64_t time_offset_seconds)
+{
+    BUG();
+}
+
+/* shutdown.c */
+
+void machine_restart(unsigned int delay_millisecs)
+{
+    BUG();
+}
+
+void machine_halt(void)
+{
+    BUG();
+}
+
+/* vm_event.c */
+
+void vm_event_fill_regs(vm_event_request_t *req)
+{
+    BUG();
+}
+
+void vm_event_set_registers(struct vcpu *v, vm_event_response_t *rsp)
+{
+    BUG();
+}
+
+void vm_event_monitor_next_interrupt(struct vcpu *v)
+{
+    /* Not supported on RISCV. */
+}
+
+void vm_event_reset_vmtrace(struct vcpu *v)
+{
+    /* Not supported on RISCV. */
+}
+
+/* domctl.c */
+
+long arch_do_domctl(struct xen_domctl *domctl, struct domain *d,
+                    XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl)
+{
+    BUG();
+}
+
+void arch_get_domain_info(const struct domain *d,
+                          struct xen_domctl_getdomaininfo *info)
+{
+    BUG();
+}
+
+void arch_get_info_guest(struct vcpu *v, vcpu_guest_context_u c)
+{
+    BUG();
+}
+
+/* monitor.c */
+
+int arch_monitor_domctl_event(struct domain *d,
+                              struct xen_domctl_monitor_op *mop)
+{
+    BUG();
+}
+
+/* smp.c */
+
+void arch_flush_tlb_mask(const cpumask_t *mask)
+{
+    BUG();
+}
+
+void smp_send_event_check_mask(const cpumask_t *mask)
+{
+    BUG();
+}
+
+void smp_send_call_function_mask(const cpumask_t *mask)
+{
+    BUG();
+}
+
+/* irq.c */
+
+struct pirq *alloc_pirq_struct(struct domain *d)
+{
+    BUG();
+}
+
+int pirq_guest_bind(struct vcpu *v, struct pirq *pirq, int will_share)
+{
+    BUG();
+}
+
+void pirq_guest_unbind(struct domain *d, struct pirq *pirq)
+{
+    BUG();
+}
+
+void pirq_set_affinity(struct domain *d, int pirq, const cpumask_t *mask)
+{
+    BUG();
+}
+
+static void ack_none(struct irq_desc *irq)
+{
+    BUG();
+}
+
+static void end_none(struct irq_desc *irq)
+{
+    BUG();
+}
+
+hw_irq_controller no_irq_type = {
+    .typename = "none",
+    .startup = irq_startup_none,
+    .shutdown = irq_shutdown_none,
+    .enable = irq_enable_none,
+    .disable = irq_disable_none,
+    .ack = ack_none,
+    .end = end_none
+};
+
+int arch_init_one_irq_desc(struct irq_desc *desc)
+{
+    BUG();
+}
+
+void smp_send_state_dump(unsigned int cpu)
+{
+    BUG();
+}
+
+/* domain.c */
+
+DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
+unsigned long __per_cpu_offset[NR_CPUS];
+
+void context_switch(struct vcpu *prev, struct vcpu *next)
+{
+    BUG();
+}
+
+void continue_running(struct vcpu *same)
+{
+    BUG();
+}
+
+void sync_local_execstate(void)
+{
+    BUG();
+}
+
+void sync_vcpu_execstate(struct vcpu *v)
+{
+    BUG();
+}
+
+void startup_cpu_idle_loop(void)
+{
+    BUG();
+}
+
+void free_domain_struct(struct domain *d)
+{
+    BUG();
+}
+
+void dump_pageframe_info(struct domain *d)
+{
+    BUG();
+}
+
+void free_vcpu_struct(struct vcpu *v)
+{
+    BUG();
+}
+
+int arch_vcpu_create(struct vcpu *v)
+{
+    BUG();
+}
+
+void arch_vcpu_destroy(struct vcpu *v)
+{
+    BUG();
+}
+
+void vcpu_switch_to_aarch64_mode(struct vcpu *v)
+{
+    BUG();
+}
+
+int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
+{
+    BUG();
+}
+
+int arch_domain_create(struct domain *d,
+                       struct xen_domctl_createdomain *config,
+                       unsigned int flags)
+{
+    BUG();
+}
+
+int arch_domain_teardown(struct domain *d)
+{
+    BUG();
+}
+
+void arch_domain_destroy(struct domain *d)
+{
+    BUG();
+}
+
+void arch_domain_shutdown(struct domain *d)
+{
+    BUG();
+}
+
+void arch_domain_pause(struct domain *d)
+{
+    BUG();
+}
+
+void arch_domain_unpause(struct domain *d)
+{
+    BUG();
+}
+
+int arch_domain_soft_reset(struct domain *d)
+{
+    BUG();
+}
+
+void arch_domain_creation_finished(struct domain *d)
+{
+    BUG();
+}
+
+int arch_set_info_guest(struct vcpu *v, vcpu_guest_context_u c)
+{
+    BUG();
+}
+
+int arch_initialise_vcpu(struct vcpu *v, XEN_GUEST_HANDLE_PARAM(void) arg)
+{
+    BUG();
+}
+
+int arch_vcpu_reset(struct vcpu *v)
+{
+    BUG();
+}
+
+int domain_relinquish_resources(struct domain *d)
+{
+    BUG();
+}
+
+void arch_dump_domain_info(struct domain *d)
+{
+    BUG();
+}
+
+void arch_dump_vcpu_info(struct vcpu *v)
+{
+    BUG();
+}
+
+void vcpu_mark_events_pending(struct vcpu *v)
+{
+    BUG();
+}
+
+void vcpu_update_evtchn_irq(struct vcpu *v)
+{
+    BUG();
+}
+
+void vcpu_block_unless_event_pending(struct vcpu *v)
+{
+    BUG();
+}
+
+void vcpu_kick(struct vcpu *v)
+{
+    BUG();
+}
+
+struct domain *alloc_domain_struct(void)
+{
+    BUG();
+}
+
+struct vcpu *alloc_vcpu_struct(const struct domain *d)
+{
+    BUG();
+}
+
+unsigned long
+hypercall_create_continuation(unsigned int op, const char *format, ...)
+{
+    BUG();
+}
+
+int __init parse_arch_dom0_param(const char *s, const char *e)
+{
+    BUG();
+}
+
+/* guestcopy.c */
+
+unsigned long raw_copy_to_guest(void *to, const void *from, unsigned int len)
+{
+    BUG();
+}
+
+unsigned long raw_copy_from_guest(void *to, const void __user *from,
+                                  unsigned int len)
+{
+    BUG();
+}
+
+/* sysctl.c */
+
+long arch_do_sysctl(struct xen_sysctl *sysctl,
+                    XEN_GUEST_HANDLE_PARAM(xen_sysctl_t) u_sysctl)
+{
+    BUG();
+}
+
+void arch_do_physinfo(struct xen_sysctl_physinfo *pi)
+{
+    BUG();
+}
+
+/* p2m.c */
+
+int arch_set_paging_mempool_size(struct domain *d, uint64_t size)
+{
+    BUG();
+}
+
+int unmap_mmio_regions(struct domain *d,
+                       gfn_t start_gfn,
+                       unsigned long nr,
+                       mfn_t mfn)
+{
+    BUG();
+}
+
+int map_mmio_regions(struct domain *d,
+                     gfn_t start_gfn,
+                     unsigned long nr,
+                     mfn_t mfn)
+{
+    BUG();
+}
+
+int set_foreign_p2m_entry(struct domain *d, const struct domain *fd,
+                          unsigned long gfn, mfn_t mfn)
+{
+    BUG();
+}
+
+/* Return the size of the pool, in bytes. */
+int arch_get_paging_mempool_size(struct domain *d, uint64_t *size)
+{
+    BUG();
+}
+
+/* delay.c */
+
+void udelay(unsigned long usecs)
+{
+    BUG_ON("unimplemented");
+}
diff --git a/xen/arch/riscv/traps.c b/xen/arch/riscv/traps.c
index ccd3593f5a..ca56df75d8 100644
--- a/xen/arch/riscv/traps.c
+++ b/xen/arch/riscv/traps.c
@@ -4,6 +4,10 @@
  *
  * RISC-V Trap handlers
  */
+
+#include <xen/lib.h>
+#include <xen/sched.h>
+
 #include <asm/processor.h>
 #include <asm/traps.h>
 
@@ -11,3 +15,24 @@ void do_trap(struct cpu_user_regs *cpu_regs)
 {
     die();
 }
+
+void vcpu_show_execution_state(struct vcpu *v)
+{
+    assert_failed("need to be implented");
+}
+
+void show_execution_state(const struct cpu_user_regs *regs)
+{
+    printk("implement show_execution_state(regs)\n");
+}
+
+void arch_hypercall_tasklet_result(struct vcpu *v, long res)
+{
+    assert_failed("need to be implented");
+}
+
+enum mc_disposition arch_do_multicall_call(struct mc_state *state)
+{
+    assert_failed("need to be implented");
+    return mc_continue;
+}
-- 
2.42.0

[PATCH v2 38/39] xen/riscv: enable full Xen build

[Oleksii Kurochko]

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - Nothing changed. Only rebase.
---
 xen/arch/riscv/Makefile                 | 16 +++++++++++++++-
 xen/arch/riscv/arch.mk                  |  4 ----
 xen/arch/riscv/configs/tiny64_defconfig |  1 -
 3 files changed, 15 insertions(+), 6 deletions(-)

diff --git a/xen/arch/riscv/Makefile b/xen/arch/riscv/Makefile
index 50c09469a0..9c02cb0ccf 100644
--- a/xen/arch/riscv/Makefile
+++ b/xen/arch/riscv/Makefile
@@ -12,10 +12,24 @@ $(TARGET): $(TARGET)-syms
 	$(OBJCOPY) -O binary -S $< $@
 
 $(TARGET)-syms: $(objtree)/prelink.o $(obj)/xen.lds
-	$(LD) $(XEN_LDFLAGS) -T $(obj)/xen.lds -N $< $(build_id_linker) -o $@
+	$(LD) $(XEN_LDFLAGS) -T $(obj)/xen.lds -N $< \
+	    $(objtree)/common/symbols-dummy.o -o $(dot-target).0
+	$(NM) -pa --format=sysv $(dot-target).0 \
+		| $(objtree)/tools/symbols $(all_symbols) --sysv --sort \
+		> $(dot-target).0.S
+	$(MAKE) $(build)=$(@D) $(dot-target).0.o
+	$(LD) $(XEN_LDFLAGS) -T $(obj)/xen.lds -N $< \
+	    $(dot-target).0.o -o $(dot-target).1
+	$(NM) -pa --format=sysv $(dot-target).1 \
+		| $(objtree)/tools/symbols $(all_symbols) --sysv --sort \
+		> $(dot-target).1.S
+	$(MAKE) $(build)=$(@D) $(dot-target).1.o
+	$(LD) $(XEN_LDFLAGS) -T $(obj)/xen.lds -N $< $(build_id_linker) \
+	    $(dot-target).1.o -o $@
 	$(NM) -pa --format=sysv $@ \
 		| $(objtree)/tools/symbols --all-symbols --xensyms --sysv --sort \
 		> $@.map
+	rm -f $(@D)/.$(@F).[0-9]*
 
 $(obj)/xen.lds: $(src)/xen.lds.S FORCE
 	$(call if_changed_dep,cpp_lds_S)
diff --git a/xen/arch/riscv/arch.mk b/xen/arch/riscv/arch.mk
index 8403f96b6f..12b1673fae 100644
--- a/xen/arch/riscv/arch.mk
+++ b/xen/arch/riscv/arch.mk
@@ -13,7 +13,3 @@ riscv-march-$(CONFIG_RISCV_ISA_C)       := $(riscv-march-y)c
 # -mcmodel=medlow would force Xen into the lower half.
 
 CFLAGS += -march=$(riscv-march-y) -mstrict-align -mcmodel=medany
-
-# TODO: Drop override when more of the build is working
-override ALL_OBJS-y = arch/$(SRCARCH)/built_in.o
-override ALL_LIBS-y =
diff --git a/xen/arch/riscv/configs/tiny64_defconfig b/xen/arch/riscv/configs/tiny64_defconfig
index ce37c5ae00..4b488180ae 100644
--- a/xen/arch/riscv/configs/tiny64_defconfig
+++ b/xen/arch/riscv/configs/tiny64_defconfig
@@ -24,7 +24,6 @@
 # CONFIG_COVERAGE is not set
 # CONFIG_UBSAN is not set
 # CONFIG_NEEDS_LIBELF is not set
-
 CONFIG_RISCV_64=y
 CONFIG_DEBUG=y
 CONFIG_DEBUG_INFO=y
-- 
2.42.0

[PATCH v2 39/39] xen: fix compilation issue of serial.c

[Oleksii Kurochko]

The following issue occurs on RISC-V platforms:
drivers/char/serial.c: In function 'serial_tx_interrupt':
drivers/char/serial.c:88:9: error: implicit declaration of function 'cpu_relax' [-Werror=implicit-function-declaration]
   88 |         cpu_relax();

cpu_relax() is defined in <asm/processor.h> so it was added
an inclusion of the header to serial.c.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
Changes in V2:
 - New commit introduced in V2
---
 xen/drivers/char/serial.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/xen/drivers/char/serial.c b/xen/drivers/char/serial.c
index 00efe69574..35df58fe33 100644
--- a/xen/drivers/char/serial.c
+++ b/xen/drivers/char/serial.c
@@ -13,6 +13,8 @@
 #include <xen/serial.h>
 #include <xen/cache.h>
 
+#include <asm/processor.h>
+
 /* Never drop characters, even if the async transmit buffer fills. */
 /* #define SERIAL_NEVER_DROP_CHARS 1 */
 
-- 
2.42.0

Re: [PATCH v2 01/39] xen/riscv: disable unnecessary configs

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> The patch also fixes the build script as conf util expects
> to have each config on separate line.

The approach doesn't really scale (it's already odd that you add the
(apparently) same set four times. There's also zero justification for
this kind of an adjustment (as per discussion elsewhere I don't think
we should go this route, and hence arguments towards convincing me [and
perhaps others] would be needed here).

> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -522,6 +522,38 @@ archlinux-current-gcc-riscv64:
>      CONTAINER: archlinux:current-riscv64
>      KBUILD_DEFCONFIG: tiny64_defconfig
>      HYPERVISOR_ONLY: y
> +    EXTRA_XEN_CONFIG:
> +      CONFIG_COVERAGE=n
> +      CONFIG_GRANT_TABLE=n
> +      CONFIG_SCHED_CREDIT=n
> +      CONFIG_SCHED_CREDIT2=n
> +      CONFIG_SCHED_RTDS=n
> +      CONFIG_SCHED_NULL=n
> +      CONFIG_SCHED_ARINC653=n
> +      CONFIG_TRACEBUFFER=n
> +      CONFIG_HYPFS=n
> +      CONFIG_GRANT_TABLE=n
> +      CONFIG_SPECULATIVE_HARDEN_ARRAY=n
> +      CONFIG_ARGO=n
> +      CONFIG_HYPFS_CONFIG=n
> +      CONFIG_CORE_PARKING=n
> +      CONFIG_DEBUG_TRACE=n
> +      CONFIG_IOREQ_SERVER=n
> +      CONFIG_CRASH_DEBUG=n
> +      CONFIG_KEXEC=n
> +      CONFIG_LIVEPATCH=n
> +      CONFIG_MEM_ACCESS=n
> +      CONFIG_NUMA=n
> +      CONFIG_PERF_COUNTERS=n
> +      CONFIG_HAS_PMAP=n
> +      CONFIG_TRACEBUFFER=n
> +      CONFIG_XENOPROF=n
> +      CONFIG_COMPAT=n
> +      CONFIG_COVERAGE=n
> +      CONFIG_UBSAN=n
> +      CONFIG_NEEDS_LIBELF=n
> +      CONFIG_XSM=n
> +
>  
>  archlinux-current-gcc-riscv64-debug:
>    extends: .gcc-riscv64-cross-build-debug

I think I've said so elsewhere before: Please avoid introducing double
blank lines, unless entirely unavoidable (for reasons I cannot think of).

Jan

Re: [PATCH v2 02/39] xen/riscv: use some asm-generic headers

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Some headers are the same as asm-generic verions of them
> so use them instead of arch-specific headers.
> 
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>

Acked-by: Jan Beulich <jbeulich@suse.com>
assuming ...

> --- /dev/null
> +++ b/xen/arch/riscv/include/asm/Makefile
> @@ -0,0 +1,14 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +generic-y += altp2m.h
> +generic-y += device.h
> +generic-y += div64.h
> +generic-y += hardirq.h
> +generic-y += hypercall.h
> +generic-y += iocap.h
> +generic-y += monitor.h
> +generic-y += numa.h
> +generic-y += paging.h
> +generic-y += percpu.h
> +generic-y += random.h
> +generic-y += softirq.h
> +generic-y += vm_event.h

... these headers are indeed all going to appear.

Jan

Re: [PATCH v2 03/39] xen/riscv:introduce asm/byteorder.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>

Acked-by: Jan Beulich <jbeulich@suse.com>

Re: [PATCH v2 05/39] xen/riscv: introduce spinlock.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>

Acked-by: Jan Beulich <jbeulich@suse.com>

Re: [PATCH v2 06/39] xen/riscv: introduce fence.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> --- /dev/null
> +++ b/xen/arch/riscv/include/asm/fence.h
> @@ -0,0 +1,12 @@
> +#ifndef _ASM_RISCV_FENCE_H
> +#define _ASM_RISCV_FENCE_H
> +
> +#ifdef CONFIG_SMP
> +#define RISCV_ACQUIRE_BARRIER		"\tfence r , rw\n"
> +#define RISCV_RELEASE_BARRIER		"\tfence rw,  w\n"
> +#else
> +#define RISCV_ACQUIRE_BARRIER
> +#define RISCV_RELEASE_BARRIER
> +#endif
> +
> +#endif	/* _ASM_RISCV_FENCE_H */

Imo such a header would be better to introduce once a use for the
constructs appears. Otherwise at the very least it wants explaining
in the description what this is going to be needed for. I can't
find items of these names in other architectures so far, so this
must be something RISC-V-specific.

Jan

Re: [PATCH v2 07/39] xen/riscv: introduce arch-riscv/hvm/save.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> --- /dev/null
> +++ b/xen/include/public/arch-riscv/hvm/save.h
> @@ -0,0 +1,20 @@
> +/* SPDX-License-Identifier: MIT */
> +/*
> + * Structure definitions for HVM state that is held by Xen and must
> + * be saved along with the domain's memory and device-model state.
> + */
> +
> +#ifndef __XEN_PUBLIC_HVM_SAVE_RISCV_H__
> +#define __XEN_PUBLIC_HVM_SAVE_RISCV_H__
> +
> +#endif
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * tab-width: 4
> + * indent-tabs-mode: nil
> + * End:
> + */

Seeing that Arm's is as empty, I wonder why we have it. Julien, Stefano?

> --- a/xen/include/public/hvm/save.h
> +++ b/xen/include/public/hvm/save.h
> @@ -91,6 +91,8 @@ DECLARE_HVM_SAVE_TYPE(END, 0, struct hvm_save_end);
>  #include "../arch-arm/hvm/save.h"
>  #elif defined(__powerpc64__)
>  #include "../arch-ppc.h"
> +#elif defined(__riscv)
> +#include "../arch-riscv/hvm/save.h"
>  #else
>  #error "unsupported architecture"
>  #endif

The PPC part here looks bogus altogether. Shawn?

Jan

Re: [PATCH v2 01/39] xen/riscv: disable unnecessary configs

[Oleksii]

On Tue, 2023-12-05 at 16:38 +0100, Jan Beulich wrote:
> > On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > > > The patch also fixes the build script as conf util expects
> > > > to have each config on separate line.
> > 
> > The approach doesn't really scale (it's already odd that you add
> > the
> > (apparently) same set four times. There's also zero justification
> > for
> > this kind of an adjustment (as per discussion elsewhere I don't
> > think
> > we should go this route, and hence arguments towards convincing me
> > [and
> > perhaps others] would be needed here).
I agree that this may not be the best approach, but it seems like we
don't have too many options to turn off a config for randconfig.

To be honest, in my opinion (IMO), randconfig should either be removed
or allowed to fail until most of the functionality is ready. Otherwise,
there should be a need to introduce HAS_* or depend on
'SUPPORTED_ARCHS' for each config, or introduce a lot of stubs.

Could you please suggest a better option?

> > 
> > > > --- a/automation/gitlab-ci/build.yaml
> > > > +++ b/automation/gitlab-ci/build.yaml
> > > > @@ -522,6 +522,38 @@ archlinux-current-gcc-riscv64:
> > > >      CONTAINER: archlinux:current-riscv64
> > > >      KBUILD_DEFCONFIG: tiny64_defconfig
> > > >      HYPERVISOR_ONLY: y
> > > > +    EXTRA_XEN_CONFIG:
> > > > +      CONFIG_COVERAGE=n
> > > > +      CONFIG_GRANT_TABLE=n
> > > > +      CONFIG_SCHED_CREDIT=n
> > > > +      CONFIG_SCHED_CREDIT2=n
> > > > +      CONFIG_SCHED_RTDS=n
> > > > +      CONFIG_SCHED_NULL=n
> > > > +      CONFIG_SCHED_ARINC653=n
> > > > +      CONFIG_TRACEBUFFER=n
> > > > +      CONFIG_HYPFS=n
> > > > +      CONFIG_GRANT_TABLE=n
> > > > +      CONFIG_SPECULATIVE_HARDEN_ARRAY=n
> > > > +      CONFIG_ARGO=n
> > > > +      CONFIG_HYPFS_CONFIG=n
> > > > +      CONFIG_CORE_PARKING=n
> > > > +      CONFIG_DEBUG_TRACE=n
> > > > +      CONFIG_IOREQ_SERVER=n
> > > > +      CONFIG_CRASH_DEBUG=n
> > > > +      CONFIG_KEXEC=n
> > > > +      CONFIG_LIVEPATCH=n
> > > > +      CONFIG_MEM_ACCESS=n
> > > > +      CONFIG_NUMA=n
> > > > +      CONFIG_PERF_COUNTERS=n
> > > > +      CONFIG_HAS_PMAP=n
> > > > +      CONFIG_TRACEBUFFER=n
> > > > +      CONFIG_XENOPROF=n
> > > > +      CONFIG_COMPAT=n
> > > > +      CONFIG_COVERAGE=n
> > > > +      CONFIG_UBSAN=n
> > > > +      CONFIG_NEEDS_LIBELF=n
> > > > +      CONFIG_XSM=n
> > > > +
> > > >  
> > > >  archlinux-current-gcc-riscv64-debug:
> > > >    extends: .gcc-riscv64-cross-build-debug
> > 
> > I think I've said so elsewhere before: Please avoid introducing
> > double
> > blank lines, unless entirely unavoidable (for reasons I cannot
> > think
> > of).
Sorry for that; I am not doing that on purpose. It's just a big patch
series, and I missed that double blank. I will try to be more
attentive.

Do you think it makes sense to add a script to perform basic code style
checks, similar to what Linux has?

> > 
~ Oleksii

Re: [PATCH v2 02/39] xen/riscv: use some asm-generic headers

[Oleksii]

On Tue, 2023-12-05 at 16:40 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > Some headers are the same as asm-generic verions of them
> > so use them instead of arch-specific headers.
> > 
> > Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> 
> Acked-by: Jan Beulich <jbeulich@suse.com>
> assuming ...
> 
> > --- /dev/null
> > +++ b/xen/arch/riscv/include/asm/Makefile
> > @@ -0,0 +1,14 @@
> > +# SPDX-License-Identifier: GPL-2.0-only
> > +generic-y += altp2m.h
> > +generic-y += device.h
> > +generic-y += div64.h
> > +generic-y += hardirq.h
> > +generic-y += hypercall.h
> > +generic-y += iocap.h
> > +generic-y += monitor.h
> > +generic-y += numa.h
> > +generic-y += paging.h
> > +generic-y += percpu.h
> > +generic-y += random.h
> > +generic-y += softirq.h
> > +generic-y += vm_event.h
> 
> ... these headers are indeed all going to appear.
Thanks.

Only 4 (div64.h, monitor.h, numa.h, softirq.h) of them aren't appearing
in staging. It only needs some ACKs from Arm and PPC maintainers.

But if something changes, I'll update riscv/include/asm/Makefile

~ Oleksii

Re: [PATCH v2 06/39] xen/riscv: introduce fence.h

[Oleksii]

On Tue, 2023-12-05 at 16:56 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > --- /dev/null
> > +++ b/xen/arch/riscv/include/asm/fence.h
> > @@ -0,0 +1,12 @@
> > +#ifndef _ASM_RISCV_FENCE_H
> > +#define _ASM_RISCV_FENCE_H
> > +
> > +#ifdef CONFIG_SMP
> > +#define RISCV_ACQUIRE_BARRIER		"\tfence r , rw\n"
> > +#define RISCV_RELEASE_BARRIER		"\tfence rw,  w\n"
> > +#else
> > +#define RISCV_ACQUIRE_BARRIER
> > +#define RISCV_RELEASE_BARRIER
> > +#endif
> > +
> > +#endif	/* _ASM_RISCV_FENCE_H */
> 
> Imo such a header would be better to introduce once a use for the
> constructs appears. Otherwise at the very least it wants explaining
> in the description what this is going to be needed for. I can't
> find items of these names in other architectures so far, so this
> must be something RISC-V-specific.
It is going to be used only in RISC-V. The things that use these
definitions are introduced in the patches of this patch series:
* [PATCH v2 18/39] xen/riscv: introduce cmpxchg.h
* [PATCH v2 17/39] xen/riscv: introduce asm/atomic.h

~ Oleksii

Re: [PATCH v2 06/39] xen/riscv: introduce fence.h

[Jan Beulich]

On 07.12.2023 10:42, Oleksii wrote:
> On Tue, 2023-12-05 at 16:56 +0100, Jan Beulich wrote:
>> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>>> --- /dev/null
>>> +++ b/xen/arch/riscv/include/asm/fence.h
>>> @@ -0,0 +1,12 @@
>>> +#ifndef _ASM_RISCV_FENCE_H
>>> +#define _ASM_RISCV_FENCE_H
>>> +
>>> +#ifdef CONFIG_SMP
>>> +#define RISCV_ACQUIRE_BARRIER		"\tfence r , rw\n"
>>> +#define RISCV_RELEASE_BARRIER		"\tfence rw,  w\n"
>>> +#else
>>> +#define RISCV_ACQUIRE_BARRIER
>>> +#define RISCV_RELEASE_BARRIER
>>> +#endif
>>> +
>>> +#endif	/* _ASM_RISCV_FENCE_H */
>>
>> Imo such a header would be better to introduce once a use for the
>> constructs appears. Otherwise at the very least it wants explaining
>> in the description what this is going to be needed for. I can't
>> find items of these names in other architectures so far, so this
>> must be something RISC-V-specific.
> It is going to be used only in RISC-V. The things that use these
> definitions are introduced in the patches of this patch series:
> * [PATCH v2 18/39] xen/riscv: introduce cmpxchg.h
> * [PATCH v2 17/39] xen/riscv: introduce asm/atomic.h

Then perhaps fold this patch into patch 17?

Jan

Re: [PATCH v2 01/39] xen/riscv: disable unnecessary configs

[Jan Beulich]

On 07.12.2023 10:22, Oleksii wrote:
> On Tue, 2023-12-05 at 16:38 +0100, Jan Beulich wrote:
>>> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>>>>> The patch also fixes the build script as conf util expects
>>>>> to have each config on separate line.
>>>
>>> The approach doesn't really scale (it's already odd that you add
>>> the
>>> (apparently) same set four times. There's also zero justification
>>> for
>>> this kind of an adjustment (as per discussion elsewhere I don't
>>> think
>>> we should go this route, and hence arguments towards convincing me
>>> [and
>>> perhaps others] would be needed here).
> I agree that this may not be the best approach, but it seems like we
> don't have too many options to turn off a config for randconfig.
> 
> To be honest, in my opinion (IMO), randconfig should either be removed
> or allowed to fail until most of the functionality is ready. Otherwise,
> there should be a need to introduce HAS_* or depend on
> 'SUPPORTED_ARCHS' for each config, or introduce a lot of stubs.
> 
> Could you please suggest a better option?

As to dropping randconfig tests, I'd like to refer you to Andrew, who
is of the opinion that it was wrong to drop them for ppc. (I'm agreeing
with him when taking a theoretical perspective, but I'm not happy with
the practical consequences.)

As to a better approach: Instead of listing the same set of options
several times, can't there be a template config which is used to force
randconfig to not touch certain settings? In fact at least for non-
randconfig purposes I thought tiny64_defconfig / riscv64_defconfig
already serve kind of a similar purpose. Imo the EXTRA_*CONFIG overrides
are there for at most very few special case settings, not for purposes
like you use them here.

>>>>> --- a/automation/gitlab-ci/build.yaml
>>>>> +++ b/automation/gitlab-ci/build.yaml
>>>>> @@ -522,6 +522,38 @@ archlinux-current-gcc-riscv64:
>>>>>      CONTAINER: archlinux:current-riscv64
>>>>>      KBUILD_DEFCONFIG: tiny64_defconfig
>>>>>      HYPERVISOR_ONLY: y
>>>>> +    EXTRA_XEN_CONFIG:
>>>>> +      CONFIG_COVERAGE=n
>>>>> +      CONFIG_GRANT_TABLE=n
>>>>> +      CONFIG_SCHED_CREDIT=n
>>>>> +      CONFIG_SCHED_CREDIT2=n
>>>>> +      CONFIG_SCHED_RTDS=n
>>>>> +      CONFIG_SCHED_NULL=n
>>>>> +      CONFIG_SCHED_ARINC653=n
>>>>> +      CONFIG_TRACEBUFFER=n
>>>>> +      CONFIG_HYPFS=n
>>>>> +      CONFIG_GRANT_TABLE=n
>>>>> +      CONFIG_SPECULATIVE_HARDEN_ARRAY=n
>>>>> +      CONFIG_ARGO=n
>>>>> +      CONFIG_HYPFS_CONFIG=n
>>>>> +      CONFIG_CORE_PARKING=n
>>>>> +      CONFIG_DEBUG_TRACE=n
>>>>> +      CONFIG_IOREQ_SERVER=n
>>>>> +      CONFIG_CRASH_DEBUG=n
>>>>> +      CONFIG_KEXEC=n
>>>>> +      CONFIG_LIVEPATCH=n
>>>>> +      CONFIG_MEM_ACCESS=n
>>>>> +      CONFIG_NUMA=n
>>>>> +      CONFIG_PERF_COUNTERS=n
>>>>> +      CONFIG_HAS_PMAP=n
>>>>> +      CONFIG_TRACEBUFFER=n
>>>>> +      CONFIG_XENOPROF=n
>>>>> +      CONFIG_COMPAT=n
>>>>> +      CONFIG_COVERAGE=n
>>>>> +      CONFIG_UBSAN=n
>>>>> +      CONFIG_NEEDS_LIBELF=n
>>>>> +      CONFIG_XSM=n
>>>>> +
>>>>>  
>>>>>  archlinux-current-gcc-riscv64-debug:
>>>>>    extends: .gcc-riscv64-cross-build-debug
>>>
>>> I think I've said so elsewhere before: Please avoid introducing
>>> double
>>> blank lines, unless entirely unavoidable (for reasons I cannot
>>> think
>>> of).
> Sorry for that; I am not doing that on purpose. It's just a big patch
> series, and I missed that double blank. I will try to be more
> attentive.
> 
> Do you think it makes sense to add a script to perform basic code style
> checks, similar to what Linux has?

Such a script would be nice, but it doesn't exist and re-using existing
checkers has also proven controversial. There's actually an ongoing
discussion on this topic which you may want to follow.

Jan

Re: [PATCH v2 01/39] xen/riscv: disable unnecessary configs

[Oleksii]

On Thu, 2023-12-07 at 11:00 +0100, Jan Beulich wrote:
> On 07.12.2023 10:22, Oleksii wrote:
> > On Tue, 2023-12-05 at 16:38 +0100, Jan Beulich wrote:
> > > > On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > > > > > The patch also fixes the build script as conf util expects
> > > > > > to have each config on separate line.
> > > > 
> > > > The approach doesn't really scale (it's already odd that you
> > > > add
> > > > the
> > > > (apparently) same set four times. There's also zero
> > > > justification
> > > > for
> > > > this kind of an adjustment (as per discussion elsewhere I don't
> > > > think
> > > > we should go this route, and hence arguments towards convincing
> > > > me
> > > > [and
> > > > perhaps others] would be needed here).
> > I agree that this may not be the best approach, but it seems like
> > we
> > don't have too many options to turn off a config for randconfig.
> > 
> > To be honest, in my opinion (IMO), randconfig should either be
> > removed
> > or allowed to fail until most of the functionality is ready.
> > Otherwise,
> > there should be a need to introduce HAS_* or depend on
> > 'SUPPORTED_ARCHS' for each config, or introduce a lot of stubs.
> > 
> > Could you please suggest a better option?
> 
> As to dropping randconfig tests, I'd like to refer you to Andrew, who
> is of the opinion that it was wrong to drop them for ppc. (I'm
> agreeing
> with him when taking a theoretical perspective, but I'm not happy
> with
> the practical consequences.)
> 
> As to a better approach: Instead of listing the same set of options
> several times, can't there be a template config which is used to
> force
> randconfig to not touch certain settings? In fact at least for non-
> randconfig purposes I thought tiny64_defconfig / riscv64_defconfig
> already serve kind of a similar purpose. Imo the EXTRA_*CONFIG
> overrides
> are there for at most very few special case settings, not for
> purposes
> like you use them here.
The template will be the really a good option.

What do you think about the following patch which introduces arch-
specific allrandom.config?

diff --git a/xen/Makefile b/xen/Makefile
index ca571103c8..cb1eca76c2 100644
--- a/xen/Makefile
+++ b/xen/Makefile
@@ -336,11 +336,14 @@ ifeq ($(config-build),y)
 # *config targets only - make sure prerequisites are updated, and
descend
 # in tools/kconfig to make the *config target
 
+ARCH_ALLRANDOM_CONFIG :=
$(srctree)/arch/$(SRCARCH)/configs/allrandom.config
+
 # Create a file for KCONFIG_ALLCONFIG which depends on the
environment.
 # This will be use by kconfig targets
allyesconfig/allmodconfig/allnoconfig/randconfig
 filechk_kconfig_allconfig = \
     $(if $(findstring n,$(XEN_HAS_CHECKPOLICY)), echo
'CONFIG_XSM_FLASK_POLICY=n';) \
-    $(if $(KCONFIG_ALLCONFIG), cat $(KCONFIG_ALLCONFIG);) \
+    $(if $(KCONFIG_ALLCONFIG), cat $(KCONFIG_ALLCONFIG); \
+    $(if $(wildcard $(ARCH_ALLRANDOM_CONFIG)), cat
$(ARCH_ALLRANDOM_CONFIG);) ) \
     :

If this patch is OK then it can be reused for patches:
https://lore.kernel.org/xen-devel/cdc20255540a66ba0b6946ac6d48c11029cd3385.1701453087.git.oleksii.kurochko@gmail.com/
https://lore.kernel.org/xen-devel/d42a34866edc70a12736b5c6976aa1b44b4ebd8a.1701453087.git.oleksii.kurochko@gmail.com/

> 
> > > > > > --- a/automation/gitlab-ci/build.yaml
> > > > > > +++ b/automation/gitlab-ci/build.yaml
> > > > > > @@ -522,6 +522,38 @@ archlinux-current-gcc-riscv64:
> > > > > >      CONTAINER: archlinux:current-riscv64
> > > > > >      KBUILD_DEFCONFIG: tiny64_defconfig
> > > > > >      HYPERVISOR_ONLY: y
> > > > > > +    EXTRA_XEN_CONFIG:
> > > > > > +      CONFIG_COVERAGE=n
> > > > > > +      CONFIG_GRANT_TABLE=n
> > > > > > +      CONFIG_SCHED_CREDIT=n
> > > > > > +      CONFIG_SCHED_CREDIT2=n
> > > > > > +      CONFIG_SCHED_RTDS=n
> > > > > > +      CONFIG_SCHED_NULL=n
> > > > > > +      CONFIG_SCHED_ARINC653=n
> > > > > > +      CONFIG_TRACEBUFFER=n
> > > > > > +      CONFIG_HYPFS=n
> > > > > > +      CONFIG_GRANT_TABLE=n
> > > > > > +      CONFIG_SPECULATIVE_HARDEN_ARRAY=n
> > > > > > +      CONFIG_ARGO=n
> > > > > > +      CONFIG_HYPFS_CONFIG=n
> > > > > > +      CONFIG_CORE_PARKING=n
> > > > > > +      CONFIG_DEBUG_TRACE=n
> > > > > > +      CONFIG_IOREQ_SERVER=n
> > > > > > +      CONFIG_CRASH_DEBUG=n
> > > > > > +      CONFIG_KEXEC=n
> > > > > > +      CONFIG_LIVEPATCH=n
> > > > > > +      CONFIG_MEM_ACCESS=n
> > > > > > +      CONFIG_NUMA=n
> > > > > > +      CONFIG_PERF_COUNTERS=n
> > > > > > +      CONFIG_HAS_PMAP=n
> > > > > > +      CONFIG_TRACEBUFFER=n
> > > > > > +      CONFIG_XENOPROF=n
> > > > > > +      CONFIG_COMPAT=n
> > > > > > +      CONFIG_COVERAGE=n
> > > > > > +      CONFIG_UBSAN=n
> > > > > > +      CONFIG_NEEDS_LIBELF=n
> > > > > > +      CONFIG_XSM=n
> > > > > > +
> > > > > >  
> > > > > >  archlinux-current-gcc-riscv64-debug:
> > > > > >    extends: .gcc-riscv64-cross-build-debug
> > > > 
> > > > I think I've said so elsewhere before: Please avoid introducing
> > > > double
> > > > blank lines, unless entirely unavoidable (for reasons I cannot
> > > > think
> > > > of).
> > Sorry for that; I am not doing that on purpose. It's just a big
> > patch
> > series, and I missed that double blank. I will try to be more
> > attentive.
> > 
> > Do you think it makes sense to add a script to perform basic code
> > style
> > checks, similar to what Linux has?
> 
> Such a script would be nice, but it doesn't exist and re-using
> existing
> checkers has also proven controversial. There's actually an ongoing
> discussion on this topic which you may want to follow.
Yes, I would like to follow. I'll search the topic in ML.
Thanks.

~ Oleksii

Re: [PATCH v2 01/39] xen/riscv: disable unnecessary configs

[Jan Beulich]

On 07.12.2023 14:44, Oleksii wrote:
> On Thu, 2023-12-07 at 11:00 +0100, Jan Beulich wrote:
>> On 07.12.2023 10:22, Oleksii wrote:
>>> On Tue, 2023-12-05 at 16:38 +0100, Jan Beulich wrote:
>>>>> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>>>>>>> The patch also fixes the build script as conf util expects
>>>>>>> to have each config on separate line.
>>>>>
>>>>> The approach doesn't really scale (it's already odd that you
>>>>> add
>>>>> the
>>>>> (apparently) same set four times. There's also zero
>>>>> justification
>>>>> for
>>>>> this kind of an adjustment (as per discussion elsewhere I don't
>>>>> think
>>>>> we should go this route, and hence arguments towards convincing
>>>>> me
>>>>> [and
>>>>> perhaps others] would be needed here).
>>> I agree that this may not be the best approach, but it seems like
>>> we
>>> don't have too many options to turn off a config for randconfig.
>>>
>>> To be honest, in my opinion (IMO), randconfig should either be
>>> removed
>>> or allowed to fail until most of the functionality is ready.
>>> Otherwise,
>>> there should be a need to introduce HAS_* or depend on
>>> 'SUPPORTED_ARCHS' for each config, or introduce a lot of stubs.
>>>
>>> Could you please suggest a better option?
>>
>> As to dropping randconfig tests, I'd like to refer you to Andrew, who
>> is of the opinion that it was wrong to drop them for ppc. (I'm
>> agreeing
>> with him when taking a theoretical perspective, but I'm not happy
>> with
>> the practical consequences.)
>>
>> As to a better approach: Instead of listing the same set of options
>> several times, can't there be a template config which is used to
>> force
>> randconfig to not touch certain settings? In fact at least for non-
>> randconfig purposes I thought tiny64_defconfig / riscv64_defconfig
>> already serve kind of a similar purpose. Imo the EXTRA_*CONFIG
>> overrides
>> are there for at most very few special case settings, not for
>> purposes
>> like you use them here.
> The template will be the really a good option.
> 
> What do you think about the following patch which introduces arch-
> specific allrandom.config?
> 
> diff --git a/xen/Makefile b/xen/Makefile
> index ca571103c8..cb1eca76c2 100644
> --- a/xen/Makefile
> +++ b/xen/Makefile
> @@ -336,11 +336,14 @@ ifeq ($(config-build),y)
>  # *config targets only - make sure prerequisites are updated, and
> descend
>  # in tools/kconfig to make the *config target
>  
> +ARCH_ALLRANDOM_CONFIG :=
> $(srctree)/arch/$(SRCARCH)/configs/allrandom.config
> +
>  # Create a file for KCONFIG_ALLCONFIG which depends on the
> environment.
>  # This will be use by kconfig targets
> allyesconfig/allmodconfig/allnoconfig/randconfig
>  filechk_kconfig_allconfig = \
>      $(if $(findstring n,$(XEN_HAS_CHECKPOLICY)), echo
> 'CONFIG_XSM_FLASK_POLICY=n';) \
> -    $(if $(KCONFIG_ALLCONFIG), cat $(KCONFIG_ALLCONFIG);) \
> +    $(if $(KCONFIG_ALLCONFIG), cat $(KCONFIG_ALLCONFIG); \
> +    $(if $(wildcard $(ARCH_ALLRANDOM_CONFIG)), cat
> $(ARCH_ALLRANDOM_CONFIG);) ) \
>      :

Something along these lines may be okay, but why would the name be
"allrandom" when the config is used elsewhere as well? Further, besides
keeping randconfig and all*config from creating unusable configs, it
will at least want considering whether in other cases that set of fixed
values shouldn't be used as well then.

Jan

Re: [PATCH v2 08/39] xen/riscv: introduce asm/cpufeature.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>

Acked-by: Jan Beulich <jbeulich@suse.com>

Would have been nice ...

> --- /dev/null
> +++ b/xen/arch/riscv/include/asm/cpufeature.h
> @@ -0,0 +1,22 @@
> +#ifndef __ASM_RISCV_CPUFEATURE_H
> +#define __ASM_RISCV_CPUFEATURE_H
> +
> +#ifndef __ASSEMBLY__
> +
> +static inline int cpu_nr_siblings(unsigned int cpu)

... for this to have return type unsigned int, but I see you're staying in
line with oddities elsewhere. Just one more place to touch down the road.

Jan

Re: [PATCH v2 09/39] xen/riscv: introduce asm/guest_atomics.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> --- /dev/null
> +++ b/xen/arch/riscv/include/asm/guest_atomics.h
> @@ -0,0 +1,48 @@
> +#ifndef __ASM_RISCV_GUEST_ATOMICS_H
> +#define __ASM_RISCV_GUEST_ATOMICS_H
> +
> +/*
> + * TODO: implement guest atomics
> + */

Along with this, wouldn't it be better to have e.g. ASSERT_UNREACHABLE()
in the unimplemented functions?

Jan

> +#define guest_testop(name)                                                  \
> +static inline int guest_##name(struct domain *d, int nr, volatile void *p)  \
> +{                                                                           \
> +    (void) d;                                                               \
> +    (void) nr;                                                              \
> +    (void) p;                                                               \
> +                                                                            \
> +    return 0;                                                               \
> +}
> +
> +#define guest_bitop(name)                                                   \
> +static inline void guest_##name(struct domain *d, int nr, volatile void *p) \
> +{                                                                           \
> +    (void) d;                                                               \
> +    (void) nr;                                                              \
> +    (void) p;                                                               \
> +}
> +
> +guest_bitop(set_bit)
> +guest_bitop(clear_bit)
> +guest_bitop(change_bit)
> +
> +#undef guest_bitop
> +
> +guest_testop(test_and_set_bit)
> +guest_testop(test_and_clear_bit)
> +guest_testop(test_and_change_bit)
> +
> +#undef guest_testop
> +
> +#define guest_test_bit(d, nr, p) ((void)(d), test_bit(nr, p))
> +
> +#endif /* __ASM_RISCV_GUEST_ATOMICS_H */
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */

Re: [PATCH v2 10/39] xen/riscv: introduce asm/iommu.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> --- /dev/null
> +++ b/xen/arch/riscv/include/asm/iommu.h
> @@ -0,0 +1,7 @@
> +#ifndef __ASM_RISCV_IOMMU_H__
> +#define __ASM_RISCV_IOMMU_H__
> +
> +struct arch_iommu {
> +};
> +
> +#endif /* __ASM_IOMMU_H__ */

Instead of adding this header, didn't we discuss to make the #include in
xen/iommu.h depend on CONFIG_HAS_PASSTHROUGH? Also - no SPDX or footer
here?

Jan

Re: [PATCH v2 08/39] xen/riscv: introduce asm/cpufeature.h

[Jan Beulich]

On 07.12.2023 15:19, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> 
> Acked-by: Jan Beulich <jbeulich@suse.com>

Actually - with an SPDX header added. I only now realize that I
committed the earlier two patches without paying attention to this
aspect. I'd appreciate if in the next version you could include an
incremental change. And obviously in all other new headers such a
comment (and perhaps also a formatting footer) wants introducing
as well.

Jan

Re: [PATCH v2 11/39] xen/riscv: introduce asm/nospec.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> --- /dev/null
> +++ b/xen/arch/riscv/include/asm/nospec.h
> @@ -0,0 +1,25 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. */
> +
> +#ifndef _ASM_RISCV_NOSPEC_H
> +#define _ASM_RISCV_NOSPEC_H
> +
> +static inline bool evaluate_nospec(bool condition)
> +{
> +    return condition;
> +}
> +
> +static inline void block_speculation(void)
> +{
> +}
> +
> +#endif /* _ASM_RISCV_NOSPEC_H */

This being identical between Arm, PPC, and now RISC-V, wouldn't this be another
asm-generic/ candidate? (Whether such trivial stubs are copyrightable is, as
per earlier remarks, at least questionable to me.)

Jan

Re: [PATCH v2 12/39] xen/riscv: introduce asm/setup.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>

With at least SPDX header
Acked-by: Jan Beulich <jbeulich@suse.com>

Re: [PATCH v2 00/39] Enable build of full Xen for RISC-V

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Bobby Eshleman (1):
>   xen/riscv: introduce asm/atomic.h
> 
> Oleksii Kurochko (38):
>   xen/riscv: disable unnecessary configs
>   xen/riscv: use some asm-generic headers
>   xen/riscv:introduce asm/byteorder.h
>   xen/riscv: add public arch-riscv.h
>   xen/riscv: introduce spinlock.h
>   xen/riscv: introduce fence.h
>   xen/riscv: introduce arch-riscv/hvm/save.h
>   xen/riscv: introduce asm/cpufeature.h
>   xen/riscv: introduce asm/guest_atomics.h
>   xen/riscv: introduce asm/iommu.h
>   xen/riscv: introduce asm/nospec.h
>   xen/riscv: introduce asm/setup.h
>   xen/riscv: introduce asm/system.h
>   xen/riscv: introduce bitops.h
>   xen/riscv: introduce flushtlb.h
>   xen/riscv: introduce asm/smp.h
>   xen/riscv: introduce cmpxchg.h
>   xen/riscv: introduce asm/io.h
>   xen/riscv: define bug frame tables in xen.lds.S
>   xen/riscv: introduce bit operations
>   xen/riscv: introduce asm/domain.h
>   xen/riscv: introduce asm/guest_access.h
>   xen/riscv: introduce asm/irq.h
>   xen/riscv: introduce asm/p2m.h
>   xen/riscv: introduce asm/regs.h
>   xen/riscv: introduce asm/time.h
>   xen/riscv: introduce asm/event.h

Throughout here, would you please try to be consistent about (not) using asm/
prefixes?

Jan

Re: [PATCH v2 01/39] xen/riscv: disable unnecessary configs

[Oleksii]

On Thu, 2023-12-07 at 15:11 +0100, Jan Beulich wrote:
> On 07.12.2023 14:44, Oleksii wrote:
> > On Thu, 2023-12-07 at 11:00 +0100, Jan Beulich wrote:
> > > On 07.12.2023 10:22, Oleksii wrote:
> > > > On Tue, 2023-12-05 at 16:38 +0100, Jan Beulich wrote:
> > > > > > On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > > > > > > > The patch also fixes the build script as conf util
> > > > > > > > expects
> > > > > > > > to have each config on separate line.
> > > > > > 
> > > > > > The approach doesn't really scale (it's already odd that
> > > > > > you
> > > > > > add
> > > > > > the
> > > > > > (apparently) same set four times. There's also zero
> > > > > > justification
> > > > > > for
> > > > > > this kind of an adjustment (as per discussion elsewhere I
> > > > > > don't
> > > > > > think
> > > > > > we should go this route, and hence arguments towards
> > > > > > convincing
> > > > > > me
> > > > > > [and
> > > > > > perhaps others] would be needed here).
> > > > I agree that this may not be the best approach, but it seems
> > > > like
> > > > we
> > > > don't have too many options to turn off a config for
> > > > randconfig.
> > > > 
> > > > To be honest, in my opinion (IMO), randconfig should either be
> > > > removed
> > > > or allowed to fail until most of the functionality is ready.
> > > > Otherwise,
> > > > there should be a need to introduce HAS_* or depend on
> > > > 'SUPPORTED_ARCHS' for each config, or introduce a lot of stubs.
> > > > 
> > > > Could you please suggest a better option?
> > > 
> > > As to dropping randconfig tests, I'd like to refer you to Andrew,
> > > who
> > > is of the opinion that it was wrong to drop them for ppc. (I'm
> > > agreeing
> > > with him when taking a theoretical perspective, but I'm not happy
> > > with
> > > the practical consequences.)
> > > 
> > > As to a better approach: Instead of listing the same set of
> > > options
> > > several times, can't there be a template config which is used to
> > > force
> > > randconfig to not touch certain settings? In fact at least for
> > > non-
> > > randconfig purposes I thought tiny64_defconfig /
> > > riscv64_defconfig
> > > already serve kind of a similar purpose. Imo the EXTRA_*CONFIG
> > > overrides
> > > are there for at most very few special case settings, not for
> > > purposes
> > > like you use them here.
> > The template will be the really a good option.
> > 
> > What do you think about the following patch which introduces arch-
> > specific allrandom.config?
> > 
> > diff --git a/xen/Makefile b/xen/Makefile
> > index ca571103c8..cb1eca76c2 100644
> > --- a/xen/Makefile
> > +++ b/xen/Makefile
> > @@ -336,11 +336,14 @@ ifeq ($(config-build),y)
> >  # *config targets only - make sure prerequisites are updated, and
> > descend
> >  # in tools/kconfig to make the *config target
> >  
> > +ARCH_ALLRANDOM_CONFIG :=
> > $(srctree)/arch/$(SRCARCH)/configs/allrandom.config
> > +
> >  # Create a file for KCONFIG_ALLCONFIG which depends on the
> > environment.
> >  # This will be use by kconfig targets
> > allyesconfig/allmodconfig/allnoconfig/randconfig
> >  filechk_kconfig_allconfig = \
> >      $(if $(findstring n,$(XEN_HAS_CHECKPOLICY)), echo
> > 'CONFIG_XSM_FLASK_POLICY=n';) \
> > -    $(if $(KCONFIG_ALLCONFIG), cat $(KCONFIG_ALLCONFIG);) \
> > +    $(if $(KCONFIG_ALLCONFIG), cat $(KCONFIG_ALLCONFIG); \
> > +    $(if $(wildcard $(ARCH_ALLRANDOM_CONFIG)), cat
> > $(ARCH_ALLRANDOM_CONFIG);) ) \
> >      :
> 
> Something along these lines may be okay, but why would the name be
> "allrandom" when the config is used elsewhere as well?
The naming is not optimal. "unused.config" or "ignored.config" would be
a better choice.

>  Further, besides
> keeping randconfig and all*config from creating unusable configs, it
> will at least want considering whether in other cases that set of
> fixed
> values shouldn't be used as well then.
If I understood you correctly, the other case is *defconfig targets.
Therefore, the following targets might also need to be updated by
merging "unused.config" with {defconfig,%_defconfig}:


defconfig: $(obj)/conf
ifneq ($(wildcard
$(srctree)/arch/$(SRCARCH)/configs/$(KBUILD_DEFCONFIG)),)
	@$(kecho) "*** Default configuration is based on
'$(KBUILD_DEFCONFIG)'"
	$(Q)$< $(silent) --
defconfig=arch/$(SRCARCH)/configs/$(KBUILD_DEFCONFIG) $(Kconfig)
else
	@$(kecho) "*** Default configuration is based on target
'$(KBUILD_DEFCONFIG)'"
	$(Q)$(MAKE) -f $(srctree)/Makefile $(KBUILD_DEFCONFIG)
endif

%_defconfig: $(obj)/conf
	$(Q)$< $(silent) --defconfig=arch/$(SRCARCH)/configs/$@
$(Kconfig)

However, I believe it's possible that for *defconfig, a configuration
should be set to N, but in randconfig, it is still acceptable to be set
to Y.

~ Oleksii

Re: [PATCH v2 13/39] xen/riscv: introduce asm/system.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> --- /dev/null
> +++ b/xen/arch/riscv/include/asm/system.h
> @@ -0,0 +1,79 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +#ifndef _ASM_RISCV_BARRIER_H
> +#define _ASM_RISCV_BARRIER_H
> +
> +#include <asm/csr.h>
> +
> +#ifndef __ASSEMBLY__
> +
> +#define RISCV_FENCE(p, s) \
> +    __asm__ __volatile__ ("fence " #p "," #s : : : "memory")

Nit (style): Missing blanks immediately inside the parentheses.

> +/* These barriers need to enforce ordering on both devices or memory. */
> +#define mb()                    RISCV_FENCE(iorw,iorw)
> +#define rmb()                   RISCV_FENCE(ir,ir)
> +#define wmb()                   RISCV_FENCE(ow,ow)

Nit (style): Missing blanks after the commas (also again below).

> +/* These barriers do not need to enforce ordering on devices, just memory. */
> +#define smp_mb()                RISCV_FENCE(rw,rw)
> +#define smp_rmb()               RISCV_FENCE(r,r)
> +#define smp_wmb()               RISCV_FENCE(w,w)
> +#define smp_mb__before_atomic() smp_mb()
> +#define smp_mb__after_atomic()  smp_mb()
> +
> +/*
> +#define __smp_store_release(p, v)       \

Is there a need for the double underscores here? We try to not
introduce new instances of undue leading underscores, but there might
be e.g. a strong desire to stay in sync with, say, Linux.

> +do {                                    \
> +	compiletime_assert_atomic_type(*p); \
> +	RISCV_FENCE(rw,w);                  \
> +	WRITE_ONCE(*p, v);                  \

Nit: Can the trailing backslashes be aligned, please?

> +} while (0)
> +
> +#define __smp_load_acquire(p)           \
> +({                                      \
> +    typeof(*p) ___p1 = READ_ONCE(*p);   \

Hmm, yet more leading underscores, and here surely not needed.

> +    compiletime_assert_atomic_type(*p); \
> +    RISCV_FENCE(r,rw);                  \
> +    ___p1;                              \
> +})
> +*/
> +
> +static inline unsigned long local_save_flags(void)
> +{
> +    return csr_read(sstatus);
> +}
> +
> +static inline void local_irq_enable(void)
> +{
> +    csr_set(sstatus, SSTATUS_SIE);
> +}
> +
> +static inline void local_irq_disable(void)
> +{
> +    csr_clear(sstatus, SSTATUS_SIE);
> +}
> +
> +#define local_irq_save(x)                           \
> +({                                                  \
> +    x = csr_read_clear(CSR_SSTATUS, SSTATUS_SIE);   \
> +    local_irq_disable();                            \
> +})
> +
> +static inline void local_irq_restore(unsigned long flags)
> +{
> +	csr_set(CSR_SSTATUS, flags & SSTATUS_SIE);
> +}
> +
> +static inline int local_irq_is_enabled(void)
> +{
> +    unsigned long flags = local_save_flags();
> +
> +    return flags & SSTATUS_SIE;

SSTATUS_SIE doesn't even happen to be 1, so I think you're better off
adding != 0, unless you would do as I think I had suggested before and
have the function return bool right away.

Jan

Re: [PATCH v2 01/39] xen/riscv: disable unnecessary configs

[Jan Beulich]

On 07.12.2023 15:51, Oleksii wrote:
> On Thu, 2023-12-07 at 15:11 +0100, Jan Beulich wrote:
>> On 07.12.2023 14:44, Oleksii wrote:
>>> On Thu, 2023-12-07 at 11:00 +0100, Jan Beulich wrote:
>>>> On 07.12.2023 10:22, Oleksii wrote:
>>>>> On Tue, 2023-12-05 at 16:38 +0100, Jan Beulich wrote:
>>>>>>> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>>>>>>>>> The patch also fixes the build script as conf util
>>>>>>>>> expects
>>>>>>>>> to have each config on separate line.
>>>>>>>
>>>>>>> The approach doesn't really scale (it's already odd that
>>>>>>> you
>>>>>>> add
>>>>>>> the
>>>>>>> (apparently) same set four times. There's also zero
>>>>>>> justification
>>>>>>> for
>>>>>>> this kind of an adjustment (as per discussion elsewhere I
>>>>>>> don't
>>>>>>> think
>>>>>>> we should go this route, and hence arguments towards
>>>>>>> convincing
>>>>>>> me
>>>>>>> [and
>>>>>>> perhaps others] would be needed here).
>>>>> I agree that this may not be the best approach, but it seems
>>>>> like
>>>>> we
>>>>> don't have too many options to turn off a config for
>>>>> randconfig.
>>>>>
>>>>> To be honest, in my opinion (IMO), randconfig should either be
>>>>> removed
>>>>> or allowed to fail until most of the functionality is ready.
>>>>> Otherwise,
>>>>> there should be a need to introduce HAS_* or depend on
>>>>> 'SUPPORTED_ARCHS' for each config, or introduce a lot of stubs.
>>>>>
>>>>> Could you please suggest a better option?
>>>>
>>>> As to dropping randconfig tests, I'd like to refer you to Andrew,
>>>> who
>>>> is of the opinion that it was wrong to drop them for ppc. (I'm
>>>> agreeing
>>>> with him when taking a theoretical perspective, but I'm not happy
>>>> with
>>>> the practical consequences.)
>>>>
>>>> As to a better approach: Instead of listing the same set of
>>>> options
>>>> several times, can't there be a template config which is used to
>>>> force
>>>> randconfig to not touch certain settings? In fact at least for
>>>> non-
>>>> randconfig purposes I thought tiny64_defconfig /
>>>> riscv64_defconfig
>>>> already serve kind of a similar purpose. Imo the EXTRA_*CONFIG
>>>> overrides
>>>> are there for at most very few special case settings, not for
>>>> purposes
>>>> like you use them here.
>>> The template will be the really a good option.
>>>
>>> What do you think about the following patch which introduces arch-
>>> specific allrandom.config?
>>>
>>> diff --git a/xen/Makefile b/xen/Makefile
>>> index ca571103c8..cb1eca76c2 100644
>>> --- a/xen/Makefile
>>> +++ b/xen/Makefile
>>> @@ -336,11 +336,14 @@ ifeq ($(config-build),y)
>>>  # *config targets only - make sure prerequisites are updated, and
>>> descend
>>>  # in tools/kconfig to make the *config target
>>>  
>>> +ARCH_ALLRANDOM_CONFIG :=
>>> $(srctree)/arch/$(SRCARCH)/configs/allrandom.config
>>> +
>>>  # Create a file for KCONFIG_ALLCONFIG which depends on the
>>> environment.
>>>  # This will be use by kconfig targets
>>> allyesconfig/allmodconfig/allnoconfig/randconfig
>>>  filechk_kconfig_allconfig = \
>>>      $(if $(findstring n,$(XEN_HAS_CHECKPOLICY)), echo
>>> 'CONFIG_XSM_FLASK_POLICY=n';) \
>>> -    $(if $(KCONFIG_ALLCONFIG), cat $(KCONFIG_ALLCONFIG);) \
>>> +    $(if $(KCONFIG_ALLCONFIG), cat $(KCONFIG_ALLCONFIG); \
>>> +    $(if $(wildcard $(ARCH_ALLRANDOM_CONFIG)), cat
>>> $(ARCH_ALLRANDOM_CONFIG);) ) \
>>>      :
>>
>> Something along these lines may be okay, but why would the name be
>> "allrandom" when the config is used elsewhere as well?
> The naming is not optimal. "unused.config" or "ignored.config" would be
> a better choice.

I don't think "unused" or "ignored" are properly describing what such
a file would hold. I was vaguely playing with "fixed" or "forced", fwiw.

>>  Further, besides
>> keeping randconfig and all*config from creating unusable configs, it
>> will at least want considering whether in other cases that set of
>> fixed
>> values shouldn't be used as well then.
> If I understood you correctly, the other case is *defconfig targets.
> Therefore, the following targets might also need to be updated by
> merging "unused.config" with {defconfig,%_defconfig}:
> 
> 
> defconfig: $(obj)/conf
> ifneq ($(wildcard
> $(srctree)/arch/$(SRCARCH)/configs/$(KBUILD_DEFCONFIG)),)
> 	@$(kecho) "*** Default configuration is based on
> '$(KBUILD_DEFCONFIG)'"
> 	$(Q)$< $(silent) --
> defconfig=arch/$(SRCARCH)/configs/$(KBUILD_DEFCONFIG) $(Kconfig)
> else
> 	@$(kecho) "*** Default configuration is based on target
> '$(KBUILD_DEFCONFIG)'"
> 	$(Q)$(MAKE) -f $(srctree)/Makefile $(KBUILD_DEFCONFIG)
> endif
> 
> %_defconfig: $(obj)/conf
> 	$(Q)$< $(silent) --defconfig=arch/$(SRCARCH)/configs/$@
> $(Kconfig)
> 
> However, I believe it's possible that for *defconfig, a configuration
> should be set to N, but in randconfig, it is still acceptable to be set
> to Y.

I'm not sure *defconfig should be affected by what's in the "locked
down" config, assuming they're consistent with what may not be changed.
But of course one could also model this incrementally, such that
defconfigs don't need to repeat what the "locked down" one has.

Anyway, I think this needs sending an RFC with the right people in Cc,
such that this can be properly discussed.

Jan

Re: [PATCH v2 14/39] xen/riscv: introduce bitops.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>

So this looks to have been taken from Linux, which could do with saying
(including which version or most recent commit). It may e.g. justify you
using tab indentation here, albeit ...

> --- /dev/null
> +++ b/xen/arch/riscv/include/asm/bitops.h
> @@ -0,0 +1,288 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/* Copyright (C) 2012 Regents of the University of California */
> +
> +#ifndef _ASM_RISCV_BITOPS_H
> +#define _ASM_RISCV_BITOPS_H
> +
> +#include <asm/system.h>
> +
> +#define BITOP_BITS_PER_WORD     32
> +#define BITOP_MASK(nr)	        (1UL << ((nr) % BITOP_BITS_PER_WORD))
> +#define BITOP_WORD(nr)	        ((nr) / BITOP_BITS_PER_WORD)
> +#define BITS_PER_BYTE	        8
> +
> +#define __set_bit(n,p)          set_bit(n,p)
> +#define __clear_bit(n,p)        clear_bit(n,p)

... then please consistently. Other style related remarks made on the
system.h patch apply here as well (unless again there's a goal of
keeping the diff to the Linux original small; yet then I guess the
delta to the Linux file is already pretty large).

> +/* Based on linux/include/asm-generic/bitops/find.h */
> +
> +#ifndef find_next_bit
> +/**
> + * find_next_bit - find the next set bit in a memory region
> + * @addr: The address to base the search on
> + * @offset: The bitnumber to start searching at
> + * @size: The bitmap size in bits
> + */
> +extern unsigned long find_next_bit(const unsigned long *addr, unsigned long
> +		size, unsigned long offset);
> +#endif
> +
> +#ifndef find_next_zero_bit
> +/**
> + * find_next_zero_bit - find the next cleared bit in a memory region
> + * @addr: The address to base the search on
> + * @offset: The bitnumber to start searching at
> + * @size: The bitmap size in bits
> + */
> +extern unsigned long find_next_zero_bit(const unsigned long *addr, unsigned
> +		long size, unsigned long offset);
> +#endif
> +
> +/**
> + * find_first_bit - find the first set bit in a memory region
> + * @addr: The address to start the search at
> + * @size: The maximum size to search
> + *
> + * Returns the bit number of the first set bit.
> + */
> +extern unsigned long find_first_bit(const unsigned long *addr,
> +				    unsigned long size);
> +
> +/**
> + * find_first_zero_bit - find the first cleared bit in a memory region
> + * @addr: The address to start the search at
> + * @size: The maximum size to search
> + *
> + * Returns the bit number of the first cleared bit.
> + */
> +extern unsigned long find_first_zero_bit(const unsigned long *addr,
> +					 unsigned long size);

Looking over the titles of the rest of the series, I can't spot where
these are going to be implemented. The again maybe you indeed can get
away without those, at least initially.

> +#define ffs(x) ({ unsigned int __t = (x); fls(__t & -__t); })

This wants to use ISOLATE_LSB() now.

Jan

Re: [PATCH v2 15/39] xen/riscv: introduce flushtlb.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>

Again, with an SPDX header
Acked-by: Jan Beulich <jbeulich@suse.com>

Re: [PATCH v2 16/39] xen/riscv: introduce asm/smp.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> --- /dev/null
> +++ b/xen/arch/riscv/include/asm/smp.h
> @@ -0,0 +1,23 @@
> +#ifndef __ASM_RISCV_SMP_H
> +#define __ASM_RISCV_SMP_H
> +
> +#ifndef __ASSEMBLY__
> +#include <xen/cpumask.h>
> +#include <xen/percpu.h>
> +#endif

If you want this to be possible to include from assembly files (I don't
know why you would want that), ...

> +DECLARE_PER_CPU(cpumask_var_t, cpu_sibling_mask);
> +DECLARE_PER_CPU(cpumask_var_t, cpu_core_mask);

... these two would also need to live inside the #ifdef. Otherwise the
#ifdef wants dropping.

> +#define cpu_is_offline(cpu) unlikely(!cpu_online(cpu))

Seeing this is now the 4th instance, I guess we want to move it to
xen/smp.h. I'll try to remember making a patch.

> +/*
> + * Do we, for platform reasons, need to actually keep CPUs online when we
> + * would otherwise prefer them to be off?
> + */
> +#define park_offline_cpus false
> +
> +/* TODO: need to be implemeted */
> +#define smp_processor_id() (0)
> +
> +#endif
> \ No newline at end of file

You want to take care of this.

Jan

Re: [PATCH v2 17/39] xen/riscv: introduce asm/atomic.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> From: Bobby Eshleman <bobbyeshleman@gmail.com>
> 
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> ---
> Changes in V2:
>  - Change an author of commit. I got this header from Bobby's old repo.

Not sure how to deal with that when there's not also an S-o-b.

> --- /dev/null
> +++ b/xen/arch/riscv/include/asm/atomic.h
> @@ -0,0 +1,375 @@
> +/* SPDX-License-Identifier: GPL-2.0-or-later */
> +/*
> + * Taken and modified from Linux.
> + * 
> + * Copyright (C) 2007 Red Hat, Inc. All Rights Reserved.
> + * Copyright (C) 2012 Regents of the University of California
> + * Copyright (C) 2017 SiFive
> + * Copyright (C) 2021 Vates SAS
> + */
> +
> +#ifndef _ASM_RISCV_ATOMIC_H
> +#define _ASM_RISCV_ATOMIC_H
> +
> +#include <xen/atomic.h>
> +#include <asm/cmpxchg.h>

This and ...

> +#include <asm/fence.h>
> +#include <asm/io.h>

.. this header are only introduced later. Bad ordering of the series?

> +#include <asm/system.h>
> +
> +void __bad_atomic_size(void);
> +
> +static always_inline void read_atomic_size(const volatile void *p,
> +                                           void *res,
> +                                           unsigned int size)
> +{
> +    switch ( size ) {

Nit (style): Brace on its own line (again further down).

> +    case 1: *(uint8_t *)res = readb((uint8_t *)p); break;
> +    case 2: *(uint16_t *)res = readw((uint16_t *)p); break;
> +    case 4: *(uint32_t *)res = readl((uint32_t *)p); break;
> +    case 8: *(uint32_t *)res  = readq((uint64_t *)p); break;

Please don't cast away const-ness.

> +    default: __bad_atomic_size(); break;
> +    }
> +}
> +
> +#define read_atomic(p) ({                                               \
> +    union { typeof(*p) val; char c[0]; } x_;                            \

Hmm, you avoid leading underscores here, but then ...

> +    read_atomic_size(p, x_.c, sizeof(*p));                              \
> +    x_.val;                                                             \
> +})
> +
> +
> +#define write_atomic(p, x) ({                                           \
> +    typeof(*p) __x = (x);                                               \

... they're still there here.

> +    switch ( sizeof(*p) ) {                                             \
> +    case 1: writeb((uint8_t)__x,  (uint8_t *)  p); break;              \
> +    case 2: writew((uint16_t)__x, (uint16_t *) p); break;              \
> +    case 4: writel((uint32_t)__x, (uint32_t *) p); break;              \
> +    case 8: writeq((uint64_t)__x, (uint64_t *) p); break;              \
> +    default: __bad_atomic_size(); break;                                \
> +    }                                                                   \
> +    __x;                                                                \
> +})
> +
> +/* TODO: Fix this */
> +#define add_sized(p, x) ({                                              \
> +    typeof(*(p)) __x = (x);                                             \
> +    switch ( sizeof(*(p)) )                                             \
> +    {                                                                   \
> +    case 1: writeb(read_atomic(p) + __x, (uint8_t *)(p)); break;        \
> +    case 2: writew(read_atomic(p) + __x, (uint16_t *)(p)); break;       \
> +    case 4: writel(read_atomic(p) + __x, (uint32_t *)(p)); break;       \

Instead of this, considering the comment perhaps better just BUG()?

> +    default: __bad_atomic_size(); break;                                \
> +    }                                                                   \
> +})
> +
> +/*
> + *  __unqual_scalar_typeof(x) - Declare an unqualified scalar type, leaving
> + *               non-scalar types unchanged.
> + *
> + * Prefer C11 _Generic for better compile-times and simpler code. Note: 'char'
> + * is not type-compatible with 'signed char', and we define a separate case.
> + */
> +#define __scalar_type_to_expr_cases(type)               \
> +    unsigned type:  (unsigned type)0,                   \
> +    signed type:    (signed type)0
> +
> +#define __unqual_scalar_typeof(x) typeof(               \
> +    _Generic((x),                                       \

I think you still owe us an update to ./README, clarifying what compiler versions
may be used for building RISC-V. Unless of course all that exist support _Generic
(which then would be nice to say in the description).

> +        char:  (char)0,                                 \
> +        __scalar_type_to_expr_cases(char),              \
> +        __scalar_type_to_expr_cases(short),             \
> +        __scalar_type_to_expr_cases(int),               \
> +        __scalar_type_to_expr_cases(long),              \
> +        __scalar_type_to_expr_cases(long long),         \
> +        default: (x)))
> +
> +#define READ_ONCE(x)  (*(const volatile __unqual_scalar_typeof(x) *)&(x))
> +#define WRITE_ONCE(x, val)                                      \
> +    do {                                                        \
> +            *(volatile typeof(x) *)&(x) = (val);                \

Nit (style): Too deep indentation.

> +    } while (0)
> +
> +#define __atomic_acquire_fence()					\
> +	__asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory")

Suddenly using tab indentation here and below? And missing blanks again.

Jan

Re: [PATCH v2 07/39] xen/riscv: introduce arch-riscv/hvm/save.h

[Shawn Anastasio]

On 12/5/23 9:59 AM, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>> --- a/xen/include/public/hvm/save.h
>> +++ b/xen/include/public/hvm/save.h
>> @@ -91,6 +91,8 @@ DECLARE_HVM_SAVE_TYPE(END, 0, struct hvm_save_end);
>>  #include "../arch-arm/hvm/save.h"
>>  #elif defined(__powerpc64__)
>>  #include "../arch-ppc.h"
>> +#elif defined(__riscv)
>> +#include "../arch-riscv/hvm/save.h"
>>  #else
>>  #error "unsupported architecture"
>>  #endif
> 
> The PPC part here looks bogus altogether. Shawn?
>

I think my original intention here was to avoid creating yet another
empty header while still having a place to put PPC-specific definitions
that might be required.

See as how the ARM file is entirely empty though, I doubt we'll be any
different, so this could definitely be dropped.

> Jan

Thanks,
Shawn

Re: [PATCH v2 08/39] xen/riscv: introduce asm/cpufeature.h

[Oleksii]

On Thu, 2023-12-07 at 15:25 +0100, Jan Beulich wrote:
> On 07.12.2023 15:19, Jan Beulich wrote:
> > On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > > Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> > 
> > Acked-by: Jan Beulich <jbeulich@suse.com>
> 
> Actually - with an SPDX header added. I only now realize that I
> committed the earlier two patches without paying attention to this
> aspect. I'd appreciate if in the next version you could include an
> incremental change. And obviously in all other new headers such a
> comment (and perhaps also a formatting footer) wants introducing
> as well.
Thanks a lot. I'll add SPDX headers.

~ Oleksii

Re: [PATCH v2 09/39] xen/riscv: introduce asm/guest_atomics.h

[Oleksii]

On Thu, 2023-12-07 at 15:20 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > --- /dev/null
> > +++ b/xen/arch/riscv/include/asm/guest_atomics.h
> > @@ -0,0 +1,48 @@
> > +#ifndef __ASM_RISCV_GUEST_ATOMICS_H
> > +#define __ASM_RISCV_GUEST_ATOMICS_H
> > +
> > +/*
> > + * TODO: implement guest atomics
> > + */
> 
> Along with this, wouldn't it be better to have e.g.
> ASSERT_UNREACHABLE()
> in the unimplemented functions?
Thanks for recommendation. It would be better to add
ASSERT_UNREACHABLE.

~ Oleksii
> 
> > +#define
> > guest_testop(name)                                                 
> > \
> > +static inline int guest_##name(struct domain *d, int nr, volatile
> > void *p)  \
> > +{                                                                 
> >           \
> > +    (void)
> > d;                                                               \
> > +    (void)
> > nr;                                                              \
> > +    (void)
> > p;                                                               \
> > +                                                                  
> >           \
> > +    return
> > 0;                                                               \
> > +}
> > +
> > +#define
> > guest_bitop(name)                                                  
> > \
> > +static inline void guest_##name(struct domain *d, int nr, volatile
> > void *p) \
> > +{                                                                 
> >           \
> > +    (void)
> > d;                                                               \
> > +    (void)
> > nr;                                                              \
> > +    (void)
> > p;                                                               \
> > +}
> > +
> > +guest_bitop(set_bit)
> > +guest_bitop(clear_bit)
> > +guest_bitop(change_bit)
> > +
> > +#undef guest_bitop
> > +
> > +guest_testop(test_and_set_bit)
> > +guest_testop(test_and_clear_bit)
> > +guest_testop(test_and_change_bit)
> > +
> > +#undef guest_testop
> > +
> > +#define guest_test_bit(d, nr, p) ((void)(d), test_bit(nr, p))
> > +
> > +#endif /* __ASM_RISCV_GUEST_ATOMICS_H */
> > +/*
> > + * Local variables:
> > + * mode: C
> > + * c-file-style: "BSD"
> > + * c-basic-offset: 4
> > + * indent-tabs-mode: nil
> > + * End:
> > + */
> 

Re: [PATCH v2 10/39] xen/riscv: introduce asm/iommu.h

[Oleksii]

On Thu, 2023-12-07 at 15:22 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > --- /dev/null
> > +++ b/xen/arch/riscv/include/asm/iommu.h
> > @@ -0,0 +1,7 @@
> > +#ifndef __ASM_RISCV_IOMMU_H__
> > +#define __ASM_RISCV_IOMMU_H__
> > +
> > +struct arch_iommu {
> > +};
> > +
> > +#endif /* __ASM_IOMMU_H__ */
> 
> Instead of adding this header, didn't we discuss to make the #include
> in
> xen/iommu.h depend on CONFIG_HAS_PASSTHROUGH? Also - no SPDX or
> footer
> here?
We had discussion about some stuff in device.h, but we can apply it
here too. Only to place will be needed to update:


+ #ifdef CONFIG_HAS_PASSTHROUGH
#include <asm/iommu.h>
+ #endif

#ifndef iommu_call
# define iommu_call(ops, fn, args...) ((ops)->fn(args))
# define iommu_vcall iommu_call
#endif

struct domain_iommu {
+ #ifdef CONFIG_HAS_PASSTHROUGH
    struct arch_iommu arch;
+ #endif
...

I'll do that in the next iteration of this patch series.

Thanks.

~ Oleksii

Re: [PATCH v2 11/39] xen/riscv: introduce asm/nospec.h

[Oleksii]

On Thu, 2023-12-07 at 15:28 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > --- /dev/null
> > +++ b/xen/arch/riscv/include/asm/nospec.h
> > @@ -0,0 +1,25 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights
> > Reserved. */
> > +
> > +#ifndef _ASM_RISCV_NOSPEC_H
> > +#define _ASM_RISCV_NOSPEC_H
> > +
> > +static inline bool evaluate_nospec(bool condition)
> > +{
> > +    return condition;
> > +}
> > +
> > +static inline void block_speculation(void)
> > +{
> > +}
> > +
> > +#endif /* _ASM_RISCV_NOSPEC_H */
> 
> This being identical between Arm, PPC, and now RISC-V, wouldn't this
> be another
> asm-generic/ candidate? (Whether such trivial stubs are copyrightable
> is, as
> per earlier remarks, at least questionable to me.)
It is a good candidate to be moved to asm-generic. Thanks for notice
that. I'll update move this patch to generic headers patch series.

I am not sure too but the copyright was presented in Arm's file and
RISC-V's version is just a copy so I decided to leave it.
Does any documented rule exist in which cases copyright should be and
not?

~ Oleksii

Re: [PATCH v2 13/39] xen/riscv: introduce asm/system.h

[Oleksii]

On Thu, 2023-12-07 at 16:07 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > --- /dev/null
> > +++ b/xen/arch/riscv/include/asm/system.h
> > @@ -0,0 +1,79 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +
> > +#ifndef _ASM_RISCV_BARRIER_H
> > +#define _ASM_RISCV_BARRIER_H
> > +
> > +#include <asm/csr.h>
> > +
> > +#ifndef __ASSEMBLY__
> > +
> > +#define RISCV_FENCE(p, s) \
> > +    __asm__ __volatile__ ("fence " #p "," #s : : : "memory")
> 
> Nit (style): Missing blanks immediately inside the parentheses.
Thanks for the comment. I'll update code style.

> 
> > +/* These barriers need to enforce ordering on both devices or
> > memory. */
> > +#define mb()                    RISCV_FENCE(iorw,iorw)
> > +#define rmb()                   RISCV_FENCE(ir,ir)
> > +#define wmb()                   RISCV_FENCE(ow,ow)
> 
> Nit (style): Missing blanks after the commas (also again below).
Thanks for the comment. I'll update code style.

> 
> > +/* These barriers do not need to enforce ordering on devices, just
> > memory. */
> > +#define smp_mb()                RISCV_FENCE(rw,rw)
> > +#define smp_rmb()               RISCV_FENCE(r,r)
> > +#define smp_wmb()               RISCV_FENCE(w,w)
> > +#define smp_mb__before_atomic() smp_mb()
> > +#define smp_mb__after_atomic()  smp_mb()
> > +
> > +/*
> > +#define __smp_store_release(p, v)       \
> 
> Is there a need for the double underscores here? We try to not
> introduce new instances of undue leading underscores, but there might
> be e.g. a strong desire to stay in sync with, say, Linux.
I don't have such a strong desire to be in sync with Linux so let's
stick to Xen code style. I'll update this place in next patch version.

> 
> > +do {                                    \
> > +	compiletime_assert_atomic_type(*p); \
> > +	RISCV_FENCE(rw,w);                  \
> > +	WRITE_ONCE(*p, v);                  \
> 
> Nit: Can the trailing backslashes be aligned, please?
Sure. I'll aligned them. Thanks.
> 
> > +} while (0)
> > +
> > +#define __smp_load_acquire(p)           \
> > +({                                      \
> > +    typeof(*p) ___p1 = READ_ONCE(*p);   \
> 
> Hmm, yet more leading underscores, and here surely not needed.
I'll update the code according to your recommendation. Thanks.

> 
> > +    compiletime_assert_atomic_type(*p); \
> > +    RISCV_FENCE(r,rw);                  \
> > +    ___p1;                              \
> > +})
> > +*/
> > +
> > +static inline unsigned long local_save_flags(void)
> > +{
> > +    return csr_read(sstatus);
> > +}
> > +
> > +static inline void local_irq_enable(void)
> > +{
> > +    csr_set(sstatus, SSTATUS_SIE);
> > +}
> > +
> > +static inline void local_irq_disable(void)
> > +{
> > +    csr_clear(sstatus, SSTATUS_SIE);
> > +}
> > +
> > +#define local_irq_save(x)                           \
> > +({                                                  \
> > +    x = csr_read_clear(CSR_SSTATUS, SSTATUS_SIE);   \
> > +    local_irq_disable();                            \
> > +})
> > +
> > +static inline void local_irq_restore(unsigned long flags)
> > +{
> > +	csr_set(CSR_SSTATUS, flags & SSTATUS_SIE);
> > +}
> > +
> > +static inline int local_irq_is_enabled(void)
> > +{
> > +    unsigned long flags = local_save_flags();
> > +
> > +    return flags & SSTATUS_SIE;
> 
> SSTATUS_SIE doesn't even happen to be 1, so I think you're better off
> adding != 0, unless you would do as I think I had suggested before
> and
> have the function return bool right away.
It makes sense. I'll apply your recommendations in the next patch
version.

~ Oleksii

Re: [PATCH v2 14/39] xen/riscv: introduce bitops.h

[Oleksii]

On Thu, 2023-12-07 at 16:37 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> 
> So this looks to have been taken from Linux, which could do with
> saying
> (including which version or most recent commit). It may e.g. justify
> you
> using tab indentation here, albeit ...
Thanks. I'll update the commit message.

> 
> > --- /dev/null
> > +++ b/xen/arch/riscv/include/asm/bitops.h
> > @@ -0,0 +1,288 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/* Copyright (C) 2012 Regents of the University of California */
> > +
> > +#ifndef _ASM_RISCV_BITOPS_H
> > +#define _ASM_RISCV_BITOPS_H
> > +
> > +#include <asm/system.h>
> > +
> > +#define BITOP_BITS_PER_WORD     32
> > +#define BITOP_MASK(nr)	        (1UL << ((nr) %
> > BITOP_BITS_PER_WORD))
> > +#define BITOP_WORD(nr)	        ((nr) / BITOP_BITS_PER_WORD)
> > +#define BITS_PER_BYTE	        8
> > +
> > +#define __set_bit(n,p)          set_bit(n,p)
> > +#define __clear_bit(n,p)        clear_bit(n,p)
> 
> ... then please consistently. Other style related remarks made on the
> system.h patch apply here as well (unless again there's a goal of
> keeping the diff to the Linux original small; yet then I guess the
> delta to the Linux file is already pretty large).
> 
> > +/* Based on linux/include/asm-generic/bitops/find.h */
> > +
> > +#ifndef find_next_bit
> > +/**
> > + * find_next_bit - find the next set bit in a memory region
> > + * @addr: The address to base the search on
> > + * @offset: The bitnumber to start searching at
> > + * @size: The bitmap size in bits
> > + */
> > +extern unsigned long find_next_bit(const unsigned long *addr,
> > unsigned long
> > +		size, unsigned long offset);
> > +#endif
> > +
> > +#ifndef find_next_zero_bit
> > +/**
> > + * find_next_zero_bit - find the next cleared bit in a memory
> > region
> > + * @addr: The address to base the search on
> > + * @offset: The bitnumber to start searching at
> > + * @size: The bitmap size in bits
> > + */
> > +extern unsigned long find_next_zero_bit(const unsigned long *addr,
> > unsigned
> > +		long size, unsigned long offset);
> > +#endif
> > +
> > +/**
> > + * find_first_bit - find the first set bit in a memory region
> > + * @addr: The address to start the search at
> > + * @size: The maximum size to search
> > + *
> > + * Returns the bit number of the first set bit.
> > + */
> > +extern unsigned long find_first_bit(const unsigned long *addr,
> > +				    unsigned long size);
> > +
> > +/**
> > + * find_first_zero_bit - find the first cleared bit in a memory
> > region
> > + * @addr: The address to start the search at
> > + * @size: The maximum size to search
> > + *
> > + * Returns the bit number of the first cleared bit.
> > + */
> > +extern unsigned long find_first_zero_bit(const unsigned long
> > *addr,
> > +					 unsigned long size);
> 
> Looking over the titles of the rest of the series, I can't spot where
> these are going to be implemented. The again maybe you indeed can get
> away without those, at least initially.
It's introduced in:
	[PATCH v2 21/39] xen/riscv: introduce bit operations
I think we have to merge this patch with patch 21.
> 
> > +#define ffs(x) ({ unsigned int __t = (x); fls(__t & -__t); })
> 
> This wants to use ISOLATE_LSB() now.
> 
~ Oleksii

Re: [PATCH v2 16/39] xen/riscv: introduce asm/smp.h

[Oleksii]

On Thu, 2023-12-07 at 16:43 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > --- /dev/null
> > +++ b/xen/arch/riscv/include/asm/smp.h
> > @@ -0,0 +1,23 @@
> > +#ifndef __ASM_RISCV_SMP_H
> > +#define __ASM_RISCV_SMP_H
> > +
> > +#ifndef __ASSEMBLY__
> > +#include <xen/cpumask.h>
> > +#include <xen/percpu.h>
> > +#endif
> 
> If you want this to be possible to include from assembly files (I
> don't
> know why you would want that), ...
> 
> > +DECLARE_PER_CPU(cpumask_var_t, cpu_sibling_mask);
> > +DECLARE_PER_CPU(cpumask_var_t, cpu_core_mask);
> 
> ... these two would also need to live inside the #ifdef. Otherwise
> the
> #ifdef wants dropping.
I think we can just drop $ifdef. I am not sure that this header will be
used in assembly code.

> 
> > +#define cpu_is_offline(cpu) unlikely(!cpu_online(cpu))
> 
> Seeing this is now the 4th instance, I guess we want to move it to
> xen/smp.h. I'll try to remember making a patch.
It will be nice.

Thanks.

> 
> > +/*
> > + * Do we, for platform reasons, need to actually keep CPUs online
> > when we
> > + * would otherwise prefer them to be off?
> > + */
> > +#define park_offline_cpus false
> > +
> > +/* TODO: need to be implemeted */
> > +#define smp_processor_id() (0)
> > +
> > +#endif
> > \ No newline at end of file
> 
> You want to take care of this.
Sure. I'll add a newline. Thanks for noticing that.

> 
~ Oleksii

Re: [PATCH v2 00/39] Enable build of full Xen for RISC-V

[Oleksii]

On Thu, 2023-12-07 at 15:30 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > Bobby Eshleman (1):
> >   xen/riscv: introduce asm/atomic.h
> > 
> > Oleksii Kurochko (38):
> >   xen/riscv: disable unnecessary configs
> >   xen/riscv: use some asm-generic headers
> >   xen/riscv:introduce asm/byteorder.h
> >   xen/riscv: add public arch-riscv.h
> >   xen/riscv: introduce spinlock.h
> >   xen/riscv: introduce fence.h
> >   xen/riscv: introduce arch-riscv/hvm/save.h
> >   xen/riscv: introduce asm/cpufeature.h
> >   xen/riscv: introduce asm/guest_atomics.h
> >   xen/riscv: introduce asm/iommu.h
> >   xen/riscv: introduce asm/nospec.h
> >   xen/riscv: introduce asm/setup.h
> >   xen/riscv: introduce asm/system.h
> >   xen/riscv: introduce bitops.h
> >   xen/riscv: introduce flushtlb.h
> >   xen/riscv: introduce asm/smp.h
> >   xen/riscv: introduce cmpxchg.h
> >   xen/riscv: introduce asm/io.h
> >   xen/riscv: define bug frame tables in xen.lds.S
> >   xen/riscv: introduce bit operations
> >   xen/riscv: introduce asm/domain.h
> >   xen/riscv: introduce asm/guest_access.h
> >   xen/riscv: introduce asm/irq.h
> >   xen/riscv: introduce asm/p2m.h
> >   xen/riscv: introduce asm/regs.h
> >   xen/riscv: introduce asm/time.h
> >   xen/riscv: introduce asm/event.h
> 
> Throughout here, would you please try to be consistent about (not)
> using asm/
> prefixes?
Sure. I'll skip asm, there is no any sense to use it as it is mentioned
xen/riscv...

~ Oleksii

Re: [PATCH v2 10/39] xen/riscv: introduce asm/iommu.h

[Jan Beulich]

On 08.12.2023 10:29, Oleksii wrote:
> On Thu, 2023-12-07 at 15:22 +0100, Jan Beulich wrote:
>> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>>> --- /dev/null
>>> +++ b/xen/arch/riscv/include/asm/iommu.h
>>> @@ -0,0 +1,7 @@
>>> +#ifndef __ASM_RISCV_IOMMU_H__
>>> +#define __ASM_RISCV_IOMMU_H__
>>> +
>>> +struct arch_iommu {
>>> +};
>>> +
>>> +#endif /* __ASM_IOMMU_H__ */
>>
>> Instead of adding this header, didn't we discuss to make the #include
>> in
>> xen/iommu.h depend on CONFIG_HAS_PASSTHROUGH? Also - no SPDX or
>> footer
>> here?
> We had discussion about some stuff in device.h, but we can apply it
> here too. Only to place will be needed to update:
> 
> 
> + #ifdef CONFIG_HAS_PASSTHROUGH
> #include <asm/iommu.h>
> + #endif
> 
> #ifndef iommu_call
> # define iommu_call(ops, fn, args...) ((ops)->fn(args))
> # define iommu_vcall iommu_call
> #endif
> 
> struct domain_iommu {
> + #ifdef CONFIG_HAS_PASSTHROUGH
>     struct arch_iommu arch;
> + #endif
> ...
> 
> I'll do that in the next iteration of this patch series.

What you suggest is to eliminate the need for an asm/ header. I asked
about an asm-generic/ one, though. (But I'd be okay either way, as long
as the redundancy is avoided.)

Jan

Re: [PATCH v2 11/39] xen/riscv: introduce asm/nospec.h

[Jan Beulich]

On 08.12.2023 10:33, Oleksii wrote:
> On Thu, 2023-12-07 at 15:28 +0100, Jan Beulich wrote:
>> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>>> --- /dev/null
>>> +++ b/xen/arch/riscv/include/asm/nospec.h
>>> @@ -0,0 +1,25 @@
>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>> +/* Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights
>>> Reserved. */
>>> +
>>> +#ifndef _ASM_RISCV_NOSPEC_H
>>> +#define _ASM_RISCV_NOSPEC_H
>>> +
>>> +static inline bool evaluate_nospec(bool condition)
>>> +{
>>> +    return condition;
>>> +}
>>> +
>>> +static inline void block_speculation(void)
>>> +{
>>> +}
>>> +
>>> +#endif /* _ASM_RISCV_NOSPEC_H */
>>
>> This being identical between Arm, PPC, and now RISC-V, wouldn't this
>> be another
>> asm-generic/ candidate? (Whether such trivial stubs are copyrightable
>> is, as
>> per earlier remarks, at least questionable to me.)
> It is a good candidate to be moved to asm-generic. Thanks for notice
> that. I'll update move this patch to generic headers patch series.
> 
> I am not sure too but the copyright was presented in Arm's file and
> RISC-V's version is just a copy so I decided to leave it.
> Does any documented rule exist in which cases copyright should be and
> not?

I'm afraid there's nothing. But see how PPC has dropped the copyright,
too. Generally I'm of the opinion that purely trivial stubs aren't
sensible to put under a copyright.

Jan

Re: [PATCH v2 18/39] xen/riscv: introduce cmpxchg.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> ---
> Changes in V2:
> 	- update the comment at the top of the header.
> 	- change xen/lib.h to xen/bug.h.
> 	- sort inclusion of headers properly.
> ---
>  xen/arch/riscv/include/asm/cmpxchg.h | 379 +++++++++++++++++++++++++++
>  1 file changed, 379 insertions(+)
>  create mode 100644 xen/arch/riscv/include/asm/cmpxchg.h
> 
> diff --git a/xen/arch/riscv/include/asm/cmpxchg.h b/xen/arch/riscv/include/asm/cmpxchg.h
> new file mode 100644
> index 0000000000..c535bc9660
> --- /dev/null
> +++ b/xen/arch/riscv/include/asm/cmpxchg.h
> @@ -0,0 +1,379 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + *  Taken and modified from Linux.

Depending on how heavily this file was modified, further adjustments may or
may not be advisable. Can the description here please be non-empty to cover
how close to the original the result is, plus also what version or commit
you started from?

Jan

Re: [PATCH v2 19/39] xen/riscv: introduce asm/io.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> ---
> Changes in V2:
>  - Nothing changed. Only rebase.
> ---
>  xen/arch/riscv/include/asm/io.h | 134 ++++++++++++++++++++++++++++++++
>  1 file changed, 134 insertions(+)
>  create mode 100644 xen/arch/riscv/include/asm/io.h
> 
> diff --git a/xen/arch/riscv/include/asm/io.h b/xen/arch/riscv/include/asm/io.h
> new file mode 100644
> index 0000000000..987fddf902
> --- /dev/null
> +++ b/xen/arch/riscv/include/asm/io.h
> @@ -0,0 +1,134 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Taken and modified from Linux.

Somewhat similar the the previous patch, further style related adjustments
may or may not wants making depending on how close to the original the
result is. Same request therefore here as just mentioned there. One further
question though:

> + * {read,write}{b,w,l,q} based on arch/arm64/include/asm/io.h
> + *   which was based on arch/arm/include/io.h
> + *
> + * Copyright (C) 1996-2000 Russell King
> + * Copyright (C) 2012 ARM Ltd.
> + * Copyright (C) 2014 Regents of the University of California
> + */
> +
> +#ifndef _ASM_RISCV_IO_H
> +#define _ASM_RISCV_IO_H
> +
> +#include <asm/byteorder.h>
> +
> +/*
> + * The RISC-V ISA doesn't yet specify how to query or modify PMAs, so we can't
> + * change the properties of memory regions.  This should be fixed by the
> + * upcoming platform spec.
> + */
> +#define ioremap_nocache(addr, size) ioremap((addr), (size))
> +#define ioremap_wc(addr, size) ioremap((addr), (size))
> +#define ioremap_wt(addr, size) ioremap((addr), (size))
> +
> +/* Generic IO read/write.  These perform native-endian accesses. */
> +#define __raw_writeb __raw_writeb
> +static inline void __raw_writeb(u8 val, volatile void __iomem *addr)
> +{
> +	asm volatile("sb %0, 0(%1)" : : "r" (val), "r" (addr));
> +}
> +
> +#define __raw_writew __raw_writew
> +static inline void __raw_writew(u16 val, volatile void __iomem *addr)
> +{
> +	asm volatile("sh %0, 0(%1)" : : "r" (val), "r" (addr));
> +}
> +
> +#define __raw_writel __raw_writel
> +static inline void __raw_writel(u32 val, volatile void __iomem *addr)
> +{
> +	asm volatile("sw %0, 0(%1)" : : "r" (val), "r" (addr));
> +}
> +
> +#ifdef CONFIG_64BIT
> +#define __raw_writeq __raw_writeq
> +static inline void __raw_writeq(u64 val, volatile void __iomem *addr)
> +{
> +	asm volatile("sd %0, 0(%1)" : : "r" (val), "r" (addr));
> +}
> +#endif
> +
> +#define __raw_readb __raw_readb
> +static inline u8 __raw_readb(const volatile void __iomem *addr)
> +{
> +	u8 val;
> +
> +	asm volatile("lb %0, 0(%1)" : "=r" (val) : "r" (addr));
> +	return val;
> +}
> +
> +#define __raw_readw __raw_readw
> +static inline u16 __raw_readw(const volatile void __iomem *addr)
> +{
> +	u16 val;
> +
> +	asm volatile("lh %0, 0(%1)" : "=r" (val) : "r" (addr));
> +	return val;
> +}
> +
> +#define __raw_readl __raw_readl
> +static inline u32 __raw_readl(const volatile void __iomem *addr)
> +{
> +	u32 val;
> +
> +	asm volatile("lw %0, 0(%1)" : "=r" (val) : "r" (addr));
> +	return val;
> +}
> +
> +#ifdef CONFIG_64BIT
> +#define __raw_readq __raw_readq
> +static inline u64 __raw_readq(const volatile void __iomem *addr)
> +{
> +	u64 val;
> +
> +	asm volatile("ld %0, 0(%1)" : "=r" (val) : "r" (addr));
> +	return val;
> +}
> +#endif

If all of these are plain loads and stores, is it really necessary to use
inline assembly in the first place? Even more so that you don't properly
make the compiler aware of which range of memory you access.

Jan

Re: [PATCH v2 20/39] xen/riscv: define bug frame tables in xen.lds.S

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>

Acked-by: Jan Beulich <jbeulich@suse.com>

Re: [PATCH v2 29/39] xen/riscv: add definition of __read_mostly

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> The definition of __read_mostly should be removed in:
> https://lore.kernel.org/xen-devel/f25eb5c9-7c14-6e23-8535-2c66772b333e@suse.com/

Andrew, can we settle on what to do with that patch? If you don't like me
putting __read_mostly in xen/cache.h (consistent with __ro_after_init),
would you please make an alternative suggestion? Personally I don't really
understand why that patch hasn't long gone in. If further reorg is wanted,
it can always be done subsequently. In whatever adjustments to the patch
you want me to make to get past your objection, please make sure that it
doesn't end up scope creeping.

Jan

> The patch introduces it in arch-specific header to not
> block enabling of full Xen build for RISC-V.
> 
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> ---
>  - update the commit message
> ---
>  xen/arch/riscv/include/asm/cache.h | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/xen/arch/riscv/include/asm/cache.h b/xen/arch/riscv/include/asm/cache.h
> index 69573eb051..94bd94db53 100644
> --- a/xen/arch/riscv/include/asm/cache.h
> +++ b/xen/arch/riscv/include/asm/cache.h
> @@ -3,4 +3,6 @@
>  #ifndef _ASM_RISCV_CACHE_H
>  #define _ASM_RISCV_CACHE_H
>  
> +#define __read_mostly __section(".data.read_mostly")
> +
>  #endif /* _ASM_RISCV_CACHE_H */

Re: [PATCH v2 18/39] xen/riscv: introduce cmpxchg.h

[Oleksii]

On Tue, 2023-12-12 at 17:51 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> > ---
> > Changes in V2:
> > 	- update the comment at the top of the header.
> > 	- change xen/lib.h to xen/bug.h.
> > 	- sort inclusion of headers properly.
> > ---
> >  xen/arch/riscv/include/asm/cmpxchg.h | 379
> > +++++++++++++++++++++++++++
> >  1 file changed, 379 insertions(+)
> >  create mode 100644 xen/arch/riscv/include/asm/cmpxchg.h
> > 
> > diff --git a/xen/arch/riscv/include/asm/cmpxchg.h
> > b/xen/arch/riscv/include/asm/cmpxchg.h
> > new file mode 100644
> > index 0000000000..c535bc9660
> > --- /dev/null
> > +++ b/xen/arch/riscv/include/asm/cmpxchg.h
> > @@ -0,0 +1,379 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + *  Taken and modified from Linux.
> 
> Depending on how heavily this file was modified, further adjustments
> may or
> may not be advisable. Can the description here please be non-empty to
> cover
> how close to the original the result is, plus also what version or
> commit
> you started from?
Sure. I'll add but it was only minor changes:

This header isn't changed to much in Linux kernel so this one version
can be used as a base:
https://elixir.bootlin.com/linux/latest/source/arch/riscv/include/asm/cmpxchg.h#L310

The following was changed:
1. 	default: \
		BUILD_BUG();
  BUILD_BUG -> ASSERT_UNREACHABLE
2. Remove prefixes arch_ before some macros.

But it will be changed more as Xen uses xchg and cmpxchg with 1 and 2
bytes types, so it should be handled...

Anyway, I'll be happy with your comments, and I'll apply them in the
next patch version.

~ Oleksii

Re: [PATCH v2 04/39] xen/riscv: add public arch-riscv.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>

If this is enough for the time being, so be it:
Acked-by: Jan Beulich <jbeulich@suse.com>

Jan

Re: [PATCH v2 21/39] xen/riscv: introduce bit operations

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>

Did you write this code from scratch? If not, you need to at least point
out the origin. But: None of this looks RISC-V specific, so shouldn't it
instead be put under xen/lib/, as a fallback implementation for arch-es
not having optimized implementations? Looks also at least very similar
to Arm64's, so that one perhaps then will want dropping as well? We
surely don't want to carry two (almost) identical copies of the same
logic.

> --- /dev/null
> +++ b/xen/arch/riscv/lib/Makefile
> @@ -0,0 +1 @@
> +obj-y += find_next_bit.o

Dashes instead of underscores please in new file's names, whenever
possible.

Jan

Re: [PATCH v2 22/39] xen/riscv: introduce asm/domain.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>

Acked-by: Jan Beulich <jbeulich@suse.com>

Re: [PATCH v2 23/39] xen/riscv: introduce asm/guest_access.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> --- /dev/null
> +++ b/xen/arch/riscv/include/asm/guest_access.h
> @@ -0,0 +1,29 @@
> +#ifndef __ASM_RISCV_GUEST_ACCESS_H__
> +#define __ASM_RISCV_GUEST_ACCESS_H__
> +
> +#include <xen/types.h>
> +
> +unsigned long raw_copy_to_guest(void *to, const void *from, unsigned len);
> +unsigned long raw_copy_from_guest(void *to, const void *from, unsigned len);

For the full build to work, you will need to implement these somewhere.
From the titles of further patches I can't guess where that would be.
Note how PPC has decided to have these be inline dummies for the time
being. It also looks as if you would also need raw_clear_guest()?

> +#define __raw_copy_to_guest raw_copy_to_guest
> +#define __raw_copy_from_guest raw_copy_from_guest
> +
> +#define guest_handle_okay(hnd, nr) (1)
> +#define guest_handle_subrange_okay(hnd, first, last) (1)

Arm has a comment next to these (actually two, I mean the latter), which
I think would be sensible to also have here.

> +struct domain;
> +unsigned long copy_to_guest_phys(struct domain *d,
> +                                 paddr_t gpa,
> +                                 void *buf,
> +                                 unsigned int len);

You don't need this just yet, do you?

Jan

Re: [PATCH v2 24/39] xen/riscv: introduce asm/irq.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> ---
> Changes in V2:
> 	- add ifdef CONFIG_HAS_DEVICE_TREE for things that shouldn't be
>       in case !CONFIG_HAS_DEVICE_TREE

Is there going to be a RISC-V build without this enabled (selected)? If
not, I'd recommend against such pointless #ifdef-ary.

Jan

Re: [PATCH v2 25/39] xen/riscv: introduce asm/p2m.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> --- /dev/null
> +++ b/xen/arch/riscv/include/asm/p2m.h
> @@ -0,0 +1,105 @@
> +#ifndef __ASM_RISCV_P2M_H__
> +#define __ASM_RISCV_P2M_H__
> +
> +#include <asm/page-bits.h>
> +
> +#define paddr_bits PADDR_BITS
> +
> +/*
> + * List of possible type for each page in the p2m entry.
> + * The number of available bit per page in the pte for this purpose is 4 bits.
> + * So it's possible to only have 16 fields. If we run out of value in the
> + * future, it's possible to use higher value for pseudo-type and don't store
> + * them in the p2m entry.
> + */
> +typedef enum {
> +    p2m_invalid = 0,    /* Nothing mapped here */
> +    p2m_ram_rw,         /* Normal read/write guest RAM */
> +    p2m_ram_ro,         /* Read-only; writes are silently dropped */
> +    p2m_mmio_direct_dev,/* Read/write mapping of genuine Device MMIO area */
> +    p2m_mmio_direct_nc, /* Read/write mapping of genuine MMIO area non-cacheable */
> +    p2m_mmio_direct_c,  /* Read/write mapping of genuine MMIO area cacheable */
> +    p2m_map_foreign_rw, /* Read/write RAM pages from foreign domain */
> +    p2m_map_foreign_ro, /* Read-only RAM pages from foreign domain */
> +    p2m_grant_map_rw,   /* Read/write grant mapping */
> +    p2m_grant_map_ro,   /* Read-only grant mapping */
> +    /* The types below are only used to decide the page attribute in the P2M */
> +    p2m_iommu_map_rw,   /* Read/write iommu mapping */
> +    p2m_iommu_map_ro,   /* Read-only iommu mapping */
> +    p2m_max_real_type,  /* Types after this won't be store in the p2m */
> +} p2m_type_t;

As indicated before, I think you should have only those types here which
you're sure you'll need, or even just those which you really need right
away. I question in particular p2m_mmio_direct_*, which all look like
you simply took them from Arm, without regard as to applicability to
RISC-V. The fewer types you have here, the easier it is going to be to
tell what needs adding and what is already _properly_ supported.

> +#include <xen/p2m-common.h>
> +
> +static inline int get_page_and_type(struct page_info *page,
> +                                    struct domain *domain,
> +                                    unsigned long type)
> +{
> +    BUG();
> +    return 1;
> +}

Imo despite the BUG() any such stub would better return failure, just
like ...

> +/* Look up a GFN and take a reference count on the backing page. */
> +typedef unsigned int p2m_query_t;
> +#define P2M_ALLOC    (1u<<0)   /* Populate PoD and paged-out entries */
> +#define P2M_UNSHARE  (1u<<1)   /* Break CoW sharing */
> +
> +static inline struct page_info *get_page_from_gfn(
> +    struct domain *d, unsigned long gfn, p2m_type_t *t, p2m_query_t q)
> +{
> +    BUG();
> +    return NULL;
> +}

... you do here. May apply again further down.

> +static inline void memory_type_changed(struct domain *d)
> +{
> +    BUG();
> +}
> +
> +
> +static inline int guest_physmap_mark_populate_on_demand(struct domain *d, unsigned long gfn,
> +                                                        unsigned int order)
> +{
> +    BUG();
> +    return 1;
> +}
> +
> +static inline int guest_physmap_add_entry(struct domain *d,
> +                            gfn_t gfn,
> +                            mfn_t mfn,
> +                            unsigned long page_order,
> +                            p2m_type_t t)
> +{
> +    BUG();
> +    return 1;
> +}
> +
> +/* Untyped version for RAM only, for compatibility */
> +static inline int __must_check
> +guest_physmap_add_page(struct domain *d, gfn_t gfn, mfn_t mfn,
> +                       unsigned int page_order)
> +{
> +    return guest_physmap_add_entry(d, gfn, mfn, page_order, p2m_ram_rw);
> +}
> +
> +static inline mfn_t gfn_to_mfn(struct domain *d, gfn_t gfn)
> +{
> +    BUG();
> +    return _mfn(0);
> +}
> +
> +static inline bool arch_acquire_resource_check(struct domain *d)
> +{
> +    /*
> +     * The reference counting of foreign entries in set_foreign_p2m_entry()
> +     * is supported on RISCV.
> +     */
> +    return true;
> +}
> +
> +static inline void p2m_altp2m_check(struct vcpu *v, uint16_t idx)
> +{
> +    /* Not supported on RISCV. */
> +}
> +
> +#endif /* __ASM_RISCV_P2M_H__ */
> \ No newline at end of file

This wants taking care of.

Jan

Re: [PATCH v2 25/39] xen/riscv: introduce asm/p2m.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> +static inline int guest_physmap_mark_populate_on_demand(struct domain *d, unsigned long gfn,
> +                                                        unsigned int order)
> +{
> +    BUG();
> +    return 1;
> +}

This one I actually don't think needs to be a stub. It can return
-EOPNOTSUPP or -EINVAL right away, and then wouldn't need touching
again afaict.

Jan

Re: [PATCH v2 26/39] xen/riscv: introduce asm/regs.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> --- /dev/null
> +++ b/xen/arch/riscv/include/asm/regs.h
> @@ -0,0 +1,26 @@
> +#ifndef __ARM_RISCV_REGS_H__
> +#define __ARM_RISCV_REGS_H__
> +
> +#ifndef __ASSEMBLY__
> +
> +#include <xen/bug.h>
> +#include <asm/current.h>

Does one of these bring in asm/processor.h, for ...

> +#define hyp_mode(r)     (0)
> +
> +static inline bool guest_mode(const struct cpu_user_regs *r)

... struct cpu_user_regs to be available? Else a forward declaration
would be needed here.

Acked-by: Jan Beulich <jbeulich@suse.com>

Jan

Re: [PATCH v2 27/39] xen/riscv: introduce asm/time.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> --- /dev/null
> +++ b/xen/arch/riscv/include/asm/time.h
> @@ -0,0 +1,19 @@
> +#ifndef __ASM_RISCV_TIME_H__
> +#define __ASM_RISCV_TIME_H__
> +
> +#include <xen/bug.h>
> +#include <asm/csr.h>
> +
> +struct vcpu;
> +
> +/* TODO: implement */
> +static inline void force_update_vcpu_system_time(struct vcpu *v) { BUG(); }
> +
> +typedef unsigned long cycles_t;
> +
> +static inline cycles_t get_cycles(void)
> +{
> +	return csr_read(CSR_TIME);
> +}
> +
> +#endif /* __ASM_RISCV_TIME_H__ */
> \ No newline at end of file

With this taken care of (and the SPDX header added)
Acked-by: Jan Beulich <jbeulich@suse.com>

Jan

Re: [PATCH v2 28/39] xen/riscv: introduce asm/event.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> --- /dev/null
> +++ b/xen/arch/riscv/include/asm/event.h
> @@ -0,0 +1,34 @@
> +#ifndef __ASM_RISCV_EVENT_H__
> +#define __ASM_RISCV_EVENT_H__
> +
> +void vcpu_mark_events_pending(struct vcpu *v);
> +
> +static inline int vcpu_event_delivery_is_enabled(struct vcpu *v)
> +{
> +    return 0;
> +}
> +
> +static inline int local_events_need_delivery(void)
> +{
> +    return 0;
> +}
> +
> +static inline void local_event_delivery_enable(void)
> +{
> +}

These are stubs aiui, and hence would better have BUG() in them?

> +/* No arch specific virq definition now. Default to global. */
> +static inline bool arch_virq_is_global(unsigned int virq)
> +{
> +    return true;
> +}

This otoh may well remain as is for the foreseeable future.

Jan

Re: [PATCH v2 30/39] xen/riscv: define an address of frame table

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Also the patchs adds some helpful macros.

In how far they're (going to be) helpful is hard to tell without uses
and without some suitable comments.

> --- a/xen/arch/riscv/include/asm/config.h
> +++ b/xen/arch/riscv/include/asm/config.h
> @@ -77,12 +77,31 @@
>    name:
>  #endif
>  
> +#define VPN_BITS    (9)
> +#define OFFSET_BITS (12)

Whose offset? In how far is this different from PAGE_SHIFT?

>  #ifdef CONFIG_RISCV_64
> +
> +#define SLOTN_ENTRY_BITS        (HYP_PT_ROOT_LEVEL * VPN_BITS + OFFSET_BITS)
> +#define SLOTN(slot)             (_AT(vaddr_t,slot) << SLOTN_ENTRY_BITS)

Nit: Missing blank after comma.

> +#define SLOTN_ENTRY_SIZE        SLOTN(1)
> +
>  #define XEN_VIRT_START 0xFFFFFFFFC0000000 /* (_AC(-1, UL) + 1 - GB(1)) */
> +
> +#define FRAMETABLE_VIRT_START   SLOTN(196)
> +#define FRAMETABLE_SIZE         GB(3)
> +#define FRAMETABLE_NR           (FRAMETABLE_SIZE / sizeof(*frame_table))
> +#define FRAMETABLE_VIRT_END     (FRAMETABLE_VIRT_START + FRAMETABLE_SIZE - 1)
> +
> +#define VMAP_VIRT_START         SLOTN(194)
> +#define VMAP_VIRT_SIZE          GB(1)

May I suggest that you keep these blocks sorted by slot number? Or wait,
the layout comment further up is also in decreasing order, so that's
fine here, but then can all of this please be moved next to the comment
actually providing the necessary context (thus eliminating the need for
new comments)? You'll then also notice that the generalization here
(keeping basically the same layout for e.g. SATP_MODE_SV48, just shifted
by 9 bits) isn't in line with the comment there.

> @@ -95,6 +114,8 @@
>  #define RV_STAGE1_MODE SATP_MODE_SV32
>  #endif
>  
> +#define HYP_PT_ROOT_LEVEL (CONFIG_PAGING_LEVELS - 1)

I understand that CONFIG_PAGING_LEVELS is defined only just up from here,
but what that identifier stands for is quite clear. It would seem to me
that moving this up ahead if its first use would help clarity.

Jan

Re: [PATCH v2 31/39] xen/riscv: add required things to asm/current.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> --- a/xen/arch/riscv/include/asm/current.h
> +++ b/xen/arch/riscv/include/asm/current.h
> @@ -3,6 +3,22 @@
>  #ifndef __ASM_CURRENT_H
>  #define __ASM_CURRENT_H
>  
> +#include <xen/percpu.h>
> +#include <asm/processor.h>
> +
> +#ifndef __ASSEMBLY__
> +
> +struct vcpu;

I don't think you need this here?

> +/* Which VCPU is "current" on this PCPU. */
> +DECLARE_PER_CPU(struct vcpu *, curr_vcpu);
> +
> +#define current            (this_cpu(curr_vcpu))

Nit: No need for the extra parentheses.

> +#define set_current(vcpu)  do { current = (vcpu); } while (0)
> +#define get_cpu_current(cpu)  (per_cpu(curr_vcpu, cpu))

Same here then.

> +#define guest_cpu_user_regs() (0)

0 or NULL? Also perhaps better 

#define guest_cpu_user_regs() ({ BUG(); NULL; })

until it's properly implemented?

> @@ -10,4 +26,8 @@
>      unreachable();                                          \
>  } while ( false )
>  
> +#define get_per_cpu_offset() __per_cpu_offset[get_processor_id()]

I'd like to remind you that there's no get_processor_id() anymore.

Jan

Re: [PATCH v2 32/39] xen/riscv: add minimal stuff to asm/page.h to build full Xen

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>

Acked-by: Jan Beulich <jbeulich@suse.com>

I wonder though ...

> --- a/xen/arch/riscv/include/asm/page.h
> +++ b/xen/arch/riscv/include/asm/page.h
> @@ -6,6 +6,7 @@
>  #ifndef __ASSEMBLY__
>  
>  #include <xen/const.h>
> +#include <xen/bug.h>
>  #include <xen/types.h>
>  
>  #include <asm/mm.h>
> @@ -32,6 +33,9 @@
>  #define PTE_LEAF_DEFAULT            (PTE_VALID | PTE_READABLE | PTE_WRITABLE)
>  #define PTE_TABLE                   (PTE_VALID)
>  
> +/* TODO */
> +#define PAGE_HYPERVISOR 0

... whether this couldn't be defined properly right away.

Jan

Re: [PATCH v2 33/39] xen/riscv: add minimal stuff to asm/processor.h to build full Xen

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> --- a/xen/arch/riscv/include/asm/processor.h
> +++ b/xen/arch/riscv/include/asm/processor.h
> @@ -12,6 +12,9 @@
>  
>  #ifndef __ASSEMBLY__
>  
> +/* TODO: need to be implemeted */
> +#define get_processor_id() 0

Please don't re-introduce this - it was just recently dropped from the
code base.

> @@ -53,6 +56,18 @@ struct cpu_user_regs
>      unsigned long pregs;
>  };
>  
> +/* TODO: need to implement */
> +#define cpu_to_core(_cpu)   (0)
> +#define cpu_to_socket(_cpu) (0)

No need for leading underscores here.

> +static inline void cpu_relax(void)
> +{
> +	int dummy;
> +	/* In lieu of a halt instruction, induce a long-latency stall. */
> +	__asm__ __volatile__ ("div %0, %0, zero" : "=r" (dummy));

Any reason for this, when Arm's is just barrier(), and apparently they got
away with this quite fine? Also isn't this causing a division by zero,
which I'd expect to cause some kind of exception? (Terminology-wise I'm of
course biased by x86, where "halt instruction" wouldn't be suitable to use
here. But if that terminology is fine on RISC-V, then obviously no
objection.)

Jan

Re: [PATCH v2 34/39] xen: add RISCV support for pmu.h

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>

Acked-by: Jan Beulich <jbeulich@suse.com>

I think though that this would make sense to fold into patch 4, which is
where the relevant (stub) structure appears.

Jan

> --- a/xen/include/public/pmu.h
> +++ b/xen/include/public/pmu.h
> @@ -13,6 +13,8 @@
>  #include "arch-arm.h"
>  #elif defined (__powerpc64__)
>  #include "arch-ppc.h"
> +#elif defined(__riscv)
> +#include "arch-riscv.h"
>  #else
>  #error "Unsupported architecture"
>  #endif

Re: [PATCH v2 35/39] xen: add necessary headers to common to build full Xen for RISC-V

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>

With an empty description it is hard to judge whether this is really needed.
I would sincerely hope we can get away without. Note how there already a few
struct xen_domctl_* forward declarations there, which - if the #include
really needs adding - should be dropped in exchange.

As a nit on the subject - it's only singular (now?) and could hence do with
being more precise on what the specific change is.

Jan

> --- a/xen/include/xen/domain.h
> +++ b/xen/include/xen/domain.h
> @@ -4,6 +4,7 @@
>  
>  #include <xen/types.h>
>  
> +#include <public/domctl.h>
>  #include <public/xen.h>
>  
>  struct guest_area {

Re: [PATCH v2 39/39] xen: fix compilation issue of serial.c

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> The following issue occurs on RISC-V platforms:
> drivers/char/serial.c: In function 'serial_tx_interrupt':
> drivers/char/serial.c:88:9: error: implicit declaration of function 'cpu_relax' [-Werror=implicit-function-declaration]
>    88 |         cpu_relax();
> 
> cpu_relax() is defined in <asm/processor.h> so it was added
> an inclusion of the header to serial.c.
> 
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>

Acked-by: Jan Beulich <jbeulich@suse.com>

I think this needs moving ahead at least by one patch in the series,
but I guess I'll put it in ahead of most of the rest anyway.

Jan

Re: [PATCH v2 39/39] xen: fix compilation issue of serial.c

[Oleksii]

On Thu, 2023-12-14 at 17:24 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > The following issue occurs on RISC-V platforms:
> > drivers/char/serial.c: In function 'serial_tx_interrupt':
> > drivers/char/serial.c:88:9: error: implicit declaration of function
> > 'cpu_relax' [-Werror=implicit-function-declaration]
> >    88 |         cpu_relax();
> > 
> > cpu_relax() is defined in <asm/processor.h> so it was added
> > an inclusion of the header to serial.c.
> > 
> > Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> 
> Acked-by: Jan Beulich <jbeulich@suse.com>
> 
> I think this needs moving ahead at least by one patch in the series,
> but I guess I'll put it in ahead of most of the rest anyway.
Thanks. I'll moved it.

~ Oleksii

Re: [PATCH v2 36/39] xen/riscv: add minimal stuff to asm/mm.h to build full Xen

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> --- a/xen/arch/riscv/include/asm/mm.h
> +++ b/xen/arch/riscv/include/asm/mm.h
> @@ -3,10 +3,271 @@
>  #ifndef _ASM_RISCV_MM_H
>  #define _ASM_RISCV_MM_H
>  
> +#include <public/xen.h>
> +#include <xen/pdx.h>
> +#include <xen/types.h>
> +
> +#include <asm/page.h>
>  #include <asm/page-bits.h>
>  
> -#define pfn_to_paddr(pfn) ((paddr_t)(pfn) << PAGE_SHIFT)
> -#define paddr_to_pfn(pa)  ((unsigned long)((pa) >> PAGE_SHIFT))

I'm a little puzzled here: Just a few patches ago you introduced them into
asm/page.h, and (only) now you're removing them here.

> +#define paddr_to_pdx(pa)    mfn_to_pdx(maddr_to_mfn(pa))
> +#define gfn_to_gaddr(gfn)   pfn_to_paddr(gfn_x(gfn))
> +#define gaddr_to_gfn(ga)    _gfn(paddr_to_pfn(ga))
> +#define mfn_to_maddr(mfn)   pfn_to_paddr(mfn_x(mfn))
> +#define maddr_to_mfn(ma)    _mfn(paddr_to_pfn(ma))
> +#define vmap_to_mfn(va)     maddr_to_mfn(virt_to_maddr((vaddr_t)va))
> +#define vmap_to_page(va)    mfn_to_page(vmap_to_mfn(va))
> +#define paddr_to_pdx(pa)    mfn_to_pdx(maddr_to_mfn(pa))
> +#define gfn_to_gaddr(gfn)   pfn_to_paddr(gfn_x(gfn))
> +#define gaddr_to_gfn(ga)    _gfn(paddr_to_pfn(ga))
> +#define mfn_to_maddr(mfn)   pfn_to_paddr(mfn_x(mfn))
> +#define maddr_to_mfn(ma)    _mfn(paddr_to_pfn(ma))
> +#define vmap_to_mfn(va)     maddr_to_mfn(virt_to_maddr((vaddr_t)va))
> +#define vmap_to_page(va)    mfn_to_page(vmap_to_mfn(va))
> +
> +#define virt_to_maddr(va) ((paddr_t)((vaddr_t)(va) & PADDR_MASK))
> +#define maddr_to_virt(pa) ((void *)((paddr_t)(pa) | XEN_VIRT_START))

Is this really XEN_VIRT_START? I.e. does your directmap start there,
right where the Xen image area also starts?

> +/* Convert between Xen-heap virtual addresses and machine frame numbers. */
> +#define __virt_to_mfn(va) (virt_to_maddr(va) >> PAGE_SHIFT)
> +#define __mfn_to_virt(mfn) (maddr_to_virt((paddr_t)(mfn) << PAGE_SHIFT))

Nit: Excess parentheses again.

> +/* Convert between Xen-heap virtual addresses and page-info structures. */
> +static inline struct page_info *virt_to_page(const void *v)
> +{
> +    BUG();
> +    return NULL;
> +}
> +
> +/*
> + * We define non-underscored wrappers for above conversion functions.
> + * These are overriden in various source files while underscored version
> + * remain intact.
> + */
> +#define virt_to_mfn(va)     __virt_to_mfn(va)
> +#define mfn_to_virt(mfn)    __mfn_to_virt(mfn)
> +
> +struct page_info
> +{
> +    /* Each frame can be threaded onto a doubly-linked list. */
> +    struct page_list_entry list;
> +
> +    /* Reference count and various PGC_xxx flags and fields. */
> +    unsigned long count_info;
> +
> +    /* Context-dependent fields follow... */
> +    union {
> +        /* Page is in use: ((count_info & PGC_count_mask) != 0). */
> +        struct {
> +            /* Type reference count and various PGT_xxx flags and fields. */
> +            unsigned long type_info;
> +        } inuse;
> +        /* Page is on a free list: ((count_info & PGC_count_mask) == 0). */
> +        union {
> +            struct {
> +                /*
> +                 * Index of the first *possibly* unscrubbed page in the buddy.
> +                 * One more bit than maximum possible order to accommodate
> +                 * INVALID_DIRTY_IDX.
> +                 */
> +#define INVALID_DIRTY_IDX ((1UL << (MAX_ORDER + 1)) - 1)
> +                unsigned long first_dirty:MAX_ORDER + 1;
> +
> +                /* Do TLBs need flushing for safety before next page use? */
> +                bool need_tlbflush:1;
> +
> +#define BUDDY_NOT_SCRUBBING    0
> +#define BUDDY_SCRUBBING        1
> +#define BUDDY_SCRUB_ABORT      2
> +                unsigned long scrub_state:2;
> +            };
> +
> +            unsigned long val;
> +            } free;

Something's wrong with indentation here.

> +    } u;
> +
> +    union {
> +        /* Page is in use, but not as a shadow. */
> +        struct {
> +            /* Owner of this page (zero if page is anonymous). */
> +            struct domain *domain;
> +        } inuse;
> +
> +        /* Page is on a free list. */
> +        struct {
> +            /* Order-size of the free chunk this page is the head of. */
> +            unsigned int order;
> +        } free;
> +
> +    } v;
> +
> +    union {
> +        /*
> +         * Timestamp from 'TLB clock', used to avoid extra safety flushes.
> +         * Only valid for: a) free pages, and b) pages with zero type count
> +         */
> +        u32 tlbflush_timestamp;

Nit: uint32_t and ...

> +    };
> +    u64 pad;

... uint64_t please in new code. Assuming, for the latter, you really need
the field in the first place: I can't see what it's needed for.

> +};
> +
> +#define frame_table ((struct page_info *)FRAMETABLE_VIRT_START)
> +
> +/* PDX of the first page in the frame table. */
> +extern unsigned long frametable_base_pdx;
> +
> +/* Convert between machine frame numbers and page-info structures. */
> +#define mfn_to_page(mfn)                                            \
> +    (frame_table + (mfn_to_pdx(mfn) - frametable_base_pdx))
> +#define page_to_mfn(pg)                                             \
> +    pdx_to_mfn((unsigned long)((pg) - frame_table) + frametable_base_pdx)
> +
> +static inline void *page_to_virt(const struct page_info *pg)
> +{
> +    return mfn_to_virt(mfn_x(page_to_mfn(pg)));
> +}
> +
> +/*
> + * Common code requires get_page_type and put_page_type.
> + * We don't care about typecounts so we just do the minimum to make it
> + * happy.
> + */
> +static inline int get_page_type(struct page_info *page, unsigned long type)
> +{
> +    return 1;
> +}
> +
> +static inline void put_page_type(struct page_info *page)
> +{
> +    return;

No need for this; the body can ve entirely empty, as we have it elsewhere.

> +}
> +
> +/* TODO */
> +static inline bool get_page_nr(struct page_info *page, const struct domain *domain,
> +                        unsigned long nr)
> +{
> +    BUG();
> +}
> +static inline void put_page_nr(struct page_info *page, unsigned long nr)
> +{
> +    BUG();
> +}

What are these two needed for?

> +static inline void put_page_and_type(struct page_info *page)
> +{
> +    put_page_type(page);
> +    put_page(page);
> +}
> +
> +/*
> + * RISCV does not have an M2P, but common code expects a handful of
> + * M2P-related defines and functions. Provide dummy versions of these.
> + */
> +#define INVALID_M2P_ENTRY        (~0UL)
> +#define SHARED_M2P_ENTRY         (~0UL - 1UL)
> +#define SHARED_M2P(_e)           ((_e) == SHARED_M2P_ENTRY)
> +
> +/* Xen always owns P2M on PPC */

PPC?

> +#define set_gpfn_from_mfn(mfn, pfn) do { (void) (mfn), (void)(pfn); } while (0)
> +#define mfn_to_gfn(d, mfn) ((void)(d), _gfn(mfn_x(mfn)))
> +
> +#define PDX_GROUP_SHIFT (16 + 5)
> +
> +static inline unsigned long domain_get_maximum_gpfn(struct domain *d)
> +{
> +    BUG();
> +    return 0;
> +}
> +
> +static inline long arch_memory_op(int op, XEN_GUEST_HANDLE_PARAM(void) arg)
> +{
> +    BUG();
> +    return 0;
> +}
> +
> +/*
> + * On RISCV, all the RAM is currently direct mapped in Xen.
> + * Hence return always true.
> + */
> +static inline bool arch_mfns_in_directmap(unsigned long mfn, unsigned long nr)
> +{
> +    return true;
> +}
> +
> +#define PG_shift(idx)   (BITS_PER_LONG - (idx))
> +#define PG_mask(x, idx) (x ## UL << PG_shift(idx))
> +
> +#define PGT_none          PG_mask(0, 1)  /* no special uses of this page   */
> +#define PGT_writable_page PG_mask(1, 1)  /* has writable mappings?         */
> +#define PGT_type_mask     PG_mask(1, 1)  /* Bits 31 or 63.                 */
> +
> + /* Count of uses of this frame as its current type. */
> +#define PGT_count_width   PG_shift(2)
> +#define PGT_count_mask    ((1UL<<PGT_count_width)-1)
> +
> +/*
> + * Page needs to be scrubbed. Since this bit can only be set on a page that is
> + * free (i.e. in PGC_state_free) we can reuse PGC_allocated bit.
> + */
> +#define _PGC_need_scrub   _PGC_allocated
> +#define PGC_need_scrub    PGC_allocated
> +
> +//  /* Cleared when the owning guest 'frees' this page. */
> +#define _PGC_allocated    PG_shift(1)
> +#define PGC_allocated     PG_mask(1, 1)
> +  /* Page is Xen heap? */
> +#define _PGC_xen_heap     PG_shift(2)
> +#define PGC_xen_heap      PG_mask(1, 2)
> +#ifdef CONFIG_STATIC_MEMORY
> +/* Page is static memory */
> +#define _PGC_static    PG_shift(3)
> +#define PGC_static     PG_mask(1, 3)
> +#else
> +#define PGC_static     0
> +#endif

Please omit this until you really know whether you're going to support
static memory.

> +/* ... */

???

> +/* Page is broken? */
> +#define _PGC_broken       PG_shift(7)
> +#define PGC_broken        PG_mask(1, 7)
> + /* Mutually-exclusive page states: { inuse, offlining, offlined, free }. */
> +#define PGC_state         PG_mask(3, 9)
> +#define PGC_state_inuse   PG_mask(0, 9)
> +#define PGC_state_offlining PG_mask(1, 9)
> +#define PGC_state_offlined PG_mask(2, 9)
> +#define PGC_state_free    PG_mask(3, 9)
> +// #define page_state_is(pg, st) (((pg)->count_info&PGC_state) == PGC_state_##st)
> +
> +/* Count of references to this frame. */
> +#define PGC_count_width   PG_shift(9)
> +#define PGC_count_mask    ((1UL<<PGC_count_width)-1)
> +
> +#define page_state_is(pg, st) (((pg)->count_info&PGC_state) == PGC_state_##st)
> +
> +#define _PGC_extra        PG_shift(10)
> +#define PGC_extra         PG_mask(1, 10)
> +
> +#define is_xen_heap_page(page) ((page)->count_info & PGC_xen_heap)
> +#define is_xen_heap_mfn(mfn) \
> +    (mfn_valid(mfn) && is_xen_heap_page(mfn_to_page(mfn)))
> +
> +#define is_xen_fixed_mfn(mfn)                                   \
> +    ((mfn_to_maddr(mfn) >= virt_to_maddr(&_start)) &&           \
> +     (mfn_to_maddr(mfn) <= virt_to_maddr((vaddr_t)_end - 1)))
> +
> +#define page_get_owner(_p)    (_p)->v.inuse.domain
> +#define page_set_owner(_p,_d) ((_p)->v.inuse.domain = (_d))
> +
> +/* TODO: implement */
> +#define mfn_valid(mfn) ({ (void) (mfn); 0; })
> +// #define max_page (0UL)

???

> +#define mfn_to_gfn(d, mfn) ((void)(d), _gfn(mfn_x(mfn)))
> +
> +#define domain_set_alloc_bitsize(d) ((void)0)
> +#define domain_clamp_alloc_bitsize(d, b) (b)
> +
> +#define PFN_ORDER(_pfn) ((_pfn)->v.free.order)

No leading underscore needed here, I suppose.

Jan

Re: [PATCH v2 21/39] xen/riscv: introduce bit operations

[Oleksii]

On Thu, 2023-12-14 at 14:27 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> 
> Did you write this code from scratch? If not, you need to at least
> point
> out the origin. But: None of this looks RISC-V specific, so shouldn't
> it
> instead be put under xen/lib/, as a fallback implementation for arch-
> es
> not having optimized implementations? Looks also at least very
> similar
> to Arm64's, so that one perhaps then will want dropping as well? We
> surely don't want to carry two (almost) identical copies of the same
> logic.
No, I took the code from Arm because as you mentioned it doesn't have
RISC-V specific things.

I think we can move it to xen/lib. Would it be better to as part of
this patch series?

> 
> > --- /dev/null
> > +++ b/xen/arch/riscv/lib/Makefile
> > @@ -0,0 +1 @@
> > +obj-y += find_next_bit.o
> 
> Dashes instead of underscores please in new file's names, whenever
> possible.
Sure. I'll update that. Thanks.

~ Oleksii

Re: [PATCH v2 23/39] xen/riscv: introduce asm/guest_access.h

[Oleksii]

On Thu, 2023-12-14 at 15:06 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > --- /dev/null
> > +++ b/xen/arch/riscv/include/asm/guest_access.h
> > @@ -0,0 +1,29 @@
> > +#ifndef __ASM_RISCV_GUEST_ACCESS_H__
> > +#define __ASM_RISCV_GUEST_ACCESS_H__
> > +
> > +#include <xen/types.h>
> > +
> > +unsigned long raw_copy_to_guest(void *to, const void *from,
> > unsigned len);
> > +unsigned long raw_copy_from_guest(void *to, const void *from,
> > unsigned len);
> 
> For the full build to work, you will need to implement these
> somewhere.
> From the titles of further patches I can't guess where that would be.
> Note how PPC has decided to have these be inline dummies for the time
> being. It also looks as if you would also need raw_clear_guest()?
It is implemented in xen/arch/riscv/stubs.c:362,367.
Regarding raw_clear_guest() I'll double check. I don't have a
compilation issue with it, so I ignored this macros.

> > +#define __raw_copy_to_guest raw_copy_to_guest
> > +#define __raw_copy_from_guest raw_copy_from_guest
> > +
> > +#define guest_handle_okay(hnd, nr) (1)
> > +#define guest_handle_subrange_okay(hnd, first, last) (1)
> 
> Arm has a comment next to these (actually two, I mean the latter),
> which
> I think would be sensible to also have here.
Thanks. I'll add the comment.

> 
> > +struct domain;
> > +unsigned long copy_to_guest_phys(struct domain *d,
> > +                                 paddr_t gpa,
> > +                                 void *buf,
> > +                                 unsigned int len);
> 
> You don't need this just yet, do you?
Yes, you are right. This function isn't needed for now.

Thanks. I'll drop it.

~ Oleksii

Re: [PATCH v2 24/39] xen/riscv: introduce asm/irq.h

[Oleksii]

On Thu, 2023-12-14 at 15:09 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> > ---
> > Changes in V2:
> > 	- add ifdef CONFIG_HAS_DEVICE_TREE for things that
> > shouldn't be
> >       in case !CONFIG_HAS_DEVICE_TREE
> 
> Is there going to be a RISC-V build without this enabled (selected)?
> If
> not, I'd recommend against such pointless #ifdef-ary.
For this stage (Xen RISC-V full build), CONFIG_HAS_DEVICE_TREE will not
be selected, but it will be in the near future.

~ Oleksii

Re: [PATCH v2 21/39] xen/riscv: introduce bit operations

[Jan Beulich]

On 18.12.2023 10:56, Oleksii wrote:
> On Thu, 2023-12-14 at 14:27 +0100, Jan Beulich wrote:
>> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>>> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
>>
>> Did you write this code from scratch? If not, you need to at least
>> point
>> out the origin. But: None of this looks RISC-V specific, so shouldn't
>> it
>> instead be put under xen/lib/, as a fallback implementation for arch-
>> es
>> not having optimized implementations? Looks also at least very
>> similar
>> to Arm64's, so that one perhaps then will want dropping as well? We
>> surely don't want to carry two (almost) identical copies of the same
>> logic.
> No, I took the code from Arm because as you mentioned it doesn't have
> RISC-V specific things.
> 
> I think we can move it to xen/lib. Would it be better to as part of
> this patch series?

Doesn't matter much whether it's separate or part of this series, I'd say.

Jan

Re: [PATCH v2 25/39] xen/riscv: introduce asm/p2m.h

[Oleksii]

On Thu, 2023-12-14 at 15:19 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > --- /dev/null
> > +++ b/xen/arch/riscv/include/asm/p2m.h
> > @@ -0,0 +1,105 @@
> > +#ifndef __ASM_RISCV_P2M_H__
> > +#define __ASM_RISCV_P2M_H__
> > +
> > +#include <asm/page-bits.h>
> > +
> > +#define paddr_bits PADDR_BITS
> > +
> > +/*
> > + * List of possible type for each page in the p2m entry.
> > + * The number of available bit per page in the pte for this
> > purpose is 4 bits.
> > + * So it's possible to only have 16 fields. If we run out of value
> > in the
> > + * future, it's possible to use higher value for pseudo-type and
> > don't store
> > + * them in the p2m entry.
> > + */
> > +typedef enum {
> > +    p2m_invalid = 0,    /* Nothing mapped here */
> > +    p2m_ram_rw,         /* Normal read/write guest RAM */
> > +    p2m_ram_ro,         /* Read-only; writes are silently dropped
> > */
> > +    p2m_mmio_direct_dev,/* Read/write mapping of genuine Device
> > MMIO area */
> > +    p2m_mmio_direct_nc, /* Read/write mapping of genuine MMIO area
> > non-cacheable */
> > +    p2m_mmio_direct_c,  /* Read/write mapping of genuine MMIO area
> > cacheable */
> > +    p2m_map_foreign_rw, /* Read/write RAM pages from foreign
> > domain */
> > +    p2m_map_foreign_ro, /* Read-only RAM pages from foreign domain
> > */
> > +    p2m_grant_map_rw,   /* Read/write grant mapping */
> > +    p2m_grant_map_ro,   /* Read-only grant mapping */
> > +    /* The types below are only used to decide the page attribute
> > in the P2M */
> > +    p2m_iommu_map_rw,   /* Read/write iommu mapping */
> > +    p2m_iommu_map_ro,   /* Read-only iommu mapping */
> > +    p2m_max_real_type,  /* Types after this won't be store in the
> > p2m */
> > +} p2m_type_t;
> 
> As indicated before, I think you should have only those types here
> which
> you're sure you'll need, or even just those which you really need
> right
> away. I question in particular p2m_mmio_direct_*, which all look like
> you simply took them from Arm, without regard as to applicability to
> RISC-V. The fewer types you have here, the easier it is going to be
> to
> tell what needs adding and what is already _properly_ supported.
Agreed. I have to leave only the ones that are necessary for now.
I'll update that.

> 
> > +#include <xen/p2m-common.h>
> > +
> > +static inline int get_page_and_type(struct page_info *page,
> > +                                    struct domain *domain,
> > +                                    unsigned long type)
> > +{
> > +    BUG();
> > +    return 1;
> > +}
> 
> Imo despite the BUG() any such stub would better return failure, just
> like ...
Thanks. I'll update that.

> 
> > +/* Look up a GFN and take a reference count on the backing page.
> > */
> > +typedef unsigned int p2m_query_t;
> > +#define P2M_ALLOC    (1u<<0)   /* Populate PoD and paged-out
> > entries */
> > +#define P2M_UNSHARE  (1u<<1)   /* Break CoW sharing */
> > +
> > +static inline struct page_info *get_page_from_gfn(
> > +    struct domain *d, unsigned long gfn, p2m_type_t *t,
> > p2m_query_t q)
> > +{
> > +    BUG();
> > +    return NULL;
> > +}
> 
> ... you do here. May apply again further down.
> 
> > +static inline void memory_type_changed(struct domain *d)
> > +{
> > +    BUG();
> > +}
> > +
> > +
> > +static inline int guest_physmap_mark_populate_on_demand(struct
> > domain *d, unsigned long gfn,
> > +                                                        unsigned
> > int order)
> > +{
> > +    BUG();
> > +    return 1;
> > +}
> > +
> > +static inline int guest_physmap_add_entry(struct domain *d,
> > +                            gfn_t gfn,
> > +                            mfn_t mfn,
> > +                            unsigned long page_order,
> > +                            p2m_type_t t)
> > +{
> > +    BUG();
> > +    return 1;
> > +}
> > +
> > +/* Untyped version for RAM only, for compatibility */
> > +static inline int __must_check
> > +guest_physmap_add_page(struct domain *d, gfn_t gfn, mfn_t mfn,
> > +                       unsigned int page_order)
> > +{
> > +    return guest_physmap_add_entry(d, gfn, mfn, page_order,
> > p2m_ram_rw);
> > +}
> > +
> > +static inline mfn_t gfn_to_mfn(struct domain *d, gfn_t gfn)
> > +{
> > +    BUG();
> > +    return _mfn(0);
> > +}
> > +
> > +static inline bool arch_acquire_resource_check(struct domain *d)
> > +{
> > +    /*
> > +     * The reference counting of foreign entries in
> > set_foreign_p2m_entry()
> > +     * is supported on RISCV.
> > +     */
> > +    return true;
> > +}
> > +
> > +static inline void p2m_altp2m_check(struct vcpu *v, uint16_t idx)
> > +{
> > +    /* Not supported on RISCV. */
> > +}
> > +
> > +#endif /* __ASM_RISCV_P2M_H__ */
> > \ No newline at end of file
> 
> This wants taking care of.
Thanks. I'll update that.

~ Oleksii

Re: [PATCH v2 26/39] xen/riscv: introduce asm/regs.h

[Oleksii]

On Thu, 2023-12-14 at 16:05 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > --- /dev/null
> > +++ b/xen/arch/riscv/include/asm/regs.h
> > @@ -0,0 +1,26 @@
> > +#ifndef __ARM_RISCV_REGS_H__
> > +#define __ARM_RISCV_REGS_H__
> > +
> > +#ifndef __ASSEMBLY__
> > +
> > +#include <xen/bug.h>
> > +#include <asm/current.h>
> 
> Does one of these bring in asm/processor.h, for ...
> 
> > +#define hyp_mode(r)     (0)
> > +
> > +static inline bool guest_mode(const struct cpu_user_regs *r)
> 
> ... struct cpu_user_regs to be available? Else a forward declaration
> would be needed here.
Agreed. It will be better to use a forward declaration. I'll change
that in next patch series.

> 
> Acked-by: Jan Beulich <jbeulich@suse.com>
Thanks.

~ Oleksii

Re: [PATCH v2 27/39] xen/riscv: introduce asm/time.h

[Oleksii]

On Thu, 2023-12-14 at 16:06 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > --- /dev/null
> > +++ b/xen/arch/riscv/include/asm/time.h
> > @@ -0,0 +1,19 @@
> > +#ifndef __ASM_RISCV_TIME_H__
> > +#define __ASM_RISCV_TIME_H__
> > +
> > +#include <xen/bug.h>
> > +#include <asm/csr.h>
> > +
> > +struct vcpu;
> > +
> > +/* TODO: implement */
> > +static inline void force_update_vcpu_system_time(struct vcpu *v) {
> > BUG(); }
> > +
> > +typedef unsigned long cycles_t;
> > +
> > +static inline cycles_t get_cycles(void)
> > +{
> > +	return csr_read(CSR_TIME);
> > +}
> > +
> > +#endif /* __ASM_RISCV_TIME_H__ */
> > \ No newline at end of file
> 
> With this taken care of (and the SPDX header added)
> Acked-by: Jan Beulich <jbeulich@suse.com>
Thanks.

I'll add SPDX and fix "No newline..."

~ Oleksii

Re: [PATCH v2 28/39] xen/riscv: introduce asm/event.h

[Oleksii]

On Thu, 2023-12-14 at 16:08 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > --- /dev/null
> > +++ b/xen/arch/riscv/include/asm/event.h
> > @@ -0,0 +1,34 @@
> > +#ifndef __ASM_RISCV_EVENT_H__
> > +#define __ASM_RISCV_EVENT_H__
> > +
> > +void vcpu_mark_events_pending(struct vcpu *v);
> > +
> > +static inline int vcpu_event_delivery_is_enabled(struct vcpu *v)
> > +{
> > +    return 0;
> > +}
> > +
> > +static inline int local_events_need_delivery(void)
> > +{
> > +    return 0;
> > +}
> > +
> > +static inline void local_event_delivery_enable(void)
> > +{
> > +}
> 
> These are stubs aiui, and hence would better have BUG() in them?
Yes, for now it would be better to add BUG(). Thanks.

> 
> > +/* No arch specific virq definition now. Default to global. */
> > +static inline bool arch_virq_is_global(unsigned int virq)
> > +{
> > +    return true;
> > +}
> 
> This otoh may well remain as is for the foreseeable future.
> 
> Jan

Re: [PATCH v2 23/39] xen/riscv: introduce asm/guest_access.h

[Jan Beulich]

On 18.12.2023 11:02, Oleksii wrote:
> On Thu, 2023-12-14 at 15:06 +0100, Jan Beulich wrote:
>> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>>> --- /dev/null
>>> +++ b/xen/arch/riscv/include/asm/guest_access.h
>>> @@ -0,0 +1,29 @@
>>> +#ifndef __ASM_RISCV_GUEST_ACCESS_H__
>>> +#define __ASM_RISCV_GUEST_ACCESS_H__
>>> +
>>> +#include <xen/types.h>
>>> +
>>> +unsigned long raw_copy_to_guest(void *to, const void *from,
>>> unsigned len);
>>> +unsigned long raw_copy_from_guest(void *to, const void *from,
>>> unsigned len);
>>
>> For the full build to work, you will need to implement these
>> somewhere.
>> From the titles of further patches I can't guess where that would be.
>> Note how PPC has decided to have these be inline dummies for the time
>> being. It also looks as if you would also need raw_clear_guest()?
> It is implemented in xen/arch/riscv/stubs.c:362,367.

Ah, that's patch 37, which I didn't get to yet.

Jan

> Regarding raw_clear_guest() I'll double check. I don't have a
> compilation issue with it, so I ignored this macros.

Re: [PATCH v2 24/39] xen/riscv: introduce asm/irq.h

[Jan Beulich]

On 18.12.2023 11:04, Oleksii wrote:
> On Thu, 2023-12-14 at 15:09 +0100, Jan Beulich wrote:
>> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>>> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
>>> ---
>>> Changes in V2:
>>> 	- add ifdef CONFIG_HAS_DEVICE_TREE for things that
>>> shouldn't be
>>>       in case !CONFIG_HAS_DEVICE_TREE
>>
>> Is there going to be a RISC-V build without this enabled (selected)?
>> If
>> not, I'd recommend against such pointless #ifdef-ary.
> For this stage (Xen RISC-V full build), CONFIG_HAS_DEVICE_TREE will not
> be selected, but it will be in the near future.

And from then on it'll always be selected, or only conditionally? In the
former case it would still feel odd if #ifdef-s were introduced.

Jan

Re: [PATCH v2 30/39] xen/riscv: define an address of frame table

[Oleksii]

On Thu, 2023-12-14 at 16:48 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > Also the patchs adds some helpful macros.
> 
> In how far they're (going to be) helpful is hard to tell without uses
> and without some suitable comments.
> 
> > --- a/xen/arch/riscv/include/asm/config.h
> > +++ b/xen/arch/riscv/include/asm/config.h
> > @@ -77,12 +77,31 @@
> >    name:
> >  #endif
> >  
> > +#define VPN_BITS    (9)
> > +#define OFFSET_BITS (12)
> 
> Whose offset? In how far is this different from PAGE_SHIFT?
It is page offset ( RISC-V terminology names it so ) and it is not
different with PAGE_SHIFT. OFFSET_BITS can be dropped.
> 
> >  #ifdef CONFIG_RISCV_64
> > +
> > +#define SLOTN_ENTRY_BITS        (HYP_PT_ROOT_LEVEL * VPN_BITS +
> > OFFSET_BITS)
> > +#define SLOTN(slot)             (_AT(vaddr_t,slot) <<
> > SLOTN_ENTRY_BITS)
> 
> Nit: Missing blank after comma.
Thanks. I'll update that.

> 
> > +#define SLOTN_ENTRY_SIZE        SLOTN(1)
> > +
> >  #define XEN_VIRT_START 0xFFFFFFFFC0000000 /* (_AC(-1, UL) + 1 -
> > GB(1)) */
> > +
> > +#define FRAMETABLE_VIRT_START   SLOTN(196)
> > +#define FRAMETABLE_SIZE         GB(3)
> > +#define FRAMETABLE_NR           (FRAMETABLE_SIZE /
> > sizeof(*frame_table))
> > +#define FRAMETABLE_VIRT_END     (FRAMETABLE_VIRT_START +
> > FRAMETABLE_SIZE - 1)
> > +
> > +#define VMAP_VIRT_START         SLOTN(194)
> > +#define VMAP_VIRT_SIZE          GB(1)
> 
> May I suggest that you keep these blocks sorted by slot number? Or
> wait,
> the layout comment further up is also in decreasing order, so that's
> fine here, but then can all of this please be moved next to the
> comment
> actually providing the necessary context (thus eliminating the need
> for
> new comments)?
Sure, I'll put this part close to layout comment.

>  You'll then also notice that the generalization here
> (keeping basically the same layout for e.g. SATP_MODE_SV48, just
> shifted
> by 9 bits) isn't in line with the comment there.
Does it make sense to add another one table with updated addresses for
SATP_MODE_SV48?
Microchip has h/w which requires SATP_MODE_SV48 ( at least ), so I have
a patch which introduces SATP_MODE_SV48 and I planned to update the
layout table in this patch.

> 
> > @@ -95,6 +114,8 @@
> >  #define RV_STAGE1_MODE SATP_MODE_SV32
> >  #endif
> >  
> > +#define HYP_PT_ROOT_LEVEL (CONFIG_PAGING_LEVELS - 1)
> 
> I understand that CONFIG_PAGING_LEVELS is defined only just up from
> here,
> but what that identifier stands for is quite clear. It would seem to
> me
> that moving this up ahead if its first use would help clarity.
Thanks. It can be useful, so I'll move it up.

~ Oleksii

Re: [PATCH v2 31/39] xen/riscv: add required things to asm/current.h

[Oleksii]

On Thu, 2023-12-14 at 16:55 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > --- a/xen/arch/riscv/include/asm/current.h
> > +++ b/xen/arch/riscv/include/asm/current.h
> > @@ -3,6 +3,22 @@
> >  #ifndef __ASM_CURRENT_H
> >  #define __ASM_CURRENT_H
> >  
> > +#include <xen/percpu.h>
> > +#include <asm/processor.h>
> > +
> > +#ifndef __ASSEMBLY__
> > +
> > +struct vcpu;
> 
> I don't think you need this here?
Shouldn't forward declaration be for the case of curr_vcpu declaration
in the next line ?
> 
> > +/* Which VCPU is "current" on this PCPU. */
> > +DECLARE_PER_CPU(struct vcpu *, curr_vcpu);
> > +
> > +#define current            (this_cpu(curr_vcpu))
> 
> Nit: No need for the extra parentheses.
Thanks.

> 
> > +#define set_current(vcpu)  do { current = (vcpu); } while (0)
> > +#define get_cpu_current(cpu)  (per_cpu(curr_vcpu, cpu))
> 
> Same here then.
> 
> > +#define guest_cpu_user_regs() (0)
> 
> 0 or NULL? Also perhaps better 
> 
> #define guest_cpu_user_regs() ({ BUG(); NULL; })
> 
> until it's properly implemented?
Thanks. I'll use your definition.

> 
> > @@ -10,4 +26,8 @@
> >      unreachable();                                          \
> >  } while ( false )
> >  
> > +#define get_per_cpu_offset() __per_cpu_offset[get_processor_id()]
> 
> I'd like to remind you that there's no get_processor_id() anymore.
Yeah, missed that. Thanks.

~ Oleksii

Re: [PATCH v2 32/39] xen/riscv: add minimal stuff to asm/page.h to build full Xen

[Oleksii]

On Thu, 2023-12-14 at 16:57 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> 
> Acked-by: Jan Beulich <jbeulich@suse.com>
> 
> I wonder though ...
> 
> > --- a/xen/arch/riscv/include/asm/page.h
> > +++ b/xen/arch/riscv/include/asm/page.h
> > @@ -6,6 +6,7 @@
> >  #ifndef __ASSEMBLY__
> >  
> >  #include <xen/const.h>
> > +#include <xen/bug.h>
> >  #include <xen/types.h>
> >  
> >  #include <asm/mm.h>
> > @@ -32,6 +33,9 @@
> >  #define PTE_LEAF_DEFAULT            (PTE_VALID | PTE_READABLE |
> > PTE_WRITABLE)
> >  #define PTE_TABLE                   (PTE_VALID)
> >  
> > +/* TODO */
> > +#define PAGE_HYPERVISOR 0
> 
> ... whether this couldn't be defined properly right away.
It can be introduced now but it requires some additional defines to be
introduced in the same time:

#define _PAGE_W_BIT     2
#define _PAGE_XN_BIT    3
#define _PAGE_RO_BIT    1
#define _PAGE_XN        (1U << _PAGE_XN_BIT)
#define _PAGE_RO        (1U << _PAGE_RO_BIT)
#define _PAGE_W         (1U << _PAGE_W_BIT)

...
/*
 * _PAGE_DEVICE and _PAGE_NORMAL are convenience defines. They are not
 * meant to be used outside of this header.
 */
// #define _PAGE_DEVICE    _PAGE_XN
#define _PAGE_NORMAL    _PAGE_PRESENT

#define PAGE_HYPERVISOR_RW      (_PAGE_NORMAL | _PAGE_RO | _PAGE_XN |
_PAGE_W)

#define PAGE_HYPERVISOR         PAGE_HYPERVISOR_RW

And _PAGE_PRESENT in pgtbl-bits.h:

#define _PAGE_PRESENT   (1 << 0)

I prefer to introduce all this things when it will be really used.

~ Oleksii

Re: [PATCH v2 33/39] xen/riscv: add minimal stuff to asm/processor.h to build full Xen

[Oleksii]

On Thu, 2023-12-14 at 17:04 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > --- a/xen/arch/riscv/include/asm/processor.h
> > +++ b/xen/arch/riscv/include/asm/processor.h
> > @@ -12,6 +12,9 @@
> >  
> >  #ifndef __ASSEMBLY__
> >  
> > +/* TODO: need to be implemeted */
> > +#define get_processor_id() 0
> 
> Please don't re-introduce this - it was just recently dropped from
> the
> code base.
Thanks for remind me that. I'll drop it in the next version.

> 
> > @@ -53,6 +56,18 @@ struct cpu_user_regs
> >      unsigned long pregs;
> >  };
> >  
> > +/* TODO: need to implement */
> > +#define cpu_to_core(_cpu)   (0)
> > +#define cpu_to_socket(_cpu) (0)
> 
> No need for leading underscores here.
Sure. Thanks.
> 
> > +static inline void cpu_relax(void)
> > +{
> > +	int dummy;
> > +	/* In lieu of a halt instruction, induce a long-latency
> > stall. */
> > +	__asm__ __volatile__ ("div %0, %0, zero" : "=r" (dummy));
> 
> Any reason for this, when Arm's is just barrier(), and apparently
> they got
> away with this quite fine? Also isn't this causing a division by
> zero,
> which I'd expect to cause some kind of exception? (Terminology-wise
> I'm of
> course biased by x86, where "halt instruction" wouldn't be suitable
> to use
> here. But if that terminology is fine on RISC-V, then obviously no
> objection.)
It was based on Linux kernel code:
https://elixir.bootlin.com/linux/latest/source/arch/riscv/include/asm/vdso/processor.h#L9

But looks I missed barrier()...
Probably it will be better update cpu_relax() to:

	/* Encoding of the pause instruction */
	__asm__ __volatile__ (".4byte 0x100000F");

	barrier();

~ Oleksii

Re: [PATCH v2 35/39] xen: add necessary headers to common to build full Xen for RISC-V

[Oleksii]

On Thu, 2023-12-14 at 17:20 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> 
> With an empty description it is hard to judge whether this is really
> needed.
> I would sincerely hope we can get away without. Note how there
> already a few
> struct xen_domctl_* forward declarations there, which - if the
> #include
> really needs adding - should be dropped in exchange.
> 
> As a nit on the subject - it's only singular (now?) and could hence
> do with
> being more precise on what the specific change is.
It will be another one reminder for me to write proper commit messages
as I don't remember what was an issue.

I double checked what was the issue and it seems that this change isn't
needed any more... So I'll drop this patch.

~ Oleksii
> 
> Jan
> 
> > --- a/xen/include/xen/domain.h
> > +++ b/xen/include/xen/domain.h
> > @@ -4,6 +4,7 @@
> >  
> >  #include <xen/types.h>
> >  
> > +#include <public/domctl.h>
> >  #include <public/xen.h>
> >  
> >  struct guest_area {
> 

Re: [PATCH v2 30/39] xen/riscv: define an address of frame table

[Jan Beulich]

On 18.12.2023 11:36, Oleksii wrote:
> On Thu, 2023-12-14 at 16:48 +0100, Jan Beulich wrote:
>> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>>> +#define SLOTN_ENTRY_SIZE        SLOTN(1)
>>> +
>>>  #define XEN_VIRT_START 0xFFFFFFFFC0000000 /* (_AC(-1, UL) + 1 -
>>> GB(1)) */
>>> +
>>> +#define FRAMETABLE_VIRT_START   SLOTN(196)
>>> +#define FRAMETABLE_SIZE         GB(3)
>>> +#define FRAMETABLE_NR           (FRAMETABLE_SIZE /
>>> sizeof(*frame_table))
>>> +#define FRAMETABLE_VIRT_END     (FRAMETABLE_VIRT_START +
>>> FRAMETABLE_SIZE - 1)
>>> +
>>> +#define VMAP_VIRT_START         SLOTN(194)
>>> +#define VMAP_VIRT_SIZE          GB(1)
>>
>> May I suggest that you keep these blocks sorted by slot number? Or
>> wait,
>> the layout comment further up is also in decreasing order, so that's
>> fine here, but then can all of this please be moved next to the
>> comment
>> actually providing the necessary context (thus eliminating the need
>> for
>> new comments)?
> Sure, I'll put this part close to layout comment.
> 
>>  You'll then also notice that the generalization here
>> (keeping basically the same layout for e.g. SATP_MODE_SV48, just
>> shifted
>> by 9 bits) isn't in line with the comment there.
> Does it make sense to add another one table with updated addresses for
> SATP_MODE_SV48?

Well, especially if you mean to support that mode, its layout surely
wants writing down. I was hoping though that maybe you/we could get away
without multiple tables, but e.g. use one having multiple columns.

Jan

> Microchip has h/w which requires SATP_MODE_SV48 ( at least ), so I have
> a patch which introduces SATP_MODE_SV48 and I planned to update the
> layout table in this patch.

Re: [PATCH v2 31/39] xen/riscv: add required things to asm/current.h

[Jan Beulich]

On 18.12.2023 11:39, Oleksii wrote:
> On Thu, 2023-12-14 at 16:55 +0100, Jan Beulich wrote:
>> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>>> --- a/xen/arch/riscv/include/asm/current.h
>>> +++ b/xen/arch/riscv/include/asm/current.h
>>> @@ -3,6 +3,22 @@
>>>  #ifndef __ASM_CURRENT_H
>>>  #define __ASM_CURRENT_H
>>>  
>>> +#include <xen/percpu.h>
>>> +#include <asm/processor.h>
>>> +
>>> +#ifndef __ASSEMBLY__
>>> +
>>> +struct vcpu;
>>
>> I don't think you need this here?
> Shouldn't forward declaration be for the case of curr_vcpu declaration
> in the next line ?

I don't think so. In C forward decls are needed only when an otherwise
undeclared type is used as type of a function parameter. (C++ is different
in this regard.)

Jan

>>> +/* Which VCPU is "current" on this PCPU. */
>>> +DECLARE_PER_CPU(struct vcpu *, curr_vcpu);
>>> +
>>> +#define current            (this_cpu(curr_vcpu))

Re: [PATCH v2 36/39] xen/riscv: add minimal stuff to asm/mm.h to build full Xen

[Oleksii]

On Thu, 2023-12-14 at 18:08 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > --- a/xen/arch/riscv/include/asm/mm.h
> > +++ b/xen/arch/riscv/include/asm/mm.h
> > @@ -3,10 +3,271 @@
> >  #ifndef _ASM_RISCV_MM_H
> >  #define _ASM_RISCV_MM_H
> >  
> > +#include <public/xen.h>
> > +#include <xen/pdx.h>
> > +#include <xen/types.h>
> > +
> > +#include <asm/page.h>
> >  #include <asm/page-bits.h>
> >  
> > -#define pfn_to_paddr(pfn) ((paddr_t)(pfn) << PAGE_SHIFT)
> > -#define paddr_to_pfn(pa)  ((unsigned long)((pa) >> PAGE_SHIFT))
> 
> I'm a little puzzled here: Just a few patches ago you introduced them
> into
> asm/page.h, and (only) now you're removing them here.
I think it was some issue during rebasing of latest changes and
staging.

> 
> > +#define paddr_to_pdx(pa)    mfn_to_pdx(maddr_to_mfn(pa))
> > +#define gfn_to_gaddr(gfn)   pfn_to_paddr(gfn_x(gfn))
> > +#define gaddr_to_gfn(ga)    _gfn(paddr_to_pfn(ga))
> > +#define mfn_to_maddr(mfn)   pfn_to_paddr(mfn_x(mfn))
> > +#define maddr_to_mfn(ma)    _mfn(paddr_to_pfn(ma))
> > +#define vmap_to_mfn(va)    
> > maddr_to_mfn(virt_to_maddr((vaddr_t)va))
> > +#define vmap_to_page(va)    mfn_to_page(vmap_to_mfn(va))
> > +#define paddr_to_pdx(pa)    mfn_to_pdx(maddr_to_mfn(pa))
> > +#define gfn_to_gaddr(gfn)   pfn_to_paddr(gfn_x(gfn))
> > +#define gaddr_to_gfn(ga)    _gfn(paddr_to_pfn(ga))
> > +#define mfn_to_maddr(mfn)   pfn_to_paddr(mfn_x(mfn))
> > +#define maddr_to_mfn(ma)    _mfn(paddr_to_pfn(ma))
> > +#define vmap_to_mfn(va)    
> > maddr_to_mfn(virt_to_maddr((vaddr_t)va))
> > +#define vmap_to_page(va)    mfn_to_page(vmap_to_mfn(va))
> > +
> > +#define virt_to_maddr(va) ((paddr_t)((vaddr_t)(va) & PADDR_MASK))
> > +#define maddr_to_virt(pa) ((void *)((paddr_t)(pa) |
> > XEN_VIRT_START))
> 
> Is this really XEN_VIRT_START? I.e. does your directmap start there,
> right where the Xen image area also starts?
No, it is not. I tried to re-use PPC's version of mm.h ( as it was
already merged ) and missed to update this macros. Actually right now
it is defined for RISC-V as:

static inline void *maddr_to_virt(paddr_t ma)
{
    ASSERT((mfn_to_pdx(maddr_to_mfn(ma)) - directmap_base_pdx) <
           (DIRECTMAP_SIZE >> PAGE_SHIFT));
    return (void *)(XENHEAP_VIRT_START -
                    (directmap_base_pdx << PAGE_SHIFT) +
                    ((ma & ma_va_bottom_mask) |
                     ((ma & ma_top_mask) >> pfn_pdx_hole_shift)));
}


but I think about to stub it with BUG() if it will work or to change
XEN_VIRT_START to DIRECTMAP_VIRT_START.

> 
> > +/* Convert between Xen-heap virtual addresses and machine frame
> > numbers. */
> > +#define __virt_to_mfn(va) (virt_to_maddr(va) >> PAGE_SHIFT)
> > +#define __mfn_to_virt(mfn) (maddr_to_virt((paddr_t)(mfn) <<
> > PAGE_SHIFT))
> 
> Nit: Excess parentheses again.
Thanks. Ill update.

> 
> > +/* Convert between Xen-heap virtual addresses and page-info
> > structures. */
> > +static inline struct page_info *virt_to_page(const void *v)
> > +{
> > +    BUG();
> > +    return NULL;
> > +}
> > +
> > +/*
> > + * We define non-underscored wrappers for above conversion
> > functions.
> > + * These are overriden in various source files while underscored
> > version
> > + * remain intact.
> > + */
> > +#define virt_to_mfn(va)     __virt_to_mfn(va)
> > +#define mfn_to_virt(mfn)    __mfn_to_virt(mfn)
> > +
> > +struct page_info
> > +{
> > +    /* Each frame can be threaded onto a doubly-linked list. */
> > +    struct page_list_entry list;
> > +
> > +    /* Reference count and various PGC_xxx flags and fields. */
> > +    unsigned long count_info;
> > +
> > +    /* Context-dependent fields follow... */
> > +    union {
> > +        /* Page is in use: ((count_info & PGC_count_mask) != 0).
> > */
> > +        struct {
> > +            /* Type reference count and various PGT_xxx flags and
> > fields. */
> > +            unsigned long type_info;
> > +        } inuse;
> > +        /* Page is on a free list: ((count_info & PGC_count_mask)
> > == 0). */
> > +        union {
> > +            struct {
> > +                /*
> > +                 * Index of the first *possibly* unscrubbed page
> > in the buddy.
> > +                 * One more bit than maximum possible order to
> > accommodate
> > +                 * INVALID_DIRTY_IDX.
> > +                 */
> > +#define INVALID_DIRTY_IDX ((1UL << (MAX_ORDER + 1)) - 1)
> > +                unsigned long first_dirty:MAX_ORDER + 1;
> > +
> > +                /* Do TLBs need flushing for safety before next
> > page use? */
> > +                bool need_tlbflush:1;
> > +
> > +#define BUDDY_NOT_SCRUBBING    0
> > +#define BUDDY_SCRUBBING        1
> > +#define BUDDY_SCRUB_ABORT      2
> > +                unsigned long scrub_state:2;
> > +            };
> > +
> > +            unsigned long val;
> > +            } free;
> 
> Something's wrong with indentation here.
Thanks. Ill double check.

> 
> > +    } u;
> > +
> > +    union {
> > +        /* Page is in use, but not as a shadow. */
> > +        struct {
> > +            /* Owner of this page (zero if page is anonymous). */
> > +            struct domain *domain;
> > +        } inuse;
> > +
> > +        /* Page is on a free list. */
> > +        struct {
> > +            /* Order-size of the free chunk this page is the head
> > of. */
> > +            unsigned int order;
> > +        } free;
> > +
> > +    } v;
> > +
> > +    union {
> > +        /*
> > +         * Timestamp from 'TLB clock', used to avoid extra safety
> > flushes.
> > +         * Only valid for: a) free pages, and b) pages with zero
> > type count
> > +         */
> > +        u32 tlbflush_timestamp;
> 
> Nit: uint32_t and ...
> 
> > +    };
> > +    u64 pad;
> 
> ... uint64_t please in new code. Assuming, for the latter, you really
> need
> the field in the first place: I can't see what it's needed for.
This structure was copied from Arm's mm.h.
I am not sure too if it is needed. It seems to me it should be 8 byte
aligned without this pad for 64-bit archs.


> 
> > +};
> > +
> > +#define frame_table ((struct page_info *)FRAMETABLE_VIRT_START)
> > +
> > +/* PDX of the first page in the frame table. */
> > +extern unsigned long frametable_base_pdx;
> > +
> > +/* Convert between machine frame numbers and page-info structures.
> > */
> > +#define
> > mfn_to_page(mfn)                                            \
> > +    (frame_table + (mfn_to_pdx(mfn) - frametable_base_pdx))
> > +#define
> > page_to_mfn(pg)                                             \
> > +    pdx_to_mfn((unsigned long)((pg) - frame_table) +
> > frametable_base_pdx)
> > +
> > +static inline void *page_to_virt(const struct page_info *pg)
> > +{
> > +    return mfn_to_virt(mfn_x(page_to_mfn(pg)));
> > +}
> > +
> > +/*
> > + * Common code requires get_page_type and put_page_type.
> > + * We don't care about typecounts so we just do the minimum to
> > make it
> > + * happy.
> > + */
> > +static inline int get_page_type(struct page_info *page, unsigned
> > long type)
> > +{
> > +    return 1;
> > +}
> > +
> > +static inline void put_page_type(struct page_info *page)
> > +{
> > +    return;
> 
> No need for this; the body can ve entirely empty, as we have it
> elsewhere.
Thanks. I'll update.

> 
> > +}
> > +
> > +/* TODO */
> > +static inline bool get_page_nr(struct page_info *page, const
> > struct domain *domain,
> > +                        unsigned long nr)
> > +{
> > +    BUG();
> > +}
> > +static inline void put_page_nr(struct page_info *page, unsigned
> > long nr)
> > +{
> > +    BUG();
> > +}
> 
> What are these two needed for?
> 
> > +static inline void put_page_and_type(struct page_info *page)
> > +{
> > +    put_page_type(page);
> > +    put_page(page);
> > +}
> > +
> > +/*
> > + * RISCV does not have an M2P, but common code expects a handful
> > of
> > + * M2P-related defines and functions. Provide dummy versions of
> > these.
> > + */
> > +#define INVALID_M2P_ENTRY        (~0UL)
> > +#define SHARED_M2P_ENTRY         (~0UL - 1UL)
> > +#define SHARED_M2P(_e)           ((_e) == SHARED_M2P_ENTRY)
> > +
> > +/* Xen always owns P2M on PPC */
> 
> PPC?
Should be RISC-V. Missed to change.

> 
> > +#define set_gpfn_from_mfn(mfn, pfn) do { (void) (mfn),
> > (void)(pfn); } while (0)
> > +#define mfn_to_gfn(d, mfn) ((void)(d), _gfn(mfn_x(mfn)))
> > +
> > +#define PDX_GROUP_SHIFT (16 + 5)
> > +
> > +static inline unsigned long domain_get_maximum_gpfn(struct domain
> > *d)
> > +{
> > +    BUG();
> > +    return 0;
> > +}
> > +
> > +static inline long arch_memory_op(int op,
> > XEN_GUEST_HANDLE_PARAM(void) arg)
> > +{
> > +    BUG();
> > +    return 0;
> > +}
> > +
> > +/*
> > + * On RISCV, all the RAM is currently direct mapped in Xen.
> > + * Hence return always true.
> > + */
> > +static inline bool arch_mfns_in_directmap(unsigned long mfn,
> > unsigned long nr)
> > +{
> > +    return true;
> > +}
> > +
> > +#define PG_shift(idx)   (BITS_PER_LONG - (idx))
> > +#define PG_mask(x, idx) (x ## UL << PG_shift(idx))
> > +
> > +#define PGT_none          PG_mask(0, 1)  /* no special uses of
> > this page   */
> > +#define PGT_writable_page PG_mask(1, 1)  /* has writable
> > mappings?         */
> > +#define PGT_type_mask     PG_mask(1, 1)  /* Bits 31 or
> > 63.                 */
> > +
> > + /* Count of uses of this frame as its current type. */
> > +#define PGT_count_width   PG_shift(2)
> > +#define PGT_count_mask    ((1UL<<PGT_count_width)-1)
> > +
> > +/*
> > + * Page needs to be scrubbed. Since this bit can only be set on a
> > page that is
> > + * free (i.e. in PGC_state_free) we can reuse PGC_allocated bit.
> > + */
> > +#define _PGC_need_scrub   _PGC_allocated
> > +#define PGC_need_scrub    PGC_allocated
> > +
> > +//  /* Cleared when the owning guest 'frees' this page. */
> > +#define _PGC_allocated    PG_shift(1)
> > +#define PGC_allocated     PG_mask(1, 1)
> > +  /* Page is Xen heap? */
> > +#define _PGC_xen_heap     PG_shift(2)
> > +#define PGC_xen_heap      PG_mask(1, 2)
> > +#ifdef CONFIG_STATIC_MEMORY
> > +/* Page is static memory */
> > +#define _PGC_static    PG_shift(3)
> > +#define PGC_static     PG_mask(1, 3)
> > +#else
> > +#define PGC_static     0
> > +#endif
> 
> Please omit this until you really know whether you're going to
> support
> static memory.
Ok. I'll drop that.
> 
> > +/* ... */
> 
> ???
It was taken for Arm and I don't know what is the purpose of the
comment so it makes sense to drop it.
> 
> > +/* Page is broken? */
> > +#define _PGC_broken       PG_shift(7)
> > +#define PGC_broken        PG_mask(1, 7)
> > + /* Mutually-exclusive page states: { inuse, offlining, offlined,
> > free }. */
> > +#define PGC_state         PG_mask(3, 9)
> > +#define PGC_state_inuse   PG_mask(0, 9)
> > +#define PGC_state_offlining PG_mask(1, 9)
> > +#define PGC_state_offlined PG_mask(2, 9)
> > +#define PGC_state_free    PG_mask(3, 9)
> > +// #define page_state_is(pg, st) (((pg)->count_info&PGC_state) ==
> > PGC_state_##st)
> > +
> > +/* Count of references to this frame. */
> > +#define PGC_count_width   PG_shift(9)
> > +#define PGC_count_mask    ((1UL<<PGC_count_width)-1)
> > +
> > +#define page_state_is(pg, st) (((pg)->count_info&PGC_state) ==
> > PGC_state_##st)
> > +
> > +#define _PGC_extra        PG_shift(10)
> > +#define PGC_extra         PG_mask(1, 10)
> > +
> > +#define is_xen_heap_page(page) ((page)->count_info & PGC_xen_heap)
> > +#define is_xen_heap_mfn(mfn) \
> > +    (mfn_valid(mfn) && is_xen_heap_page(mfn_to_page(mfn)))
> > +
> > +#define is_xen_fixed_mfn(mfn)                                   \
> > +    ((mfn_to_maddr(mfn) >= virt_to_maddr(&_start)) &&           \
> > +     (mfn_to_maddr(mfn) <= virt_to_maddr((vaddr_t)_end - 1)))
> > +
> > +#define page_get_owner(_p)    (_p)->v.inuse.domain
> > +#define page_set_owner(_p,_d) ((_p)->v.inuse.domain = (_d))
> > +
> > +/* TODO: implement */
> > +#define mfn_valid(mfn) ({ (void) (mfn); 0; })
> > +// #define max_page (0UL)
> 
> ???
This macros isn't needed for now ( when I tried to make minimal build I
just commented such places and forgot to remove it ).
> 
> > +#define mfn_to_gfn(d, mfn) ((void)(d), _gfn(mfn_x(mfn)))
> > +
> > +#define domain_set_alloc_bitsize(d) ((void)0)
> > +#define domain_clamp_alloc_bitsize(d, b) (b)
> > +
> > +#define PFN_ORDER(_pfn) ((_pfn)->v.free.order)
> 
> No leading underscore needed here, I suppose.
Thanks. I'll remove underscore.

~ Oleksii

Re: [PATCH v2 32/39] xen/riscv: add minimal stuff to asm/page.h to build full Xen

[Jan Beulich]

On 18.12.2023 11:45, Oleksii wrote:
> On Thu, 2023-12-14 at 16:57 +0100, Jan Beulich wrote:
>> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>>> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
>>
>> Acked-by: Jan Beulich <jbeulich@suse.com>
>>
>> I wonder though ...
>>
>>> --- a/xen/arch/riscv/include/asm/page.h
>>> +++ b/xen/arch/riscv/include/asm/page.h
>>> @@ -6,6 +6,7 @@
>>>  #ifndef __ASSEMBLY__
>>>  
>>>  #include <xen/const.h>
>>> +#include <xen/bug.h>
>>>  #include <xen/types.h>
>>>  
>>>  #include <asm/mm.h>
>>> @@ -32,6 +33,9 @@
>>>  #define PTE_LEAF_DEFAULT            (PTE_VALID | PTE_READABLE |
>>> PTE_WRITABLE)
>>>  #define PTE_TABLE                   (PTE_VALID)
>>>  
>>> +/* TODO */
>>> +#define PAGE_HYPERVISOR 0
>>
>> ... whether this couldn't be defined properly right away.
> It can be introduced now but it requires some additional defines to be
> introduced in the same time:
> 
> #define _PAGE_W_BIT     2
> #define _PAGE_XN_BIT    3
> #define _PAGE_RO_BIT    1
> #define _PAGE_XN        (1U << _PAGE_XN_BIT)
> #define _PAGE_RO        (1U << _PAGE_RO_BIT)
> #define _PAGE_W         (1U << _PAGE_W_BIT)

Why would you need these, when you already have
PTE_{READABLE,WRITABLE,EXECUTABLE} just out of context from above? (And
that's aside from me (a) not following the naming (vs their purpose) and
(b) not seeing what _PAGE_*_BIT are needed for, not even thinking about
the leading underscores in these identifiers again.)

> ...
> /*
>  * _PAGE_DEVICE and _PAGE_NORMAL are convenience defines. They are not
>  * meant to be used outside of this header.
>  */
> // #define _PAGE_DEVICE    _PAGE_XN
> #define _PAGE_NORMAL    _PAGE_PRESENT
> 
> #define PAGE_HYPERVISOR_RW      (_PAGE_NORMAL | _PAGE_RO | _PAGE_XN |
> _PAGE_W)
> 
> #define PAGE_HYPERVISOR         PAGE_HYPERVISOR_RW
> 
> And _PAGE_PRESENT in pgtbl-bits.h:
> 
> #define _PAGE_PRESENT   (1 << 0)
> 
> I prefer to introduce all this things when it will be really used.

I understand that, yet for easy things it may help doing them right
away, rather than leaving them to be touched (in a straightforward way)
again very soon.

Jan

Re: [PATCH v2 33/39] xen/riscv: add minimal stuff to asm/processor.h to build full Xen

[Jan Beulich]

On 18.12.2023 11:49, Oleksii wrote:
> On Thu, 2023-12-14 at 17:04 +0100, Jan Beulich wrote:
>> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>>> +static inline void cpu_relax(void)
>>> +{
>>> +	int dummy;
>>> +	/* In lieu of a halt instruction, induce a long-latency
>>> stall. */
>>> +	__asm__ __volatile__ ("div %0, %0, zero" : "=r" (dummy));
>>
>> Any reason for this, when Arm's is just barrier(), and apparently
>> they got
>> away with this quite fine? Also isn't this causing a division by
>> zero,
>> which I'd expect to cause some kind of exception? (Terminology-wise
>> I'm of
>> course biased by x86, where "halt instruction" wouldn't be suitable
>> to use
>> here. But if that terminology is fine on RISC-V, then obviously no
>> objection.)
> It was based on Linux kernel code:
> https://elixir.bootlin.com/linux/latest/source/arch/riscv/include/asm/vdso/processor.h#L9
> 
> But looks I missed barrier()...
> Probably it will be better update cpu_relax() to:
> 
> 	/* Encoding of the pause instruction */
> 	__asm__ __volatile__ (".4byte 0x100000F");
> 
> 	barrier();

But definitely without .4byte, which defines a piece of data. If for
whatever reason you don't want to use "pause" directly, please use
.insn.

Jan

Re: [PATCH v2 24/39] xen/riscv: introduce asm/irq.h

[Oleksii]

On Mon, 2023-12-18 at 11:12 +0100, Jan Beulich wrote:
> On 18.12.2023 11:04, Oleksii wrote:
> > On Thu, 2023-12-14 at 15:09 +0100, Jan Beulich wrote:
> > > On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > > > Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> > > > ---
> > > > Changes in V2:
> > > > 	- add ifdef CONFIG_HAS_DEVICE_TREE for things that
> > > > shouldn't be
> > > >       in case !CONFIG_HAS_DEVICE_TREE
> > > 
> > > Is there going to be a RISC-V build without this enabled
> > > (selected)?
> > > If
> > > not, I'd recommend against such pointless #ifdef-ary.
> > For this stage (Xen RISC-V full build), CONFIG_HAS_DEVICE_TREE will
> > not
> > be selected, but it will be in the near future.
> 
> And from then on it'll always be selected, or only conditionally? In
> the
> former case it would still feel odd if #ifdef-s were introduced.
It will always be selected until ACPI support is provided. I've seen
patches that add ACPI support for the Linux kernel, but I'm not sure if
it is really needed at this point. So, I've planned to go with Device
Tree for quite a while

~ Oleksii

Re: [PATCH v2 31/39] xen/riscv: add required things to asm/current.h

[Oleksii]

On Mon, 2023-12-18 at 12:28 +0100, Jan Beulich wrote:
> On 18.12.2023 11:39, Oleksii wrote:
> > On Thu, 2023-12-14 at 16:55 +0100, Jan Beulich wrote:
> > > On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > > > --- a/xen/arch/riscv/include/asm/current.h
> > > > +++ b/xen/arch/riscv/include/asm/current.h
> > > > @@ -3,6 +3,22 @@
> > > >  #ifndef __ASM_CURRENT_H
> > > >  #define __ASM_CURRENT_H
> > > >  
> > > > +#include <xen/percpu.h>
> > > > +#include <asm/processor.h>
> > > > +
> > > > +#ifndef __ASSEMBLY__
> > > > +
> > > > +struct vcpu;
> > > 
> > > I don't think you need this here?
> > Shouldn't forward declaration be for the case of curr_vcpu
> > declaration
> > in the next line ?
> 
> I don't think so. In C forward decls are needed only when an
> otherwise
> undeclared type is used as type of a function parameter. (C++ is
> different
> in this regard.)
Thanks for clarifying; in this case, it can be dropped.


~ Oleksii
> > > > +/* Which VCPU is "current" on this PCPU. */
> > > > +DECLARE_PER_CPU(struct vcpu *, curr_vcpu);
> > > > +
> > > > +#define current            (this_cpu(curr_vcpu))
> 

Re: [PATCH v2 32/39] xen/riscv: add minimal stuff to asm/page.h to build full Xen

[Oleksii]

On Mon, 2023-12-18 at 12:36 +0100, Jan Beulich wrote:
> On 18.12.2023 11:45, Oleksii wrote:
> > On Thu, 2023-12-14 at 16:57 +0100, Jan Beulich wrote:
> > > On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > > > Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> > > 
> > > Acked-by: Jan Beulich <jbeulich@suse.com>
> > > 
> > > I wonder though ...
> > > 
> > > > --- a/xen/arch/riscv/include/asm/page.h
> > > > +++ b/xen/arch/riscv/include/asm/page.h
> > > > @@ -6,6 +6,7 @@
> > > >  #ifndef __ASSEMBLY__
> > > >  
> > > >  #include <xen/const.h>
> > > > +#include <xen/bug.h>
> > > >  #include <xen/types.h>
> > > >  
> > > >  #include <asm/mm.h>
> > > > @@ -32,6 +33,9 @@
> > > >  #define PTE_LEAF_DEFAULT            (PTE_VALID | PTE_READABLE
> > > > |
> > > > PTE_WRITABLE)
> > > >  #define PTE_TABLE                   (PTE_VALID)
> > > >  
> > > > +/* TODO */
> > > > +#define PAGE_HYPERVISOR 0
> > > 
> > > ... whether this couldn't be defined properly right away.
> > It can be introduced now but it requires some additional defines to
> > be
> > introduced in the same time:
> > 
> > #define _PAGE_W_BIT     2
> > #define _PAGE_XN_BIT    3
> > #define _PAGE_RO_BIT    1
> > #define _PAGE_XN        (1U << _PAGE_XN_BIT)
> > #define _PAGE_RO        (1U << _PAGE_RO_BIT)
> > #define _PAGE_W         (1U << _PAGE_W_BIT)
> 
> Why would you need these, when you already have
> PTE_{READABLE,WRITABLE,EXECUTABLE} just out of context from above?
I thought that the hypervisor page table is fully software-related, and
that's why a separate PAGE*BIT was introduced. These bits can be
different from PTE* bits, which are hardware-related
https://gitlab.com/xen-project/xen/-/blob/staging/xen/arch/arm/include/asm/page.h?ref_type=heads#L66

It looks like I misunderstood that, and PTE* can be used everywhere
instead of _PAGE*. Alternatively, we could consider renaming everything
to PAGE* to maintain consistency across architectures.

Does it make sense?


> (And
> that's aside from me (a) not following the naming (vs their purpose)
> and
> (b) not seeing what _PAGE_*_BIT are needed for, not even thinking
> about
> the leading underscores in these identifiers again.)
> 
> > ...
> > /*
> >  * _PAGE_DEVICE and _PAGE_NORMAL are convenience defines. They are
> > not
> >  * meant to be used outside of this header.
> >  */
> > // #define _PAGE_DEVICE    _PAGE_XN
> > #define _PAGE_NORMAL    _PAGE_PRESENT
> > 
> > #define PAGE_HYPERVISOR_RW      (_PAGE_NORMAL | _PAGE_RO | _PAGE_XN
> > |
> > _PAGE_W)
> > 
> > #define PAGE_HYPERVISOR         PAGE_HYPERVISOR_RW
> > 
> > And _PAGE_PRESENT in pgtbl-bits.h:
> > 
> > #define _PAGE_PRESENT   (1 << 0)
> > 
> > I prefer to introduce all this things when it will be really used.
> 
> I understand that, yet for easy things it may help doing them right
> away, rather than leaving them to be touched (in a straightforward
> way)
> again very soon.
> 
~ Oleksii

Re: [PATCH v2 32/39] xen/riscv: add minimal stuff to asm/page.h to build full Xen

[Jan Beulich]

On 18.12.2023 12:57, Oleksii wrote:
> On Mon, 2023-12-18 at 12:36 +0100, Jan Beulich wrote:
>> On 18.12.2023 11:45, Oleksii wrote:
>>> On Thu, 2023-12-14 at 16:57 +0100, Jan Beulich wrote:
>>>> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>>>>> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
>>>>
>>>> Acked-by: Jan Beulich <jbeulich@suse.com>
>>>>
>>>> I wonder though ...
>>>>
>>>>> --- a/xen/arch/riscv/include/asm/page.h
>>>>> +++ b/xen/arch/riscv/include/asm/page.h
>>>>> @@ -6,6 +6,7 @@
>>>>>  #ifndef __ASSEMBLY__
>>>>>  
>>>>>  #include <xen/const.h>
>>>>> +#include <xen/bug.h>
>>>>>  #include <xen/types.h>
>>>>>  
>>>>>  #include <asm/mm.h>
>>>>> @@ -32,6 +33,9 @@
>>>>>  #define PTE_LEAF_DEFAULT            (PTE_VALID | PTE_READABLE
>>>>> |
>>>>> PTE_WRITABLE)
>>>>>  #define PTE_TABLE                   (PTE_VALID)
>>>>>  
>>>>> +/* TODO */
>>>>> +#define PAGE_HYPERVISOR 0
>>>>
>>>> ... whether this couldn't be defined properly right away.
>>> It can be introduced now but it requires some additional defines to
>>> be
>>> introduced in the same time:
>>>
>>> #define _PAGE_W_BIT     2
>>> #define _PAGE_XN_BIT    3
>>> #define _PAGE_RO_BIT    1
>>> #define _PAGE_XN        (1U << _PAGE_XN_BIT)
>>> #define _PAGE_RO        (1U << _PAGE_RO_BIT)
>>> #define _PAGE_W         (1U << _PAGE_W_BIT)
>>
>> Why would you need these, when you already have
>> PTE_{READABLE,WRITABLE,EXECUTABLE} just out of context from above?
> I thought that the hypervisor page table is fully software-related, and
> that's why a separate PAGE*BIT was introduced. These bits can be
> different from PTE* bits, which are hardware-related
> https://gitlab.com/xen-project/xen/-/blob/staging/xen/arch/arm/include/asm/page.h?ref_type=heads#L66
> 
> It looks like I misunderstood that, and PTE* can be used everywhere
> instead of _PAGE*. Alternatively, we could consider renaming everything
> to PAGE* to maintain consistency across architectures.
> 
> Does it make sense?

Sure. Whether renaming makes sense is harder to tell though. It would
be only the name prefix that's uniform, as how exactly e.g. permissions
are controlled is arch-specific anyway. The one place I'm aware where
PAGE_* (or, sadly, still _PAGE_*) would matter for common code is
_PAGE_NONE (not sure though whether that's something that can / wants
to be expressed for RISC-V).

Jan

Re: [PATCH v2 38/39] xen/riscv: enable full Xen build

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>

Reviewed-by: Jan Beulich <jbeulich@suse.com>
with ...

> --- a/xen/arch/riscv/configs/tiny64_defconfig
> +++ b/xen/arch/riscv/configs/tiny64_defconfig
> @@ -24,7 +24,6 @@
>  # CONFIG_COVERAGE is not set
>  # CONFIG_UBSAN is not set
>  # CONFIG_NEEDS_LIBELF is not set
> -
>  CONFIG_RISCV_64=y
>  CONFIG_DEBUG=y
>  CONFIG_DEBUG_INFO=y

... this unrelated (and perhaps even unhelpful) change dropped.

Jan

Re: [PATCH v2 37/39] xen/rirscv: add minimal amount of stubs to build full Xen

[Jan Beulich]

On 24.11.2023 11:30, Oleksii Kurochko wrote:
> --- a/xen/arch/riscv/mm.c
> +++ b/xen/arch/riscv/mm.c
> @@ -1,19 +1,23 @@
>  /* SPDX-License-Identifier: GPL-2.0-only */
>  
> +#include <xen/bug.h>
>  #include <xen/cache.h>
>  #include <xen/compiler.h>
>  #include <xen/init.h>
>  #include <xen/kernel.h>
>  #include <xen/macros.h>
> +#include <xen/mm.h>
>  #include <xen/pfn.h>
>  
>  #include <asm/early_printk.h>
>  #include <asm/csr.h>
>  #include <asm/current.h>
> -#include <asm/mm.h>
>  #include <asm/page.h>
>  #include <asm/processor.h>
>  
> +unsigned long frametable_base_pdx __read_mostly;
> +unsigned long frametable_virt_end __read_mostly;

Nit (style):

unsigned long __read_mostly frametable_base_pdx;
unsigned long __read_mostly frametable_virt_end;

(i.e. attributes generally between type and identifier). Plus
__read_mostly or __ro_after_init?

> @@ -294,3 +298,49 @@ unsigned long __init calc_phys_offset(void)
>      phys_offset = load_start - XEN_VIRT_START;
>      return phys_offset;
>  }
> +
> +void put_page(struct page_info *page)
> +{
> +    assert_failed(__func__);
> +}
> +
> +unsigned long get_upper_mfn_bound(void)
> +{
> +    /* No memory hotplug yet, so current memory limit is the final one. */
> +    return max_page - 1;
> +}
> +
> +void arch_dump_shared_mem_info(void)
> +{
> +    WARN();
> +}
> +
> +int populate_pt_range(unsigned long virt, unsigned long nr_mfns)
> +{
> +    assert_failed(__func__);
> +    return -1;
> +}

Whats the pattern between picking WARN(), assert_failed() (which I don't
think you should be using anyway; if an assertion, then ASSERT_UNREACHABLE())
and BUG() (as used earlier in stubs living in header files)?

> --- /dev/null
> +++ b/xen/arch/riscv/stubs.c
> @@ -0,0 +1,426 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +#include <xen/cpumask.h>
> +#include <xen/domain.h>
> +#include <xen/irq.h>
> +#include <xen/nodemask.h>
> +#include <xen/time.h>
> +#include <public/domctl.h>
> +#include <public/vm_event.h>

I think I can see why you need the former of these last two, but do you
really need the latter?

> +#include <asm/current.h>
> +
> +/* smpboot.c */
> +
> +cpumask_t cpu_online_map;
> +cpumask_t cpu_present_map;
> +cpumask_t cpu_possible_map;
> +
> +/* ID of the PCPU we're running on */
> +DEFINE_PER_CPU(unsigned int, cpu_id);
> +/* XXX these seem awfully x86ish... */
> +/* representing HT siblings of each logical CPU */
> +DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_mask);
> +/* representing HT and core siblings of each logical CPU */
> +DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_core_mask);
> +
> +nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
> +
> +/* time.c */
> +
> +unsigned long __read_mostly cpu_khz;  /* CPU clock frequency in kHz. */
> +
> +s_time_t get_s_time(void)
> +{
> +    BUG();
> +}
> +
> +int reprogram_timer(s_time_t timeout)
> +{
> +    BUG();
> +}
> +
> +void send_timer_event(struct vcpu *v)
> +{
> +    BUG();
> +}
> +
> +void domain_set_time_offset(struct domain *d, int64_t time_offset_seconds)
> +{
> +    BUG();
> +}
> +
> +/* shutdown.c */
> +
> +void machine_restart(unsigned int delay_millisecs)
> +{
> +    BUG();
> +}
> +
> +void machine_halt(void)
> +{
> +    BUG();
> +}
> +
> +/* vm_event.c */
> +
> +void vm_event_fill_regs(vm_event_request_t *req)
> +{
> +    BUG();
> +}
> +
> +void vm_event_set_registers(struct vcpu *v, vm_event_response_t *rsp)
> +{
> +    BUG();
> +}
> +
> +void vm_event_monitor_next_interrupt(struct vcpu *v)
> +{
> +    /* Not supported on RISCV. */
> +}
> +
> +void vm_event_reset_vmtrace(struct vcpu *v)
> +{
> +    /* Not supported on RISCV. */
> +}
> +
> +/* domctl.c */
> +
> +long arch_do_domctl(struct xen_domctl *domctl, struct domain *d,
> +                    XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl)
> +{
> +    BUG();
> +}
> +
> +void arch_get_domain_info(const struct domain *d,
> +                          struct xen_domctl_getdomaininfo *info)
> +{
> +    BUG();
> +}
> +
> +void arch_get_info_guest(struct vcpu *v, vcpu_guest_context_u c)
> +{
> +    BUG();
> +}
> +
> +/* monitor.c */
> +
> +int arch_monitor_domctl_event(struct domain *d,
> +                              struct xen_domctl_monitor_op *mop)
> +{
> +    BUG();
> +}
> +
> +/* smp.c */
> +
> +void arch_flush_tlb_mask(const cpumask_t *mask)
> +{
> +    BUG();
> +}
> +
> +void smp_send_event_check_mask(const cpumask_t *mask)
> +{
> +    BUG();
> +}
> +
> +void smp_send_call_function_mask(const cpumask_t *mask)
> +{
> +    BUG();
> +}
> +
> +/* irq.c */
> +
> +struct pirq *alloc_pirq_struct(struct domain *d)
> +{
> +    BUG();
> +}
> +
> +int pirq_guest_bind(struct vcpu *v, struct pirq *pirq, int will_share)
> +{
> +    BUG();
> +}
> +
> +void pirq_guest_unbind(struct domain *d, struct pirq *pirq)
> +{
> +    BUG();
> +}
> +
> +void pirq_set_affinity(struct domain *d, int pirq, const cpumask_t *mask)
> +{
> +    BUG();
> +}
> +
> +static void ack_none(struct irq_desc *irq)
> +{
> +    BUG();
> +}
> +
> +static void end_none(struct irq_desc *irq)
> +{
> +    BUG();
> +}

Much like I said for PPC - I don't think you need the two, as ...

> +hw_irq_controller no_irq_type = {
> +    .typename = "none",
> +    .startup = irq_startup_none,
> +    .shutdown = irq_shutdown_none,
> +    .enable = irq_enable_none,
> +    .disable = irq_disable_none,
> +    .ack = ack_none,
> +    .end = end_none

... there's nothing right now to invoke these hooks.

Jan

Re: [PATCH v2 37/39] xen/rirscv: add minimal amount of stubs to build full Xen

[Oleksii]

On Mon, 2023-12-18 at 18:00 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > --- a/xen/arch/riscv/mm.c
> > +++ b/xen/arch/riscv/mm.c
> > @@ -1,19 +1,23 @@
> >  /* SPDX-License-Identifier: GPL-2.0-only */
> >  
> > +#include <xen/bug.h>
> >  #include <xen/cache.h>
> >  #include <xen/compiler.h>
> >  #include <xen/init.h>
> >  #include <xen/kernel.h>
> >  #include <xen/macros.h>
> > +#include <xen/mm.h>
> >  #include <xen/pfn.h>
> >  
> >  #include <asm/early_printk.h>
> >  #include <asm/csr.h>
> >  #include <asm/current.h>
> > -#include <asm/mm.h>
> >  #include <asm/page.h>
> >  #include <asm/processor.h>
> >  
> > +unsigned long frametable_base_pdx __read_mostly;
> > +unsigned long frametable_virt_end __read_mostly;
> 
> Nit (style):
> 
> unsigned long __read_mostly frametable_base_pdx;
> unsigned long __read_mostly frametable_virt_end;
> 
> (i.e. attributes generally between type and identifier). Plus
> __read_mostly or __ro_after_init?
I'll update the style.
I looked at code where this variables are used and they can be
__ro_after_init.
Thanks.
> 
> > @@ -294,3 +298,49 @@ unsigned long __init calc_phys_offset(void)
> >      phys_offset = load_start - XEN_VIRT_START;
> >      return phys_offset;
> >  }
> > +
> > +void put_page(struct page_info *page)
> > +{
> > +    assert_failed(__func__);
> > +}
> > +
> > +unsigned long get_upper_mfn_bound(void)
> > +{
> > +    /* No memory hotplug yet, so current memory limit is the final
> > one. */
> > +    return max_page - 1;
> > +}
> > +
> > +void arch_dump_shared_mem_info(void)
> > +{
> > +    WARN();
> > +}
> > +
> > +int populate_pt_range(unsigned long virt, unsigned long nr_mfns)
> > +{
> > +    assert_failed(__func__);
> > +    return -1;
> > +}
> 
> Whats the pattern between picking WARN(), assert_failed() (which I
> don't
> think you should be using anyway; if an assertion, then
> ASSERT_UNREACHABLE())
> and BUG() (as used earlier in stubs living in header files)?
There is no specific pattern; initially, I used WARN() everywhere.
However, when the time came to implement this function, it became
challenging to identify the location of some WARN() occurrences.
Consequently, I started changing them to assert_failed(__func__) to
pinpoint the source.

I'll be switching to BUG()
> 
> > --- /dev/null
> > +++ b/xen/arch/riscv/stubs.c
> > @@ -0,0 +1,426 @@
> > +/* SPDX-License-Identifier: GPL-2.0-only */
> > +#include <xen/cpumask.h>
> > +#include <xen/domain.h>
> > +#include <xen/irq.h>
> > +#include <xen/nodemask.h>
> > +#include <xen/time.h>
> > +#include <public/domctl.h>
> > +#include <public/vm_event.h>
> 
> I think I can see why you need the former of these last two, but do
> you
> really need the latter?
It is needed for vm_event_request_t and vm_event_response_t, but if use
a forward declaration that it won't be needed:

typedef struct vm_event_st vm_event_request_t;
typedef struct vm_event_st vm_event_response_t;
> 
> > +#include <asm/current.h>
> > +
> > +/* smpboot.c */
> > +
> > +cpumask_t cpu_online_map;
> > +cpumask_t cpu_present_map;
> > +cpumask_t cpu_possible_map;
> > +
> > +/* ID of the PCPU we're running on */
> > +DEFINE_PER_CPU(unsigned int, cpu_id);
> > +/* XXX these seem awfully x86ish... */
> > +/* representing HT siblings of each logical CPU */
> > +DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_mask);
> > +/* representing HT and core siblings of each logical CPU */
> > +DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_core_mask);
> > +
> > +nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
> > +
> > +/* time.c */
> > +
> > +unsigned long __read_mostly cpu_khz;  /* CPU clock frequency in
> > kHz. */
> > +
> > +s_time_t get_s_time(void)
> > +{
> > +    BUG();
> > +}
> > +
> > +int reprogram_timer(s_time_t timeout)
> > +{
> > +    BUG();
> > +}
> > +
> > +void send_timer_event(struct vcpu *v)
> > +{
> > +    BUG();
> > +}
> > +
> > +void domain_set_time_offset(struct domain *d, int64_t
> > time_offset_seconds)
> > +{
> > +    BUG();
> > +}
> > +
> > +/* shutdown.c */
> > +
> > +void machine_restart(unsigned int delay_millisecs)
> > +{
> > +    BUG();
> > +}
> > +
> > +void machine_halt(void)
> > +{
> > +    BUG();
> > +}
> > +
> > +/* vm_event.c */
> > +
> > +void vm_event_fill_regs(vm_event_request_t *req)
> > +{
> > +    BUG();
> > +}
> > +
> > +void vm_event_set_registers(struct vcpu *v, vm_event_response_t
> > *rsp)
> > +{
> > +    BUG();
> > +}
> > +
> > +void vm_event_monitor_next_interrupt(struct vcpu *v)
> > +{
> > +    /* Not supported on RISCV. */
> > +}
> > +
> > +void vm_event_reset_vmtrace(struct vcpu *v)
> > +{
> > +    /* Not supported on RISCV. */
> > +}
> > +
> > +/* domctl.c */
> > +
> > +long arch_do_domctl(struct xen_domctl *domctl, struct domain *d,
> > +                    XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl)
> > +{
> > +    BUG();
> > +}
> > +
> > +void arch_get_domain_info(const struct domain *d,
> > +                          struct xen_domctl_getdomaininfo *info)
> > +{
> > +    BUG();
> > +}
> > +
> > +void arch_get_info_guest(struct vcpu *v, vcpu_guest_context_u c)
> > +{
> > +    BUG();
> > +}
> > +
> > +/* monitor.c */
> > +
> > +int arch_monitor_domctl_event(struct domain *d,
> > +                              struct xen_domctl_monitor_op *mop)
> > +{
> > +    BUG();
> > +}
> > +
> > +/* smp.c */
> > +
> > +void arch_flush_tlb_mask(const cpumask_t *mask)
> > +{
> > +    BUG();
> > +}
> > +
> > +void smp_send_event_check_mask(const cpumask_t *mask)
> > +{
> > +    BUG();
> > +}
> > +
> > +void smp_send_call_function_mask(const cpumask_t *mask)
> > +{
> > +    BUG();
> > +}
> > +
> > +/* irq.c */
> > +
> > +struct pirq *alloc_pirq_struct(struct domain *d)
> > +{
> > +    BUG();
> > +}
> > +
> > +int pirq_guest_bind(struct vcpu *v, struct pirq *pirq, int
> > will_share)
> > +{
> > +    BUG();
> > +}
> > +
> > +void pirq_guest_unbind(struct domain *d, struct pirq *pirq)
> > +{
> > +    BUG();
> > +}
> > +
> > +void pirq_set_affinity(struct domain *d, int pirq, const cpumask_t
> > *mask)
> > +{
> > +    BUG();
> > +}
> > +
> > +static void ack_none(struct irq_desc *irq)
> > +{
> > +    BUG();
> > +}
> > +
> > +static void end_none(struct irq_desc *irq)
> > +{
> > +    BUG();
> > +}
> 
> Much like I said for PPC - I don't think you need the two, as ...
> 
> > +hw_irq_controller no_irq_type = {
> > +    .typename = "none",
> > +    .startup = irq_startup_none,
> > +    .shutdown = irq_shutdown_none,
> > +    .enable = irq_enable_none,
> > +    .disable = irq_disable_none,
> > +    .ack = ack_none,
> > +    .end = end_none
> 
> ... there's nothing right now to invoke these hooks.
They really can be dropped.
I'll take into account that.

Thanks.


~ Oleksii

Re: [PATCH v2 07/39] xen/riscv: introduce arch-riscv/hvm/save.h

[Oleksii]

On Tue, 2023-12-05 at 16:59 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > --- /dev/null
> > +++ b/xen/include/public/arch-riscv/hvm/save.h
> > @@ -0,0 +1,20 @@
> > +/* SPDX-License-Identifier: MIT */
> > +/*
> > + * Structure definitions for HVM state that is held by Xen and
> > must
> > + * be saved along with the domain's memory and device-model state.
> > + */
> > +
> > +#ifndef __XEN_PUBLIC_HVM_SAVE_RISCV_H__
> > +#define __XEN_PUBLIC_HVM_SAVE_RISCV_H__
> > +
> > +#endif
> > +
> > +/*
> > + * Local variables:
> > + * mode: C
> > + * c-file-style: "BSD"
> > + * c-basic-offset: 4
> > + * tab-width: 4
> > + * indent-tabs-mode: nil
> > + * End:
> > + */
> 
> Seeing that Arm's is as empty, I wonder why we have it. Julien,
> Stefano?
It seems to exist to satisfy the 'install-tools-public-headers' target:
install: cannot stat 'xen/arch-arm/hvm/*.h': No such file or directory
Makefile:58: recipe for target 'install' failed
make[1]: *** [install] Error 1
make[1]: Leaving directory '/builds/xen-
project/people/olkur/xen/tools/include'
Makefile:44: recipe for target 'install-tools-public-headers' failed

From tools/include/Makefile:
install: all
...
$(DESTDIR)$(includedir)/xen/arch-arm
	$(INSTALL_DATA) xen/arch-arm/hvm/*.h
$(DESTDIR)$(includedir)/xen/arch-arm/hvm
...

We have the following options:
1. Remove the line with $(INSTALL_DATA) xen/arch-arm/hvm/*.h (only
save.h is now in this folder, which is empty).
2. Don't touch the Arm part, but for PPC and RISC-V, do the following:
#if defined(__i386__) || defined(__x86_64__)
#include "../arch-x86/hvm/save.h"
#elif defined(__arm__) || defined(__aarch64__)
#include "../arch-arm/hvm/save.h"
+#elif defined(__powerpc64__) || defined(__riscv)
+/* no specific header to include */
#else
#error "unsupported architecture"
#endif

3. Provide an asm-generic version of save.h for Arm, PPC, and RISC-V
and use it in public/save.h.

In your opinion, which option would be better?

~ Oleksii
> ....
> > --- a/xen/include/public/hvm/save.h
> > +++ b/xen/include/public/hvm/save.h
> > @@ -91,6 +91,8 @@ DECLARE_HVM_SAVE_TYPE(END, 0, struct
> > hvm_save_end);
> >  #include "../arch-arm/hvm/save.h"
> >  #elif defined(__powerpc64__)
> >  #include "../arch-ppc.h"
> > +#elif defined(__riscv)
> > +#include "../arch-riscv/hvm/save.h"
> >  #else
> >  #error "unsupported architecture"
> >  #endif
> 
> The PPC part here looks bogus altogether. Shawn?
> 
> Jan

Re: [PATCH v2 07/39] xen/riscv: introduce arch-riscv/hvm/save.h

[Jan Beulich]

On 20.12.2023 21:05, Oleksii wrote:
> On Tue, 2023-12-05 at 16:59 +0100, Jan Beulich wrote:
>> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>>> --- /dev/null
>>> +++ b/xen/include/public/arch-riscv/hvm/save.h
>>> @@ -0,0 +1,20 @@
>>> +/* SPDX-License-Identifier: MIT */
>>> +/*
>>> + * Structure definitions for HVM state that is held by Xen and
>>> must
>>> + * be saved along with the domain's memory and device-model state.
>>> + */
>>> +
>>> +#ifndef __XEN_PUBLIC_HVM_SAVE_RISCV_H__
>>> +#define __XEN_PUBLIC_HVM_SAVE_RISCV_H__
>>> +
>>> +#endif
>>> +
>>> +/*
>>> + * Local variables:
>>> + * mode: C
>>> + * c-file-style: "BSD"
>>> + * c-basic-offset: 4
>>> + * tab-width: 4
>>> + * indent-tabs-mode: nil
>>> + * End:
>>> + */
>>
>> Seeing that Arm's is as empty, I wonder why we have it. Julien,
>> Stefano?

I'm still curious about the reason here, but ...

> It seems to exist to satisfy the 'install-tools-public-headers' target:
> install: cannot stat 'xen/arch-arm/hvm/*.h': No such file or directory
> Makefile:58: recipe for target 'install' failed
> make[1]: *** [install] Error 1
> make[1]: Leaving directory '/builds/xen-
> project/people/olkur/xen/tools/include'
> Makefile:44: recipe for target 'install-tools-public-headers' failed
> 
> From tools/include/Makefile:
> install: all
> ...
> $(DESTDIR)$(includedir)/xen/arch-arm
> 	$(INSTALL_DATA) xen/arch-arm/hvm/*.h
> $(DESTDIR)$(includedir)/xen/arch-arm/hvm
> ...
> 
> We have the following options:
> 1. Remove the line with $(INSTALL_DATA) xen/arch-arm/hvm/*.h (only
> save.h is now in this folder, which is empty).

... we can't easily remove any existing public header. We can only try to
avoid making the same mistake (even if it's just a minor one) again.

> 2. Don't touch the Arm part, but for PPC and RISC-V, do the following:
> #if defined(__i386__) || defined(__x86_64__)
> #include "../arch-x86/hvm/save.h"
> #elif defined(__arm__) || defined(__aarch64__)
> #include "../arch-arm/hvm/save.h"
> +#elif defined(__powerpc64__) || defined(__riscv)
> +/* no specific header to include */
> #else
> #error "unsupported architecture"
> #endif

Yes. Still awaiting Shawn's input here as well, though.

> 3. Provide an asm-generic version of save.h for Arm, PPC, and RISC-V
> and use it in public/save.h.

That's not an option imo - what's under public/ needs to be self-contained.
Stuff there isn't supposed to even know of asm-generic/.

Jan

Re: [PATCH v2 37/39] xen/rirscv: add minimal amount of stubs to build full Xen

[Jan Beulich]

On 20.12.2023 13:55, Oleksii wrote:
> On Mon, 2023-12-18 at 18:00 +0100, Jan Beulich wrote:
>> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>>> --- /dev/null
>>> +++ b/xen/arch/riscv/stubs.c
>>> @@ -0,0 +1,426 @@
>>> +/* SPDX-License-Identifier: GPL-2.0-only */
>>> +#include <xen/cpumask.h>
>>> +#include <xen/domain.h>
>>> +#include <xen/irq.h>
>>> +#include <xen/nodemask.h>
>>> +#include <xen/time.h>
>>> +#include <public/domctl.h>
>>> +#include <public/vm_event.h>
>>
>> I think I can see why you need the former of these last two, but do
>> you
>> really need the latter?
> It is needed for vm_event_request_t and vm_event_response_t, but if use
> a forward declaration that it won't be needed:
> 
> typedef struct vm_event_st vm_event_request_t;
> typedef struct vm_event_st vm_event_response_t;

Iirc Misra wouldn't like the duplicating of typedef-s used elsewhere. But
as long as that's not going to stay (and I expect stubs.c to go away
before Misra becomes of concern for RISC-V), that's going to be okay, I
think. Yet then avoiding the typedef-s and using struct vm_event_st
directly in the functions would be as good, and overall less code.

Jan

Re: [PATCH v2 07/39] xen/riscv: introduce arch-riscv/hvm/save.h

[Oleksii]

On Thu, 2023-12-21 at 08:58 +0100, Jan Beulich wrote:
> On 20.12.2023 21:05, Oleksii wrote:
> > On Tue, 2023-12-05 at 16:59 +0100, Jan Beulich wrote:
> > > On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > > > --- /dev/null
> > > > +++ b/xen/include/public/arch-riscv/hvm/save.h
> > > > @@ -0,0 +1,20 @@
> > > > +/* SPDX-License-Identifier: MIT */
> > > > +/*
> > > > + * Structure definitions for HVM state that is held by Xen and
> > > > must
> > > > + * be saved along with the domain's memory and device-model
> > > > state.
> > > > + */
> > > > +
> > > > +#ifndef __XEN_PUBLIC_HVM_SAVE_RISCV_H__
> > > > +#define __XEN_PUBLIC_HVM_SAVE_RISCV_H__
> > > > +
> > > > +#endif
> > > > +
> > > > +/*
> > > > + * Local variables:
> > > > + * mode: C
> > > > + * c-file-style: "BSD"
> > > > + * c-basic-offset: 4
> > > > + * tab-width: 4
> > > > + * indent-tabs-mode: nil
> > > > + * End:
> > > > + */
> > > 
> > > Seeing that Arm's is as empty, I wonder why we have it. Julien,
> > > Stefano?
> 
> I'm still curious about the reason here, but ...
> 
> > It seems to exist to satisfy the 'install-tools-public-headers'
> > target:
> > install: cannot stat 'xen/arch-arm/hvm/*.h': No such file or
> > directory
> > Makefile:58: recipe for target 'install' failed
> > make[1]: *** [install] Error 1
> > make[1]: Leaving directory '/builds/xen-
> > project/people/olkur/xen/tools/include'
> > Makefile:44: recipe for target 'install-tools-public-headers'
> > failed
> > 
> > From tools/include/Makefile:
> > install: all
> > ...
> > $(DESTDIR)$(includedir)/xen/arch-arm
> > 	$(INSTALL_DATA) xen/arch-arm/hvm/*.h
> > $(DESTDIR)$(includedir)/xen/arch-arm/hvm
> > ...
> > 
> > We have the following options:
> > 1. Remove the line with $(INSTALL_DATA) xen/arch-arm/hvm/*.h (only
> > save.h is now in this folder, which is empty).
> 
> ... we can't easily remove any existing public header. We can only
> try to
> avoid making the same mistake (even if it's just a minor one) again.
> 
> > 2. Don't touch the Arm part, but for PPC and RISC-V, do the
> > following:
> > #if defined(__i386__) || defined(__x86_64__)
> > #include "../arch-x86/hvm/save.h"
> > #elif defined(__arm__) || defined(__aarch64__)
> > #include "../arch-arm/hvm/save.h"
> > +#elif defined(__powerpc64__) || defined(__riscv)
> > +/* no specific header to include */
> > #else
> > #error "unsupported architecture"
> > #endif
> 
> Yes. Still awaiting Shawn's input here as well, though.
Perhaps you missed the email from Shawn:
https://lore.kernel.org/xen-devel/c2f3280e-2208-496b-a0b5-fda1a2076b3a@raptorengineering.com/

> 
> > 3. Provide an asm-generic version of save.h for Arm, PPC, and RISC-
> > V
> > and use it in public/save.h.
> 
> That's not an option imo - what's under public/ needs to be self-
> contained.
> Stuff there isn't supposed to even know of asm-generic/.
In this case, this is not an option.

~ Oleksii

Re: [PATCH v2 17/39] xen/riscv: introduce asm/atomic.h

[Oleksii]

On Thu, 2023-12-07 at 16:57 +0100, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > From: Bobby Eshleman <bobbyeshleman@gmail.com>
> > 
> > Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> > ---
> > Changes in V2:
> >  - Change an author of commit. I got this header from Bobby's old
> > repo.
> 
> Not sure how to deal with that when there's not also an S-o-b.
> 
> > --- /dev/null
> > +++ b/xen/arch/riscv/include/asm/atomic.h
> > @@ -0,0 +1,375 @@
> > +/* SPDX-License-Identifier: GPL-2.0-or-later */
> > +/*
> > + * Taken and modified from Linux.
> > + * 
> > + * Copyright (C) 2007 Red Hat, Inc. All Rights Reserved.
> > + * Copyright (C) 2012 Regents of the University of California
> > + * Copyright (C) 2017 SiFive
> > + * Copyright (C) 2021 Vates SAS
> > + */
> > +
> > +#ifndef _ASM_RISCV_ATOMIC_H
> > +#define _ASM_RISCV_ATOMIC_H
> > +
> > +#include <xen/atomic.h>
> > +#include <asm/cmpxchg.h>
> 
> This and ...
> 
> > +#include <asm/fence.h>
> > +#include <asm/io.h>
> 
> .. this header are only introduced later. Bad ordering of the series?
Yes, bad ordering. I'll re-order this patch
Thanks.
> 
> > +#include <asm/system.h>
> > +
> > +void __bad_atomic_size(void);
> > +
> > +static always_inline void read_atomic_size(const volatile void *p,
> > +                                           void *res,
> > +                                           unsigned int size)
> > +{
> > +    switch ( size ) {
> 
> Nit (style): Brace on its own line (again further down).
> 
> > +    case 1: *(uint8_t *)res = readb((uint8_t *)p); break;
> > +    case 2: *(uint16_t *)res = readw((uint16_t *)p); break;
> > +    case 4: *(uint32_t *)res = readl((uint32_t *)p); break;
> > +    case 8: *(uint32_t *)res  = readq((uint64_t *)p); break;
> 
> Please don't cast away const-ness.
> 
> > +    default: __bad_atomic_size(); break;
> > +    }
> > +}
> > +
> > +#define read_atomic(p)
> > ({                                               \
> > +    union { typeof(*p) val; char c[0]; }
> > x_;                            \
> 
> Hmm, you avoid leading underscores here, but then ...
> 
> > +    read_atomic_size(p, x_.c,
> > sizeof(*p));                              \
> > +   
> > x_.val;                                                            
> > \
> > +})
> > +
> > +
> > +#define write_atomic(p, x)
> > ({                                           \
> > +    typeof(*p) __x =
> > (x);                                               \
> 
> ... they're still there here.
I'll rename __x to x__. Thanks.
> 
> > +    switch ( sizeof(*p) )
> > {                                             \
> > +    case 1: writeb((uint8_t)__x,  (uint8_t *)  p);
> > break;              \
> > +    case 2: writew((uint16_t)__x, (uint16_t *) p);
> > break;              \
> > +    case 4: writel((uint32_t)__x, (uint32_t *) p);
> > break;              \
> > +    case 8: writeq((uint64_t)__x, (uint64_t *) p);
> > break;              \
> > +    default: __bad_atomic_size();
> > break;                                \
> > +   
> > }                                                                  
> > \
> > +   
> > __x;                                                               
> > \
> > +})
> > +
> > +/* TODO: Fix this */
> > +#define add_sized(p, x)
> > ({                                              \
> > +    typeof(*(p)) __x =
> > (x);                                             \
> > +    switch ( sizeof(*(p))
> > )                                             \
> > +   
> > {                                                                  
> > \
> > +    case 1: writeb(read_atomic(p) + __x, (uint8_t *)(p));
> > break;        \
> > +    case 2: writew(read_atomic(p) + __x, (uint16_t *)(p));
> > break;       \
> > +    case 4: writel(read_atomic(p) + __x, (uint32_t *)(p));
> > break;       \
> 
> Instead of this, considering the comment perhaps better just BUG()?
I think this TODO can be removed. Macros looks fine to me.

> 
> > +    default: __bad_atomic_size();
> > break;                                \
> > +   
> > }                                                                  
> > \
> > +})
> > +
> > +/*
> > + *  __unqual_scalar_typeof(x) - Declare an unqualified scalar
> > type, leaving
> > + *               non-scalar types unchanged.
> > + *
> > + * Prefer C11 _Generic for better compile-times and simpler code.
> > Note: 'char'
> > + * is not type-compatible with 'signed char', and we define a
> > separate case.
> > + */
> > +#define __scalar_type_to_expr_cases(type)               \
> > +    unsigned type:  (unsigned type)0,                   \
> > +    signed type:    (signed type)0
> > +
> > +#define __unqual_scalar_typeof(x) typeof(               \
> > +    _Generic((x),                                       \
> 
> I think you still owe us an update to ./README, clarifying what
> compiler versions
> may be used for building RISC-V. Unless of course all that exist
> support _Generic
> (which then would be nice to say in the description).
I'll provide a separate patch.

> 
> > +        char:  (char)0,                                 \
> > +        __scalar_type_to_expr_cases(char),              \
> > +        __scalar_type_to_expr_cases(short),             \
> > +        __scalar_type_to_expr_cases(int),               \
> > +        __scalar_type_to_expr_cases(long),              \
> > +        __scalar_type_to_expr_cases(long long),         \
> > +        default: (x)))
> > +
> > +#define READ_ONCE(x)  (*(const volatile __unqual_scalar_typeof(x)
> > *)&(x))
> > +#define WRITE_ONCE(x, val)                                      \
> > +    do {                                                        \
> > +            *(volatile typeof(x) *)&(x) = (val);                \
> 
> Nit (style): Too deep indentation.
> 
> > +    } while (0)
> > +
> > +#define
> > __atomic_acquire_fence()					\
> > +	__asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" :::
> > "memory")
> 
> Suddenly using tab indentation here and below? And missing blanks
> again.
Thanks. I'll fix  that.
> 
> Jan

Re: [PATCH v2 29/39] xen/riscv: add definition of __read_mostly

[Andrew Cooper]

On 12/12/2023 5:04 pm, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>> The definition of __read_mostly should be removed in:
>> https://lore.kernel.org/xen-devel/f25eb5c9-7c14-6e23-8535-2c66772b333e@suse.com/
> Andrew, can we settle on what to do with that patch? If you don't like me
> putting __read_mostly in xen/cache.h (consistent with __ro_after_init),
> would you please make an alternative suggestion?

xen/linkage.h?  xen/sections.h?

Sorry - I didn't mean to block it specifically, but I do think
xen/cache.h is the wrong place for both to live and that it's a small
enough change to warrant sorting out nicely once and for all.

~Andrew

Re: [PATCH v2 37/39] xen/rirscv: add minimal amount of stubs to build full Xen

[Oleksii]

On Thu, 2023-12-21 at 09:02 +0100, Jan Beulich wrote:
> On 20.12.2023 13:55, Oleksii wrote:
> > On Mon, 2023-12-18 at 18:00 +0100, Jan Beulich wrote:
> > > On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > > > --- /dev/null
> > > > +++ b/xen/arch/riscv/stubs.c
> > > > @@ -0,0 +1,426 @@
> > > > +/* SPDX-License-Identifier: GPL-2.0-only */
> > > > +#include <xen/cpumask.h>
> > > > +#include <xen/domain.h>
> > > > +#include <xen/irq.h>
> > > > +#include <xen/nodemask.h>
> > > > +#include <xen/time.h>
> > > > +#include <public/domctl.h>
> > > > +#include <public/vm_event.h>
> > > 
> > > I think I can see why you need the former of these last two, but
> > > do
> > > you
> > > really need the latter?
> > It is needed for vm_event_request_t and vm_event_response_t, but if
> > use
> > a forward declaration that it won't be needed:
> > 
> > typedef struct vm_event_st vm_event_request_t;
> > typedef struct vm_event_st vm_event_response_t;
> 
> Iirc Misra wouldn't like the duplicating of typedef-s used elsewhere.
> But
> as long as that's not going to stay (and I expect stubs.c to go away
> before Misra becomes of concern for RISC-V), that's going to be okay,
> I
> think. Yet then avoiding the typedef-s and using struct vm_event_st
> directly in the functions would be as good, and overall less code.
Hmm, it makes sense to use sturct vm_event_st direcly in this case.
Thanks for suggestion.

~ Oleksii

Re: [PATCH v2 30/39] xen/riscv: define an address of frame table

[Oleksii]

On Mon, 2023-12-18 at 12:22 +0100, Jan Beulich wrote:
> On 18.12.2023 11:36, Oleksii wrote:
> > On Thu, 2023-12-14 at 16:48 +0100, Jan Beulich wrote:
> > > On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > > > +#define SLOTN_ENTRY_SIZE        SLOTN(1)
> > > > +
> > > >  #define XEN_VIRT_START 0xFFFFFFFFC0000000 /* (_AC(-1, UL) + 1
> > > > -
> > > > GB(1)) */
> > > > +
> > > > +#define FRAMETABLE_VIRT_START   SLOTN(196)
> > > > +#define FRAMETABLE_SIZE         GB(3)
> > > > +#define FRAMETABLE_NR           (FRAMETABLE_SIZE /
> > > > sizeof(*frame_table))
> > > > +#define FRAMETABLE_VIRT_END     (FRAMETABLE_VIRT_START +
> > > > FRAMETABLE_SIZE - 1)
> > > > +
> > > > +#define VMAP_VIRT_START         SLOTN(194)
> > > > +#define VMAP_VIRT_SIZE          GB(1)
> > > 
> > > May I suggest that you keep these blocks sorted by slot number?
> > > Or
> > > wait,
> > > the layout comment further up is also in decreasing order, so
> > > that's
> > > fine here, but then can all of this please be moved next to the
> > > comment
> > > actually providing the necessary context (thus eliminating the
> > > need
> > > for
> > > new comments)?
> > Sure, I'll put this part close to layout comment.
> > 
> > >  You'll then also notice that the generalization here
> > > (keeping basically the same layout for e.g. SATP_MODE_SV48, just
> > > shifted
> > > by 9 bits) isn't in line with the comment there.
> > Does it make sense to add another one table with updated addresses
> > for
> > SATP_MODE_SV48?
> 
> Well, especially if you mean to support that mode, its layout surely
> wants writing down. I was hoping though that maybe you/we could get
> away
> without multiple tables, but e.g. use one having multiple columns.
I came up with the following but I am not sure that it is really
convient:
/*
 * RISC-V64 Layout:
 *
#if RV_STAGE1_MODE == SATP_MODE_SV39
 *
 * From the riscv-privileged doc:
 *   When mapping between narrower and wider addresses,
 *   RISC-V zero-extends a narrower physical address to a wider size.
 *   The mapping between 64-bit virtual addresses and the 39-bit usable
 *   address space of Sv39 is not based on zero-extension but instead
 *   follows an entrenched convention that allows an OS to use one or
 *   a few of the most-significant bits of a full-size (64-bit) virtual
 *   address to quickly distinguish user and supervisor address
regions.
 *
 * It means that:
 *   top VA bits are simply ignored for the purpose of translating to
PA.
#endif
 *
 *       SATP_MODE_SV32   SATP_MODE_SV39   SATP_MODE_SV48  
SATP_MODE_SV57
 *     ----------------------------------------------------------------
-----------
 * BA0 | FFFFFFFFFFE00000 | FFFFFFFFC0000000 | FFFFFF8000000000 |
FFFF000000000000
 * BA1 | 0000000019000000 | 0000003200000000 | 0000640000000000 |
00C8000000000000
 * BA2 | 0000000018800000 | 0000003100000000 | 0000620000000000 |
00C4000000000000
 * BA3 | 0000000018400000 | 0000003080000000 | 0000610000000000 |
00C2000000000000
 * 
 *
=======================================================================
=====
 * Start addr    |   End addr           |  Size  | Slot       |area
description
 *
=======================================================================
=====
 * BA0 + 0x800000 |  FFFFFFFFFFFFFFFF   |1016 MB |
L${HYP_PT_ROOT_LEVEL} 511     | Unused
 * BA0 + 0x400000 |  BA0 + 0x800000     |  2 MB  |
L${HYP_PT_ROOT_LEVEL} 511     | Fixmap
 * BA0 + 0x200000 |  BA0 + 0x400000     |  4 MB  |
L${HYP_PT_ROOT_LEVEL} 511     | FDT
 * BA0            |  BA0 + 0x200000     |  2 MB  |
L${HYP_PT_ROOT_LEVEL} 511     | Xen
 *                 ...                  |  1 GB  |
L${HYP_PT_ROOT_LEVEL} 510     | Unused
 * BA1 + 0x000000 |  BA1 + 0x4D80000000 | 309 GB |
L${HYP_PT_ROOT_LEVEL} 200-509 | Direct map
 *                 ...                  |  1 GB  |
L${HYP_PT_ROOT_LEVEL} 199     | Unused
 * BA2 + 0x000000 |  BA2 + 0xC0000000   |  3 GB  |
L${HYP_PT_ROOT_LEVEL} 196-198 | Frametable
 *                 ...                  |  1 GB  |
L${HYP_PT_ROOT_LEVEL} 195     | Unused
 * BA3 + 0x000000 |  BA3 + 0x40000000   |  1 GB  |
L${HYP_PT_ROOT_LEVEL} 194     | VMAP
 *                 ...                  | 194 GB |
L${HYP_PT_ROOT_LEVEL} 0 - 193 | Unused
 *
=======================================================================
=====
 */

Do you have better ideas?

Thanks in advamce.

~ Oleksii
> 
> 
> > Microchip has h/w which requires SATP_MODE_SV48 ( at least ), so I
> > have
> > a patch which introduces SATP_MODE_SV48 and I planned to update the
> > layout table in this patch.
> 
> 
> 

Re: [PATCH v2 30/39] xen/riscv: define an address of frame table

[Jan Beulich]

On 21.12.2023 20:59, Oleksii wrote:
> On Mon, 2023-12-18 at 12:22 +0100, Jan Beulich wrote:
>> On 18.12.2023 11:36, Oleksii wrote:
>>> On Thu, 2023-12-14 at 16:48 +0100, Jan Beulich wrote:
>>>> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>>>>> +#define SLOTN_ENTRY_SIZE        SLOTN(1)
>>>>> +
>>>>>  #define XEN_VIRT_START 0xFFFFFFFFC0000000 /* (_AC(-1, UL) + 1
>>>>> -
>>>>> GB(1)) */
>>>>> +
>>>>> +#define FRAMETABLE_VIRT_START   SLOTN(196)
>>>>> +#define FRAMETABLE_SIZE         GB(3)
>>>>> +#define FRAMETABLE_NR           (FRAMETABLE_SIZE /
>>>>> sizeof(*frame_table))
>>>>> +#define FRAMETABLE_VIRT_END     (FRAMETABLE_VIRT_START +
>>>>> FRAMETABLE_SIZE - 1)
>>>>> +
>>>>> +#define VMAP_VIRT_START         SLOTN(194)
>>>>> +#define VMAP_VIRT_SIZE          GB(1)
>>>>
>>>> May I suggest that you keep these blocks sorted by slot number?
>>>> Or
>>>> wait,
>>>> the layout comment further up is also in decreasing order, so
>>>> that's
>>>> fine here, but then can all of this please be moved next to the
>>>> comment
>>>> actually providing the necessary context (thus eliminating the
>>>> need
>>>> for
>>>> new comments)?
>>> Sure, I'll put this part close to layout comment.
>>>
>>>>  You'll then also notice that the generalization here
>>>> (keeping basically the same layout for e.g. SATP_MODE_SV48, just
>>>> shifted
>>>> by 9 bits) isn't in line with the comment there.
>>> Does it make sense to add another one table with updated addresses
>>> for
>>> SATP_MODE_SV48?
>>
>> Well, especially if you mean to support that mode, its layout surely
>> wants writing down. I was hoping though that maybe you/we could get
>> away
>> without multiple tables, but e.g. use one having multiple columns.
> I came up with the following but I am not sure that it is really
> convient:
> /*
>  * RISC-V64 Layout:
>  *
> #if RV_STAGE1_MODE == SATP_MODE_SV39
>  *
>  * From the riscv-privileged doc:
>  *   When mapping between narrower and wider addresses,
>  *   RISC-V zero-extends a narrower physical address to a wider size.
>  *   The mapping between 64-bit virtual addresses and the 39-bit usable
>  *   address space of Sv39 is not based on zero-extension but instead
>  *   follows an entrenched convention that allows an OS to use one or
>  *   a few of the most-significant bits of a full-size (64-bit) virtual
>  *   address to quickly distinguish user and supervisor address
> regions.
>  *
>  * It means that:
>  *   top VA bits are simply ignored for the purpose of translating to
> PA.
> #endif
>  *
>  *       SATP_MODE_SV32   SATP_MODE_SV39   SATP_MODE_SV48  
> SATP_MODE_SV57
>  *     ----------------------------------------------------------------
> -----------
>  * BA0 | FFFFFFFFFFE00000 | FFFFFFFFC0000000 | FFFFFF8000000000 |
> FFFF000000000000
>  * BA1 | 0000000019000000 | 0000003200000000 | 0000640000000000 |
> 00C8000000000000
>  * BA2 | 0000000018800000 | 0000003100000000 | 0000620000000000 |
> 00C4000000000000
>  * BA3 | 0000000018400000 | 0000003080000000 | 0000610000000000 |
> 00C2000000000000
>  * 
>  *
> =======================================================================
> =====
>  * Start addr    |   End addr           |  Size  | Slot       |area
> description
>  *
> =======================================================================
> =====
>  * BA0 + 0x800000 |  FFFFFFFFFFFFFFFF   |1016 MB |
> L${HYP_PT_ROOT_LEVEL} 511     | Unused
>  * BA0 + 0x400000 |  BA0 + 0x800000     |  2 MB  |
> L${HYP_PT_ROOT_LEVEL} 511     | Fixmap
>  * BA0 + 0x200000 |  BA0 + 0x400000     |  4 MB  |
> L${HYP_PT_ROOT_LEVEL} 511     | FDT
>  * BA0            |  BA0 + 0x200000     |  2 MB  |
> L${HYP_PT_ROOT_LEVEL} 511     | Xen
>  *                 ...                  |  1 GB  |
> L${HYP_PT_ROOT_LEVEL} 510     | Unused
>  * BA1 + 0x000000 |  BA1 + 0x4D80000000 | 309 GB |
> L${HYP_PT_ROOT_LEVEL} 200-509 | Direct map
>  *                 ...                  |  1 GB  |
> L${HYP_PT_ROOT_LEVEL} 199     | Unused
>  * BA2 + 0x000000 |  BA2 + 0xC0000000   |  3 GB  |
> L${HYP_PT_ROOT_LEVEL} 196-198 | Frametable
>  *                 ...                  |  1 GB  |
> L${HYP_PT_ROOT_LEVEL} 195     | Unused
>  * BA3 + 0x000000 |  BA3 + 0x40000000   |  1 GB  |
> L${HYP_PT_ROOT_LEVEL} 194     | VMAP
>  *                 ...                  | 194 GB |
> L${HYP_PT_ROOT_LEVEL} 0 - 193 | Unused
>  *
> =======================================================================
> =====
>  */
> 
> Do you have better ideas?

It doesn't look too bad imo, at the first glance, albeit the line
wrapping damage of course makes it a little hard to look at. In the
last table with all lines saying L${HYP_PT_ROOT_LEVEL}, perhaps that
could be put in the table heading (instead of "Slot" say e.g. "Root
PT slot")?

Jan

Re: [PATCH v2 30/39] xen/riscv: define an address of frame table

[Oleksii]

On Fri, 2023-12-22 at 09:08 +0100, Jan Beulich wrote:
> On 21.12.2023 20:59, Oleksii wrote:
> > On Mon, 2023-12-18 at 12:22 +0100, Jan Beulich wrote:
> > > On 18.12.2023 11:36, Oleksii wrote:
> > > > On Thu, 2023-12-14 at 16:48 +0100, Jan Beulich wrote:
> > > > > On 24.11.2023 11:30, Oleksii Kurochko wrote:
> > > > > > +#define SLOTN_ENTRY_SIZE        SLOTN(1)
> > > > > > +
> > > > > >  #define XEN_VIRT_START 0xFFFFFFFFC0000000 /* (_AC(-1, UL)
> > > > > > + 1
> > > > > > -
> > > > > > GB(1)) */
> > > > > > +
> > > > > > +#define FRAMETABLE_VIRT_START   SLOTN(196)
> > > > > > +#define FRAMETABLE_SIZE         GB(3)
> > > > > > +#define FRAMETABLE_NR           (FRAMETABLE_SIZE /
> > > > > > sizeof(*frame_table))
> > > > > > +#define FRAMETABLE_VIRT_END     (FRAMETABLE_VIRT_START +
> > > > > > FRAMETABLE_SIZE - 1)
> > > > > > +
> > > > > > +#define VMAP_VIRT_START         SLOTN(194)
> > > > > > +#define VMAP_VIRT_SIZE          GB(1)
> > > > > 
> > > > > May I suggest that you keep these blocks sorted by slot
> > > > > number?
> > > > > Or
> > > > > wait,
> > > > > the layout comment further up is also in decreasing order, so
> > > > > that's
> > > > > fine here, but then can all of this please be moved next to
> > > > > the
> > > > > comment
> > > > > actually providing the necessary context (thus eliminating
> > > > > the
> > > > > need
> > > > > for
> > > > > new comments)?
> > > > Sure, I'll put this part close to layout comment.
> > > > 
> > > > >  You'll then also notice that the generalization here
> > > > > (keeping basically the same layout for e.g. SATP_MODE_SV48,
> > > > > just
> > > > > shifted
> > > > > by 9 bits) isn't in line with the comment there.
> > > > Does it make sense to add another one table with updated
> > > > addresses
> > > > for
> > > > SATP_MODE_SV48?
> > > 
> > > Well, especially if you mean to support that mode, its layout
> > > surely
> > > wants writing down. I was hoping though that maybe you/we could
> > > get
> > > away
> > > without multiple tables, but e.g. use one having multiple
> > > columns.
> > I came up with the following but I am not sure that it is really
> > convient:
> > /*
> >  * RISC-V64 Layout:
> >  *
> > #if RV_STAGE1_MODE == SATP_MODE_SV39
> >  *
> >  * From the riscv-privileged doc:
> >  *   When mapping between narrower and wider addresses,
> >  *   RISC-V zero-extends a narrower physical address to a wider
> > size.
> >  *   The mapping between 64-bit virtual addresses and the 39-bit
> > usable
> >  *   address space of Sv39 is not based on zero-extension but
> > instead
> >  *   follows an entrenched convention that allows an OS to use one
> > or
> >  *   a few of the most-significant bits of a full-size (64-bit)
> > virtual
> >  *   address to quickly distinguish user and supervisor address
> > regions.
> >  *
> >  * It means that:
> >  *   top VA bits are simply ignored for the purpose of translating
> > to
> > PA.
> > #endif
> >  *
> >  *       SATP_MODE_SV32   SATP_MODE_SV39   SATP_MODE_SV48  
> > SATP_MODE_SV57
> >  *     ------------------------------------------------------------
> > ----
> > -----------
> >  * BA0 | FFFFFFFFFFE00000 | FFFFFFFFC0000000 | FFFFFF8000000000 |
> > FFFF000000000000
> >  * BA1 | 0000000019000000 | 0000003200000000 | 0000640000000000 |
> > 00C8000000000000
> >  * BA2 | 0000000018800000 | 0000003100000000 | 0000620000000000 |
> > 00C4000000000000
> >  * BA3 | 0000000018400000 | 0000003080000000 | 0000610000000000 |
> > 00C2000000000000
> >  * 
> >  *
> > ===================================================================
> > ====
> > =====
> >  * Start addr    |   End addr           |  Size  | Slot       |area
> > description
> >  *
> > ===================================================================
> > ====
> > =====
> >  * BA0 + 0x800000 |  FFFFFFFFFFFFFFFF   |1016 MB |
> > L${HYP_PT_ROOT_LEVEL} 511     | Unused
> >  * BA0 + 0x400000 |  BA0 + 0x800000     |  2 MB  |
> > L${HYP_PT_ROOT_LEVEL} 511     | Fixmap
> >  * BA0 + 0x200000 |  BA0 + 0x400000     |  4 MB  |
> > L${HYP_PT_ROOT_LEVEL} 511     | FDT
> >  * BA0            |  BA0 + 0x200000     |  2 MB  |
> > L${HYP_PT_ROOT_LEVEL} 511     | Xen
> >  *                 ...                  |  1 GB  |
> > L${HYP_PT_ROOT_LEVEL} 510     | Unused
> >  * BA1 + 0x000000 |  BA1 + 0x4D80000000 | 309 GB |
> > L${HYP_PT_ROOT_LEVEL} 200-509 | Direct map
> >  *                 ...                  |  1 GB  |
> > L${HYP_PT_ROOT_LEVEL} 199     | Unused
> >  * BA2 + 0x000000 |  BA2 + 0xC0000000   |  3 GB  |
> > L${HYP_PT_ROOT_LEVEL} 196-198 | Frametable
> >  *                 ...                  |  1 GB  |
> > L${HYP_PT_ROOT_LEVEL} 195     | Unused
> >  * BA3 + 0x000000 |  BA3 + 0x40000000   |  1 GB  |
> > L${HYP_PT_ROOT_LEVEL} 194     | VMAP
> >  *                 ...                  | 194 GB |
> > L${HYP_PT_ROOT_LEVEL} 0 - 193 | Unused
> >  *
> > ===================================================================
> > ====
> > =====
> >  */
> > 
> > Do you have better ideas?
> 
> It doesn't look too bad imo, at the first glance, albeit the line
> wrapping damage of course makes it a little hard to look at. In the
> last table with all lines saying L${HYP_PT_ROOT_LEVEL}, perhaps that
> could be put in the table heading (instead of "Slot" say e.g. "Root
> PT slot")?
Thanks for the remark. It would be definitely better.

~ Oleksii

Re: [PATCH v2 29/39] xen/riscv: add definition of __read_mostly

[Jan Beulich]

On 21.12.2023 16:23, Andrew Cooper wrote:
> On 12/12/2023 5:04 pm, Jan Beulich wrote:
>> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>>> The definition of __read_mostly should be removed in:
>>> https://lore.kernel.org/xen-devel/f25eb5c9-7c14-6e23-8535-2c66772b333e@suse.com/
>> Andrew, can we settle on what to do with that patch? If you don't like me
>> putting __read_mostly in xen/cache.h (consistent with __ro_after_init),
>> would you please make an alternative suggestion?
> 
> xen/linkage.h?  xen/sections.h?

Well, that's the problem: There's no xen/sections.h (and I don't see why we'd
introduce one just for this), and while xen/linkage.h is about to appear for
the entry point annotations, I don't think it is a reasonable fit.

Otoh with __ro_after_init ...

> Sorry - I didn't mean to block it specifically, but I do think
> xen/cache.h is the wrong place for both to live and that it's a small
> enough change to warrant sorting out nicely once and for all.

... living in xen/cache.h already, __read_mostly is even more logical to put
there than __ro_after_init. For the latter I agree the purpose isn't really
cache related, while for the former it is. From your original reply to the
patch submission, I have to admit I don't really understand what
"micro-architectural" detail you mean: The goal of avoiding unnecessary cache
line ping-pong doesn't look to be what you mean, as imo that's a valid (and
generic) goal. And that's what I think is the reason the #define presently
lives in asm/cache.h, justifying my moving of it to xen/cache.h.

So if you can't get yourself to accept xen/cache.h as the new (even if only
temporary) location, I think you can be expected to make a better proposal.
With "better" there meaning you supplying a reason for why you think that
placement is better than xen/cache.h. For example, if I knew what you
expected xen/sections.h to further contain (in the long run), I might find
myself agreeing to that. Yet other section annotation #define-s live
elsewhere anyway, with - in particular - __init and friends imo not likely
to move out of their present header (xen/init.h).

Jan