[PATCH 00/27] ARM Scalable Vector Extension (SVE)

[PATCH 00/27] ARM Scalable Vector Extension (SVE)

[Dave Martin]

This series implements Linux kernel support for the ARM Scalable Vector
Extension (SVE). [1]  It supersedes the previous RFC: see [6] for link
and a summary of changes.

This series depends on some series that are headed for v4.14:
see [3], [4], [5].

To reduce spam, some people may not been copied on the entire series.
For those who did not receive the whole series, it can be found in the
linux-arm-kernel archive. [2]

*Note* The final two patches (26-27) of the series are still RFC --
before committing to this ABI it would be good to get feedback on
whether the approach makes sense and whether it suitable for other
architectures.  These two patches are not required by the rest of the
series and can be revised or merged later.


Support for use of SVE by KVM guests is not currently included.
Instead, such use will be trapped and reflected to the guest as
undefined instruction execution.  SVE is hidden from the view of the
CPU feature registers visible to guests, so that guests will not
expect it to work.


This series has been build- and boot-tested on Juno r0 and the ARM FVP
Base model with SVE plugin.  Because there is no hardware with SVE
support yet, testing of the SVE functionality has only been performed on
the model.

Regression testing using LTP is under way and has also been completed on
previous versions of this series.


Series summary:

 * Patches 1-5 contain some individual bits of preparatory spadework,
   which are indirectly related to SVE.

Dave Martin (5):
  regset: Add support for dynamically sized regsets
  arm64: KVM: Hide unsupported AArch64 CPU features from guests
  arm64: efi: Add missing Kconfig dependency on KERNEL_MODE_NEON
  arm64: Port deprecated instruction emulation to new sysctl interface
  arm64: fpsimd: Simplify uses of {set,clear}_ti_thread_flag()

   Non-trivial changes among these are:

   * Patch 1: updates the regset core code to handle regsets whose size
     is not fixed at compile time.  This avoids bloating coredumps even
     though the maximum theoretical SVE regset size is large.

   * Patch 2: extends KVM to modify the ARM architectural ID registers
     seen by guests, by trapping and emulating certain registers.  For
     SVE this is a temporary measure, but it may be useful for other
     architecture extensions.  This patch may also be built on in the
     future, since the only registers currently emulated are those
     required for hiding SVE.

 * Patches 6-10 add SVE-specific system register and structure layout
   definitions, and the low-level boot code and accessors needed for
   making use of SVE.

Dave Martin (5):
  arm64/sve: System register and exception syndrome definitions
  arm64/sve: Low-level SVE architectural state manipulation functions
  arm64/sve: Kconfig update and conditional compilation support
  arm64/sve: Signal frame and context structure definition
  arm64/sve: Low-level CPU setup

 * Patches 11-13 implement the core context management facilities to
   provide each user task with its own SVE register context, signal
   handling facilities, and sane programmer's model interoperation
   between SVE and FPSIMD.

Dave Martin (3):
  arm64/sve: Core task context handling
  arm64/sve: Support vector length resetting for new processes
  arm64/sve: Signal handling support

 * Patches 14-15 provide backend logic for detecting and making use of
   the different SVE vector lengths supported by the hardware.

Dave Martin (2):
  arm64/sve: Backend logic for setting the vector length
  arm64/sve: Probe SVE capabilities and usable vector lengths

 * Patches 16-17 update the kernel-mode NEON / EFI FPSIMD frameworks to
   interoperate correctly with SVE.

Dave Martin (2):
  arm64/sve: Preserve SVE registers around kernel-mode NEON use
  arm64/sve: Preserve SVE registers around EFI runtime service calls

 * Patches 18-20 implement the userspace frontend for managing SVE,
   comprising ptrace, some new arch-specific prctl() calls, and a new
   sysctl for init-time setup.

Dave Martin (3):
  arm64/sve: ptrace and ELF coredump support
  arm64/sve: Add prctl controls for userspace vector length management
  arm64/sve: Add sysctl to set the default vector length for new
    processes

 * Patches 21-23 provide stub KVM extensions for using KVM only on the
   host, while denying guest access.  (A future series will extend this
   with full support for SVE in guests.)

Dave Martin (3):
  arm64/sve: KVM: Prevent guests from using SVE
  arm64/sve: KVM: Treat guest SVE use as undefined instruction
    execution
  arm64/sve: KVM: Hide SVE from CPU features exposed to guests

And finally:

 * Patch 24 disengages the safety catch, enabling the kernel SVE runtime
   support and allowing userspace to use SVE.

Dave Martin (1):
  arm64/sve: Detect SVE and activate runtime support

 * Patch 25 adds some basic documentation.

Dave Martin (1):
  arm64/sve: Add documentation

 * Patches 26-27 (which may be considered RFC) propose a mechanism to
   report the maximum runtime signal frame size to userspace.

Dave Martin (2):
  arm64: signal: Report signal frame size to userspace via auxv
  arm64/sve: signal: Include SVE when computing AT_MINSIGSTKSZ


Refernces:

[1] ARM Scalable Vector Extension
https://community.arm.com/groups/processors/blog/2016/08/22/technology-update-the-scalable-vector-extension-sve-for-the-armv8-a-architecture

[2] linux-arm-kernel August 2017 Archives by thread
http://lists.infradead.org/pipermail/linux-arm-kernel/2017-August/thread.html

[3] [PATCH v2 REPOST 0/2] arm64: Abstract syscallno manipulation
http://lists.infradead.org/pipermail/linux-arm-kernel/2017-August/522736.html
Accepted by Catalin for v4.14.

[4] [PATCH 0/5] Simplify kernel-mode NEON
http://lists.infradead.org/pipermail/linux-arm-kernel/2017-August/523415.html
(Depends on [5].)
Accepted by Catalin for v4.14, pending upstream merge of [5].

[5] [PATCH resend 00/18] crypto: ARM/arm64 roundup for v4.14
http://lists.infradead.org/pipermail/linux-arm-kernel/2017-July/520664.html
Accepted by Herbert Xu for v4.14.

[6] [RFC PATCH v2 00/41] Scalable Vector Extension (SVE) core support
http://lists.infradead.org/pipermail/linux-arm-kernel/2017-March/495966.html

 * Account for the size of the frame link record in AT_MINSIGSTKSZ.

 * Give SVE insn generation macros more human-readable names.  Make SVE
   insn generation macro arg names more uniform and sligntly less
   cryptic.  Tidy up asm "for" loop macro.

 * Describe ID_AA64PFR0 SVE field in ftr_id_aa64pfr0[]: after upstream
   refactoring, this must be present to derive an ELF hwcap from the
   field.

 * Minor simplifications to head.S: promote code simplicity over
   avoiding MSRs, since this is a cold path.

 * Get rid of obsolete (and dodgy) sve_state() struct template macro.
   Explicit offset/size calculations are used instead.  The removal of
   variably-modified types also permits some trivial functions to be
   inlined in the source.

 * Adapt to simplified kernel-mode NEON / EFI FPSIMD framework.

 * ptrace: Flush regs back to thread_struct before dumping NT_ARM_SVE.
   This is currently redundant, since NT_PRFPREG happens to be dumped
   out first, and the flush done for that regset happens to flush the
   SVE regs too.  But this might change someday.  This isn't a hot path,
   so an extra flush isn't the end of the world.

 * PR_SVE_{SET,GET}_VL: Flags and vector length arguments for SET_VL
   merged (ABI change).  This makes saving and restoring the vector
   length and flags a good deal less painful.

 * PR_SVE_SET_VL: (And all other VL-setting interfaces): clamp VL to
   8192.  Some features of the SVE ISA won't work as expected if future
   arch revisions ever permit VLs above 8192, so we'll need userspace to
   request this explicitly.  This allows SET_VL(SVE_VL_MAX) to continue
   to work as expected today.

 * thread_struct.sve_flags member removed.  These flag(s) are made into
   thread flags instead.

 * VL_THREAD flag for VL setting (via ptrace/prctl) removed.  This is
   only protecting userspace from itself, so it's superfluous.  Instead,
   VL setting will always affect the current thread only.
   General-purpose code shouldn't be setting the VL in the first place.

 * TIF_SVE_VL_INHERIT flag (previously VL_INHERIT) now accessible via
   ptrace flags too.

 * Migrate default vl procfs interface to use sysctl.  Reinventing the
   wheel here is error-prone and unnecessary.

 * Improve system_supports_sve() to remove dead code when
   CONFIG_ARM64_SVE=n,

 * Fixed a use-before-allocation bug for dynamically allocated SVE task
   state.

 * Probe available vector lengths: SVE doesn't require every vector
   length up to the maximum to be supported by the hardware -- only
   power-of-two lengths are required.  The common vector lengths across
   all early CPUs are determined, and these are the vector lengths that
   userspace is permitted to set.  Late secondaries that lack support
   for any of these vector lengths are rejected.  (There will be at
   least one common VL, since support for VL=16 is mandatory.)

 * SIGILL userspace cleanly when SVE use is attempted, if the kernel or
   hardware configuration doesn't fully support it.

 * Hide SVE from the ID registers seen by the guest, and ensure
   attempted SVE use by guests is trapped and reflected to the guest as
   an undef.

 * Patch series resplit and redescribed.

 * Documentation updated to reflect ABI changes.


Full series and diffstat:

Dave Martin (27):
  regset: Add support for dynamically sized regsets
  arm64: KVM: Hide unsupported AArch64 CPU features from guests
  arm64: efi: Add missing Kconfig dependency on KERNEL_MODE_NEON
  arm64: Port deprecated instruction emulation to new sysctl interface
  arm64: fpsimd: Simplify uses of {set,clear}_ti_thread_flag()
  arm64/sve: System register and exception syndrome definitions
  arm64/sve: Low-level SVE architectural state manipulation functions
  arm64/sve: Kconfig update and conditional compilation support
  arm64/sve: Signal frame and context structure definition
  arm64/sve: Low-level CPU setup
  arm64/sve: Core task context handling
  arm64/sve: Support vector length resetting for new processes
  arm64/sve: Signal handling support
  arm64/sve: Backend logic for setting the vector length
  arm64/sve: Probe SVE capabilities and usable vector lengths
  arm64/sve: Preserve SVE registers around kernel-mode NEON use
  arm64/sve: Preserve SVE registers around EFI runtime service calls
  arm64/sve: ptrace and ELF coredump support
  arm64/sve: Add prctl controls for userspace vector length management
  arm64/sve: Add sysctl to set the default vector length for new
    processes
  arm64/sve: KVM: Prevent guests from using SVE
  arm64/sve: KVM: Treat guest SVE use as undefined instruction execution
  arm64/sve: KVM: Hide SVE from CPU features exposed to guests
  arm64/sve: Detect SVE and activate runtime support
  arm64/sve: Add documentation
  arm64: signal: Report signal frame size to userspace via auxv
  arm64/sve: signal: Include SVE when computing AT_MINSIGSTKSZ

 Documentation/arm64/sve.txt              | 454 ++++++++++++++++++++
 arch/arm64/Kconfig                       |  12 +
 arch/arm64/include/asm/cpu.h             |   4 +
 arch/arm64/include/asm/cpucaps.h         |   3 +-
 arch/arm64/include/asm/cpufeature.h      |  34 ++
 arch/arm64/include/asm/elf.h             |   5 +
 arch/arm64/include/asm/esr.h             |   3 +-
 arch/arm64/include/asm/fpsimd.h          |  68 ++-
 arch/arm64/include/asm/fpsimdmacros.h    | 137 ++++++
 arch/arm64/include/asm/kvm_arm.h         |   4 +-
 arch/arm64/include/asm/processor.h       |  10 +
 arch/arm64/include/asm/sysreg.h          |  16 +
 arch/arm64/include/asm/thread_info.h     |   2 +
 arch/arm64/include/asm/traps.h           |   2 +
 arch/arm64/include/uapi/asm/auxvec.h     |   3 +-
 arch/arm64/include/uapi/asm/hwcap.h      |   1 +
 arch/arm64/include/uapi/asm/ptrace.h     | 130 ++++++
 arch/arm64/include/uapi/asm/sigcontext.h | 113 ++++-
 arch/arm64/kernel/armv8_deprecated.c     |  15 +-
 arch/arm64/kernel/cpufeature.c           |  64 +++
 arch/arm64/kernel/cpuinfo.c              |   7 +
 arch/arm64/kernel/entry-fpsimd.S         |  17 +
 arch/arm64/kernel/entry.S                |  14 +-
 arch/arm64/kernel/fpsimd.c               | 699 ++++++++++++++++++++++++++++++-
 arch/arm64/kernel/head.S                 |  13 +-
 arch/arm64/kernel/process.c              |   6 +-
 arch/arm64/kernel/ptrace.c               | 288 ++++++++++++-
 arch/arm64/kernel/signal.c               | 214 +++++++++-
 arch/arm64/kernel/signal32.c             |   2 +-
 arch/arm64/kernel/traps.c                |   5 +-
 arch/arm64/kvm/handle_exit.c             |   8 +
 arch/arm64/kvm/hyp/switch.c              |  12 +-
 arch/arm64/kvm/sys_regs.c                | 236 +++++++++--
 arch/arm64/mm/proc.S                     |  14 +-
 fs/binfmt_elf.c                          |   6 +-
 include/linux/regset.h                   |  67 ++-
 include/uapi/linux/elf.h                 |   1 +
 include/uapi/linux/prctl.h               |   9 +
 kernel/sys.c                             |  12 +
 39 files changed, 2583 insertions(+), 127 deletions(-)
 create mode 100644 Documentation/arm64/sve.txt

-- 
2.1.4

[PATCH 01/27] regset: Add support for dynamically sized regsets

[Dave Martin]

Currently the regset API doesn't allow for the possibility that
regsets (or at least, the amount of meaningful data in a regset)
may change in size.

In particular, this results in useless padding being added to
coredumps in a regset's current size is smaller than its
theoretical maximum size.

This patch adds a get_size() function to struct user_regset.
Individual regset implementations can implement this function to
return the current size of the regset data.  A regset_size()
function is added to provide callers with an abstract interface for
determining the size of a regset without needing to know whether
the regset is dynamically sized or not.

The only affected user of this interface is the ELF coredump code:
This patch ports ELF coredump to dump regsets with their actual
size in the coredump.  This has no effect except for new regsets
that are dynamically sized and provide a get_size() implementation.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 fs/binfmt_elf.c        |  6 ++---
 include/linux/regset.h | 67 ++++++++++++++++++++++++++++++++++++++++++++------
 2 files changed, 63 insertions(+), 10 deletions(-)

diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c
index 879ff9c..16d2403 100644
--- a/fs/binfmt_elf.c
+++ b/fs/binfmt_elf.c
@@ -1697,7 +1697,7 @@ static int fill_thread_core_info(struct elf_thread_core_info *t,
 				 long signr, size_t *total)
 {
 	unsigned int i;
-	unsigned int regset_size = view->regsets[0].n * view->regsets[0].size;
+	unsigned int size = regset_size(t->task, &view->regsets[0]);
 
 	/*
 	 * NT_PRSTATUS is the one special case, because the regset data
@@ -1706,7 +1706,7 @@ static int fill_thread_core_info(struct elf_thread_core_info *t,
 	 * We assume that regset 0 is NT_PRSTATUS.
 	 */
 	fill_prstatus(&t->prstatus, t->task, signr);
-	(void) view->regsets[0].get(t->task, &view->regsets[0], 0, regset_size,
+	(void) view->regsets[0].get(t->task, &view->regsets[0], 0, size,
 				    &t->prstatus.pr_reg, NULL);
 
 	fill_note(&t->notes[0], "CORE", NT_PRSTATUS,
@@ -1726,7 +1726,7 @@ static int fill_thread_core_info(struct elf_thread_core_info *t,
 		if (regset->core_note_type && regset->get &&
 		    (!regset->active || regset->active(t->task, regset))) {
 			int ret;
-			size_t size = regset->n * regset->size;
+			size_t size = regset_size(t->task, regset);
 			void *data = kmalloc(size, GFP_KERNEL);
 			if (unlikely(!data))
 				return 0;
diff --git a/include/linux/regset.h b/include/linux/regset.h
index 8e0c9fe..494ceda 100644
--- a/include/linux/regset.h
+++ b/include/linux/regset.h
@@ -107,6 +107,28 @@ typedef int user_regset_writeback_fn(struct task_struct *target,
 				     int immediate);
 
 /**
+ * user_regset_get_size_fn - type of @get_size function in &struct user_regset
+ * @target:	thread being examined
+ * @regset:	regset being examined
+ *
+ * This call is optional; usually the pointer is %NULL.
+ *
+ * When provided, this function must return the current size of regset
+ * data, as observed by the @get function in &struct user_regset.  The
+ * value returned must be a multiple of @size.  The returned size is
+ * required to be valid only until the next time (if any) @regset is
+ * modified for @target.
+ *
+ * This function is intended for dynamically sized regsets.  A regset
+ * that is statically sized does not need to implement it.
+ *
+ * This function should not be called directly: instead, callers should
+ * call regset_size() to determine the current size of a regset.
+ */
+typedef unsigned int user_regset_get_size_fn(struct task_struct *target,
+					     const struct user_regset *regset);
+
+/**
  * struct user_regset - accessible thread CPU state
  * @n:			Number of slots (registers).
  * @size:		Size in bytes of a slot (register).
@@ -117,19 +139,33 @@ typedef int user_regset_writeback_fn(struct task_struct *target,
  * @set:		Function to store values.
  * @active:		Function to report if regset is active, or %NULL.
  * @writeback:		Function to write data back to user memory, or %NULL.
+ * @get_size:		Function to return the regset's size, or %NULL.
  *
  * This data structure describes a machine resource we call a register set.
  * This is part of the state of an individual thread, not necessarily
  * actual CPU registers per se.  A register set consists of a number of
  * similar slots, given by @n.  Each slot is @size bytes, and aligned to
- * @align bytes (which is at least @size).
+ * @align bytes (which is at least @size).  For dynamically-sized
+ * regsets, @n must contain the maximum possible number of slots for the
+ * regset, and @get_size must point to a function that returns the
+ * current regset size.
  *
- * These functions must be called only on the current thread or on a
- * thread that is in %TASK_STOPPED or %TASK_TRACED state, that we are
- * guaranteed will not be woken up and return to user mode, and that we
- * have called wait_task_inactive() on.  (The target thread always might
- * wake up for SIGKILL while these functions are working, in which case
- * that thread's user_regset state might be scrambled.)
+ * Callers that need to know only the current size of the regset and do
+ * not care about its internal structure should call regset_size()
+ * instead of inspecting @n or calling @get_size.
+ *
+ * For backward compatibility, the @get and @set methods must pad to, or
+ * accept, @n * @size bytes, even if the current regset size is smaller.
+ * The precise semantics of these operations depend on the regset being
+ * accessed.
+ *
+ * The functions to which &struct user_regset members point must be
+ * called only on the current thread or on a thread that is in
+ * %TASK_STOPPED or %TASK_TRACED state, that we are guaranteed will not
+ * be woken up and return to user mode, and that we have called
+ * wait_task_inactive() on.  (The target thread always might wake up for
+ * SIGKILL while these functions are working, in which case that
+ * thread's user_regset state might be scrambled.)
  *
  * The @pos argument must be aligned according to @align; the @count
  * argument must be a multiple of @size.  These functions are not
@@ -156,6 +192,7 @@ struct user_regset {
 	user_regset_set_fn		*set;
 	user_regset_active_fn		*active;
 	user_regset_writeback_fn	*writeback;
+	user_regset_get_size_fn		*get_size;
 	unsigned int			n;
 	unsigned int 			size;
 	unsigned int 			align;
@@ -371,5 +408,21 @@ static inline int copy_regset_from_user(struct task_struct *target,
 	return regset->set(target, regset, offset, size, NULL, data);
 }
 
+/**
+ * regset_size - determine the current size of a regset
+ * @target:	thread to be examined
+ * @regset:	regset to be examined
+ *
+ * Note that the returned size is valid only until the next time
+ * (if any) @regset is modified for @target.
+ */
+static inline unsigned int regset_size(struct task_struct *target,
+				       const struct user_regset *regset)
+{
+	if (!regset->get_size)
+		return regset->n * regset->size;
+	else
+		return regset->get_size(target, regset);
+}
 
 #endif	/* <linux/regset.h> */
-- 
2.1.4

[PATCH 01/27] regset: Add support for dynamically sized regsets

[Dave Martin]

Currently the regset API doesn't allow for the possibility that
regsets (or at least, the amount of meaningful data in a regset)
may change in size.

In particular, this results in useless padding being added to
coredumps in a regset's current size is smaller than its
theoretical maximum size.

This patch adds a get_size() function to struct user_regset.
Individual regset implementations can implement this function to
return the current size of the regset data.  A regset_size()
function is added to provide callers with an abstract interface for
determining the size of a regset without needing to know whether
the regset is dynamically sized or not.

The only affected user of this interface is the ELF coredump code:
This patch ports ELF coredump to dump regsets with their actual
size in the coredump.  This has no effect except for new regsets
that are dynamically sized and provide a get_size() implementation.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 fs/binfmt_elf.c        |  6 ++---
 include/linux/regset.h | 67 ++++++++++++++++++++++++++++++++++++++++++++------
 2 files changed, 63 insertions(+), 10 deletions(-)

diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c
index 879ff9c..16d2403 100644
--- a/fs/binfmt_elf.c
+++ b/fs/binfmt_elf.c
@@ -1697,7 +1697,7 @@ static int fill_thread_core_info(struct elf_thread_core_info *t,
 				 long signr, size_t *total)
 {
 	unsigned int i;
-	unsigned int regset_size = view->regsets[0].n * view->regsets[0].size;
+	unsigned int size = regset_size(t->task, &view->regsets[0]);
 
 	/*
 	 * NT_PRSTATUS is the one special case, because the regset data
@@ -1706,7 +1706,7 @@ static int fill_thread_core_info(struct elf_thread_core_info *t,
 	 * We assume that regset 0 is NT_PRSTATUS.
 	 */
 	fill_prstatus(&t->prstatus, t->task, signr);
-	(void) view->regsets[0].get(t->task, &view->regsets[0], 0, regset_size,
+	(void) view->regsets[0].get(t->task, &view->regsets[0], 0, size,
 				    &t->prstatus.pr_reg, NULL);
 
 	fill_note(&t->notes[0], "CORE", NT_PRSTATUS,
@@ -1726,7 +1726,7 @@ static int fill_thread_core_info(struct elf_thread_core_info *t,
 		if (regset->core_note_type && regset->get &&
 		    (!regset->active || regset->active(t->task, regset))) {
 			int ret;
-			size_t size = regset->n * regset->size;
+			size_t size = regset_size(t->task, regset);
 			void *data = kmalloc(size, GFP_KERNEL);
 			if (unlikely(!data))
 				return 0;
diff --git a/include/linux/regset.h b/include/linux/regset.h
index 8e0c9fe..494ceda 100644
--- a/include/linux/regset.h
+++ b/include/linux/regset.h
@@ -107,6 +107,28 @@ typedef int user_regset_writeback_fn(struct task_struct *target,
 				     int immediate);
 
 /**
+ * user_regset_get_size_fn - type of @get_size function in &struct user_regset
+ * @target:	thread being examined
+ * @regset:	regset being examined
+ *
+ * This call is optional; usually the pointer is %NULL.
+ *
+ * When provided, this function must return the current size of regset
+ * data, as observed by the @get function in &struct user_regset.  The
+ * value returned must be a multiple of @size.  The returned size is
+ * required to be valid only until the next time (if any) @regset is
+ * modified for @target.
+ *
+ * This function is intended for dynamically sized regsets.  A regset
+ * that is statically sized does not need to implement it.
+ *
+ * This function should not be called directly: instead, callers should
+ * call regset_size() to determine the current size of a regset.
+ */
+typedef unsigned int user_regset_get_size_fn(struct task_struct *target,
+					     const struct user_regset *regset);
+
+/**
  * struct user_regset - accessible thread CPU state
  * @n:			Number of slots (registers).
  * @size:		Size in bytes of a slot (register).
@@ -117,19 +139,33 @@ typedef int user_regset_writeback_fn(struct task_struct *target,
  * @set:		Function to store values.
  * @active:		Function to report if regset is active, or %NULL.
  * @writeback:		Function to write data back to user memory, or %NULL.
+ * @get_size:		Function to return the regset's size, or %NULL.
  *
  * This data structure describes a machine resource we call a register set.
  * This is part of the state of an individual thread, not necessarily
  * actual CPU registers per se.  A register set consists of a number of
  * similar slots, given by @n.  Each slot is @size bytes, and aligned to
- * @align bytes (which is at least @size).
+ * @align bytes (which is at least @size).  For dynamically-sized
+ * regsets, @n must contain the maximum possible number of slots for the
+ * regset, and @get_size must point to a function that returns the
+ * current regset size.
  *
- * These functions must be called only on the current thread or on a
- * thread that is in %TASK_STOPPED or %TASK_TRACED state, that we are
- * guaranteed will not be woken up and return to user mode, and that we
- * have called wait_task_inactive() on.  (The target thread always might
- * wake up for SIGKILL while these functions are working, in which case
- * that thread's user_regset state might be scrambled.)
+ * Callers that need to know only the current size of the regset and do
+ * not care about its internal structure should call regset_size()
+ * instead of inspecting @n or calling @get_size.
+ *
+ * For backward compatibility, the @get and @set methods must pad to, or
+ * accept, @n * @size bytes, even if the current regset size is smaller.
+ * The precise semantics of these operations depend on the regset being
+ * accessed.
+ *
+ * The functions to which &struct user_regset members point must be
+ * called only on the current thread or on a thread that is in
+ * %TASK_STOPPED or %TASK_TRACED state, that we are guaranteed will not
+ * be woken up and return to user mode, and that we have called
+ * wait_task_inactive() on.  (The target thread always might wake up for
+ * SIGKILL while these functions are working, in which case that
+ * thread's user_regset state might be scrambled.)
  *
  * The @pos argument must be aligned according to @align; the @count
  * argument must be a multiple of @size.  These functions are not
@@ -156,6 +192,7 @@ struct user_regset {
 	user_regset_set_fn		*set;
 	user_regset_active_fn		*active;
 	user_regset_writeback_fn	*writeback;
+	user_regset_get_size_fn		*get_size;
 	unsigned int			n;
 	unsigned int 			size;
 	unsigned int 			align;
@@ -371,5 +408,21 @@ static inline int copy_regset_from_user(struct task_struct *target,
 	return regset->set(target, regset, offset, size, NULL, data);
 }
 
+/**
+ * regset_size - determine the current size of a regset
+ * @target:	thread to be examined
+ * @regset:	regset to be examined
+ *
+ * Note that the returned size is valid only until the next time
+ * (if any) @regset is modified for @target.
+ */
+static inline unsigned int regset_size(struct task_struct *target,
+				       const struct user_regset *regset)
+{
+	if (!regset->get_size)
+		return regset->n * regset->size;
+	else
+		return regset->get_size(target, regset);
+}
 
 #endif	/* <linux/regset.h> */
-- 
2.1.4

[PATCH 02/27] arm64: KVM: Hide unsupported AArch64 CPU features from guests

[Dave Martin]

Currently, a guest kernel sees the true CPU feature registers
(ID_*_EL1) when it reads them using MRS instructions.  This means
that the guest will observe features that are present in the
hardware but the host doesn't understand or doesn't provide support
for.  A guest may legimitately try to use such a feature as per the
architecture, but use of the feature may trap instead of working
normally, triggering undef injection into the guest.

This is not a problem for the host, but the guest may go wrong when
running on newer hardware than the host knows about.

This patch hides from guest VMs any AArch64-specific CPU features
that the host doesn't support, by exposing to the guest the
sanitised versions of the registers computed by the cpufeatures
framework, instead of the true hardware registers.  To achieve
this, HCR_EL2.TID3 is now set for AArch64 guests, and emulation
code is added to KVM to report the sanitised versions of the
affected registers in response to MRS and register reads from
userspace.

The affected registers are removed from invariant_sys_regs[] (since
the invariant_sys_regs handling is no longer quite correct for
them) and added to sys_reg_desgs[], with appropriate access(),
get_user() and set_user() methods.  No runtime vcpu storage is
allocated for the registers: instead, they are read on demand from
the cpufeatures framework.  This may need modification in the
future if there is a need for userspace to customise the features
visible to the guest.

Attempts by userspace to write the registers are handled similarly
to the current invariant_sys_regs handling: writes are permitted,
but only if they don't attempt to change the value.  This is
sufficient to support VM snapshot/restore from userspace.

Because of the additional registers, restoring a VM on an older
kernel may not work unless userspace knows how to handle the extra
VM registers exposed to the KVM user ABI by this patch.

Under the principle of least damage, this patch makes no attempt to
handle any of the other registers currently in
invariant_sys_regs[], or to emulate registers for AArch32: however,
these could be handled in a similar way in future, as necessary.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/kvm/hyp/switch.c |   6 ++
 arch/arm64/kvm/sys_regs.c   | 224 +++++++++++++++++++++++++++++++++++---------
 2 files changed, 185 insertions(+), 45 deletions(-)

diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 945e79c..35a90b8 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -81,11 +81,17 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
 	 * it will cause an exception.
 	 */
 	val = vcpu->arch.hcr_el2;
+
 	if (!(val & HCR_RW) && system_supports_fpsimd()) {
 		write_sysreg(1 << 30, fpexc32_el2);
 		isb();
 	}
+
+	if (val & HCR_RW) /* for AArch64 only: */
+		val |= HCR_TID3; /* TID3: trap feature register accesses */
+
 	write_sysreg(val, hcr_el2);
+
 	/* Trap on AArch32 cp15 c15 accesses (EL1 or EL0) */
 	write_sysreg(1 << 15, hstr_el2);
 	/*
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 2e070d3..6583dd7 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -892,6 +892,135 @@ static bool access_cntp_cval(struct kvm_vcpu *vcpu,
 	return true;
 }
 
+/* Read a sanitised cpufeature ID register by sys_reg_desc */
+static u64 read_id_reg(struct sys_reg_desc const *r, bool raz)
+{
+	u32 id = sys_reg((u32)r->Op0, (u32)r->Op1,
+			 (u32)r->CRn, (u32)r->CRm, (u32)r->Op2);
+
+	return raz ? 0 : read_sanitised_ftr_reg(id);
+}
+
+/* cpufeature ID register access trap handlers */
+
+static bool __access_id_reg(struct kvm_vcpu *vcpu,
+			    struct sys_reg_params *p,
+			    const struct sys_reg_desc const *r,
+			    bool raz)
+{
+	if (p->is_write) {
+		kvm_inject_undefined(vcpu);
+		return false;
+	}
+
+	p->regval = read_id_reg(r, raz);
+	return true;
+}
+
+static bool access_id_reg(struct kvm_vcpu *vcpu,
+			  struct sys_reg_params *p,
+			  const struct sys_reg_desc *r)
+{
+	return __access_id_reg(vcpu, p, r, false);
+}
+
+static bool access_raz_id_reg(struct kvm_vcpu *vcpu,
+			      struct sys_reg_params *p,
+			      const struct sys_reg_desc *r)
+{
+	return __access_id_reg(vcpu, p, r, true);
+}
+
+static int reg_from_user(u64 *val, const void __user *uaddr, u64 id);
+static int reg_to_user(void __user *uaddr, const u64 *val, u64 id);
+static u64 sys_reg_to_index(const struct sys_reg_desc *reg);
+
+/*
+ * cpufeature ID register user accessors
+ *
+ * For now, these registers are immutable for userspace, so no values
+ * are stored, and for set_id_reg() we don't allow the effective value
+ * to be changed.
+ */
+static int __get_id_reg(const struct sys_reg_desc *rd, void __user *uaddr,
+			bool raz)
+{
+	const u64 id = sys_reg_to_index(rd);
+	const u64 val = read_id_reg(rd, raz);
+
+	BUG_ON(KVM_REG_SIZE(id) != sizeof(val));
+	return reg_to_user(uaddr, &val, id);
+}
+
+static int __set_id_reg(const struct sys_reg_desc *rd, void __user *uaddr,
+			bool raz)
+{
+	const u64 id = sys_reg_to_index(rd);
+	int err;
+	u64 val;
+
+	BUG_ON(KVM_REG_SIZE(id) != sizeof(val));
+	err = reg_from_user(&val, uaddr, id);
+	if (err)
+		return err;
+
+	/* This is what we mean by invariant: you can't change it. */
+	if (val != read_id_reg(rd, raz))
+		return -EINVAL;
+
+	return 0;
+}
+
+static int get_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
+		      const struct kvm_one_reg *reg, void __user *uaddr)
+{
+	return __get_id_reg(rd, uaddr, false);
+}
+
+static int set_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
+		      const struct kvm_one_reg *reg, void __user *uaddr)
+{
+	return __set_id_reg(rd, uaddr, false);
+}
+
+static int get_raz_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
+			  const struct kvm_one_reg *reg, void __user *uaddr)
+{
+	return __get_id_reg(rd, uaddr, true);
+}
+
+static int set_raz_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
+			  const struct kvm_one_reg *reg, void __user *uaddr)
+{
+	return __set_id_reg(rd, uaddr, true);
+}
+
+/* sys_reg_desc initialiser for cpufeature ID register name_EL1 */
+#define _ID(name) {			\
+	SYS_DESC(SYS_##name##_EL1),	\
+	.access	= access_id_reg,	\
+	.get_user = get_id_reg,		\
+	.set_user = set_id_reg,		\
+}
+
+/*
+ * sys_reg_desc initialiser for cpufeature ID register ID_name_EL1
+ * (So we can get 4 regs to 1 line.)
+ */
+#define ID(name) _ID(ID_##name)
+
+/*
+ * sys_reg_desc initialiser for unknown (RAZ) cpufeature ID register
+ * Op0=3, Op1=0, CRn=0, CRm=crm, Op2=op2
+ * (1 <= crm < 8, 0 <= Op2 < 8).
+ */
+#define _ID_RAZ(crm, op2) {				\
+	Op0(3), Op1(0), CRn(0), CRm(crm), Op2(op2),	\
+	.access = access_raz_id_reg,			\
+	.get_user = get_raz_id_reg,			\
+	.set_user = set_raz_id_reg,			\
+}
+
 /*
  * Architected system registers.
  * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2
@@ -944,6 +1073,32 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	{ SYS_DESC(SYS_DBGVCR32_EL2), NULL, reset_val, DBGVCR32_EL2, 0 },
 
 	{ SYS_DESC(SYS_MPIDR_EL1), NULL, reset_mpidr, MPIDR_EL1 },
+
+	/*
+	 * All non-RAZ feature registers listed here must also be
+	 * present in arm64_ftr_regs[].
+	 */
+
+	/* AArch64 mappings of the AArch32 ID registers */
+	/* ID_AFR0_EL1 not exposed to guests for now */
+	ID(PFR0),	ID(PFR1),	ID(DFR0),	_ID_RAZ(1,3),
+	ID(MMFR0),	ID(MMFR1),	ID(MMFR2),	ID(MMFR3),
+	ID(ISAR0),	ID(ISAR1),	ID(ISAR2),	ID(ISAR3),
+	ID(ISAR4),	ID(ISAR5),	ID(MMFR4),	_ID_RAZ(2,7),
+	_ID(MVFR0),	_ID(MVFR1),	_ID(MVFR2),	_ID_RAZ(3,3),
+	_ID_RAZ(3,4),	_ID_RAZ(3,5),	_ID_RAZ(3,6),	_ID_RAZ(3,7),
+
+	/* AArch64 ID registers */
+	ID(AA64PFR0),	ID(AA64PFR1),	_ID_RAZ(4,2),	_ID_RAZ(4,3),
+	_ID_RAZ(4,4),	_ID_RAZ(4,5),	_ID_RAZ(4,6),	_ID_RAZ(4,7),
+	ID(AA64DFR0),	ID(AA64DFR1),	_ID_RAZ(5,2),	_ID_RAZ(5,3),
+	/* ID_AA64AFR0_EL1 and ID_AA64AFR0_EL1 not exposed to guests for now */
+	_ID_RAZ(5,4),	_ID_RAZ(5,5),	_ID_RAZ(5,6),	_ID_RAZ(5,7),
+	ID(AA64ISAR0),	ID(AA64ISAR1),	_ID_RAZ(6,2),	_ID_RAZ(6,3),
+	_ID_RAZ(6,4),	_ID_RAZ(6,5),	_ID_RAZ(6,6),	_ID_RAZ(6,7),
+	ID(AA64MMFR0),	ID(AA64MMFR1),	ID(AA64MMFR2),	_ID_RAZ(7,3),
+	_ID_RAZ(7,4),	_ID_RAZ(7,5),	_ID_RAZ(7,6),	_ID_RAZ(7,7),
+
 	{ SYS_DESC(SYS_SCTLR_EL1), access_vm_reg, reset_val, SCTLR_EL1, 0x00C50078 },
 	{ SYS_DESC(SYS_CPACR_EL1), NULL, reset_val, CPACR_EL1, 0 },
 	{ SYS_DESC(SYS_TTBR0_EL1), access_vm_reg, reset_unknown, TTBR0_EL1 },
@@ -1790,8 +1945,8 @@ static const struct sys_reg_desc *index_to_sys_reg_desc(struct kvm_vcpu *vcpu,
 	if (!r)
 		r = find_reg(&params, sys_reg_descs, ARRAY_SIZE(sys_reg_descs));
 
-	/* Not saved in the sys_reg array? */
-	if (r && !r->reg)
+	/* Not saved in the sys_reg array and not otherwise accessible? */
+	if (r && !(r->reg || r->get_user))
 		r = NULL;
 
 	return r;
@@ -1815,20 +1970,6 @@ static const struct sys_reg_desc *index_to_sys_reg_desc(struct kvm_vcpu *vcpu,
 FUNCTION_INVARIANT(midr_el1)
 FUNCTION_INVARIANT(ctr_el0)
 FUNCTION_INVARIANT(revidr_el1)
-FUNCTION_INVARIANT(id_pfr0_el1)
-FUNCTION_INVARIANT(id_pfr1_el1)
-FUNCTION_INVARIANT(id_dfr0_el1)
-FUNCTION_INVARIANT(id_afr0_el1)
-FUNCTION_INVARIANT(id_mmfr0_el1)
-FUNCTION_INVARIANT(id_mmfr1_el1)
-FUNCTION_INVARIANT(id_mmfr2_el1)
-FUNCTION_INVARIANT(id_mmfr3_el1)
-FUNCTION_INVARIANT(id_isar0_el1)
-FUNCTION_INVARIANT(id_isar1_el1)
-FUNCTION_INVARIANT(id_isar2_el1)
-FUNCTION_INVARIANT(id_isar3_el1)
-FUNCTION_INVARIANT(id_isar4_el1)
-FUNCTION_INVARIANT(id_isar5_el1)
 FUNCTION_INVARIANT(clidr_el1)
 FUNCTION_INVARIANT(aidr_el1)
 
@@ -1836,20 +1977,6 @@ FUNCTION_INVARIANT(aidr_el1)
 static struct sys_reg_desc invariant_sys_regs[] = {
 	{ SYS_DESC(SYS_MIDR_EL1), NULL, get_midr_el1 },
 	{ SYS_DESC(SYS_REVIDR_EL1), NULL, get_revidr_el1 },
-	{ SYS_DESC(SYS_ID_PFR0_EL1), NULL, get_id_pfr0_el1 },
-	{ SYS_DESC(SYS_ID_PFR1_EL1), NULL, get_id_pfr1_el1 },
-	{ SYS_DESC(SYS_ID_DFR0_EL1), NULL, get_id_dfr0_el1 },
-	{ SYS_DESC(SYS_ID_AFR0_EL1), NULL, get_id_afr0_el1 },
-	{ SYS_DESC(SYS_ID_MMFR0_EL1), NULL, get_id_mmfr0_el1 },
-	{ SYS_DESC(SYS_ID_MMFR1_EL1), NULL, get_id_mmfr1_el1 },
-	{ SYS_DESC(SYS_ID_MMFR2_EL1), NULL, get_id_mmfr2_el1 },
-	{ SYS_DESC(SYS_ID_MMFR3_EL1), NULL, get_id_mmfr3_el1 },
-	{ SYS_DESC(SYS_ID_ISAR0_EL1), NULL, get_id_isar0_el1 },
-	{ SYS_DESC(SYS_ID_ISAR1_EL1), NULL, get_id_isar1_el1 },
-	{ SYS_DESC(SYS_ID_ISAR2_EL1), NULL, get_id_isar2_el1 },
-	{ SYS_DESC(SYS_ID_ISAR3_EL1), NULL, get_id_isar3_el1 },
-	{ SYS_DESC(SYS_ID_ISAR4_EL1), NULL, get_id_isar4_el1 },
-	{ SYS_DESC(SYS_ID_ISAR5_EL1), NULL, get_id_isar5_el1 },
 	{ SYS_DESC(SYS_CLIDR_EL1), NULL, get_clidr_el1 },
 	{ SYS_DESC(SYS_AIDR_EL1), NULL, get_aidr_el1 },
 	{ SYS_DESC(SYS_CTR_EL0), NULL, get_ctr_el0 },
@@ -2079,12 +2206,31 @@ static bool copy_reg_to_user(const struct sys_reg_desc *reg, u64 __user **uind)
 	return true;
 }
 
+static int walk_one_sys_reg(const struct sys_reg_desc *rd,
+			    u64 __user **uind,
+			    unsigned int *total)
+{
+	/*
+	 * Ignore registers we trap but don't save,
+	 * and for which no custom user accessor is provided.
+	 */
+	if (!(rd->reg || rd->get_user))
+		return 0;
+
+	if (!copy_reg_to_user(rd, uind))
+		return -EFAULT;
+
+	(*total)++;
+	return 0;
+}
+
 /* Assumed ordered tables, see kvm_sys_reg_table_init. */
 static int walk_sys_regs(struct kvm_vcpu *vcpu, u64 __user *uind)
 {
 	const struct sys_reg_desc *i1, *i2, *end1, *end2;
 	unsigned int total = 0;
 	size_t num;
+	int err;
 
 	/* We check for duplicates here, to allow arch-specific overrides. */
 	i1 = get_target_table(vcpu->arch.target, true, &num);
@@ -2098,21 +2244,9 @@ static int walk_sys_regs(struct kvm_vcpu *vcpu, u64 __user *uind)
 	while (i1 || i2) {
 		int cmp = cmp_sys_reg(i1, i2);
 		/* target-specific overrides generic entry. */
-		if (cmp <= 0) {
-			/* Ignore registers we trap but don't save. */
-			if (i1->reg) {
-				if (!copy_reg_to_user(i1, &uind))
-					return -EFAULT;
-				total++;
-			}
-		} else {
-			/* Ignore registers we trap but don't save. */
-			if (i2->reg) {
-				if (!copy_reg_to_user(i2, &uind))
-					return -EFAULT;
-				total++;
-			}
-		}
+		err = walk_one_sys_reg(cmp <= 0 ? i1 : i2, &uind, &total);
+		if (err)
+			return err;
 
 		if (cmp <= 0 && ++i1 == end1)
 			i1 = NULL;
-- 
2.1.4

[PATCH 02/27] arm64: KVM: Hide unsupported AArch64 CPU features from guests

[Dave Martin]

Currently, a guest kernel sees the true CPU feature registers
(ID_*_EL1) when it reads them using MRS instructions.  This means
that the guest will observe features that are present in the
hardware but the host doesn't understand or doesn't provide support
for.  A guest may legimitately try to use such a feature as per the
architecture, but use of the feature may trap instead of working
normally, triggering undef injection into the guest.

This is not a problem for the host, but the guest may go wrong when
running on newer hardware than the host knows about.

This patch hides from guest VMs any AArch64-specific CPU features
that the host doesn't support, by exposing to the guest the
sanitised versions of the registers computed by the cpufeatures
framework, instead of the true hardware registers.  To achieve
this, HCR_EL2.TID3 is now set for AArch64 guests, and emulation
code is added to KVM to report the sanitised versions of the
affected registers in response to MRS and register reads from
userspace.

The affected registers are removed from invariant_sys_regs[] (since
the invariant_sys_regs handling is no longer quite correct for
them) and added to sys_reg_desgs[], with appropriate access(),
get_user() and set_user() methods.  No runtime vcpu storage is
allocated for the registers: instead, they are read on demand from
the cpufeatures framework.  This may need modification in the
future if there is a need for userspace to customise the features
visible to the guest.

Attempts by userspace to write the registers are handled similarly
to the current invariant_sys_regs handling: writes are permitted,
but only if they don't attempt to change the value.  This is
sufficient to support VM snapshot/restore from userspace.

Because of the additional registers, restoring a VM on an older
kernel may not work unless userspace knows how to handle the extra
VM registers exposed to the KVM user ABI by this patch.

Under the principle of least damage, this patch makes no attempt to
handle any of the other registers currently in
invariant_sys_regs[], or to emulate registers for AArch32: however,
these could be handled in a similar way in future, as necessary.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/kvm/hyp/switch.c |   6 ++
 arch/arm64/kvm/sys_regs.c   | 224 +++++++++++++++++++++++++++++++++++---------
 2 files changed, 185 insertions(+), 45 deletions(-)

diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 945e79c..35a90b8 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -81,11 +81,17 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
 	 * it will cause an exception.
 	 */
 	val = vcpu->arch.hcr_el2;
+
 	if (!(val & HCR_RW) && system_supports_fpsimd()) {
 		write_sysreg(1 << 30, fpexc32_el2);
 		isb();
 	}
+
+	if (val & HCR_RW) /* for AArch64 only: */
+		val |= HCR_TID3; /* TID3: trap feature register accesses */
+
 	write_sysreg(val, hcr_el2);
+
 	/* Trap on AArch32 cp15 c15 accesses (EL1 or EL0) */
 	write_sysreg(1 << 15, hstr_el2);
 	/*
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 2e070d3..6583dd7 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -892,6 +892,135 @@ static bool access_cntp_cval(struct kvm_vcpu *vcpu,
 	return true;
 }
 
+/* Read a sanitised cpufeature ID register by sys_reg_desc */
+static u64 read_id_reg(struct sys_reg_desc const *r, bool raz)
+{
+	u32 id = sys_reg((u32)r->Op0, (u32)r->Op1,
+			 (u32)r->CRn, (u32)r->CRm, (u32)r->Op2);
+
+	return raz ? 0 : read_sanitised_ftr_reg(id);
+}
+
+/* cpufeature ID register access trap handlers */
+
+static bool __access_id_reg(struct kvm_vcpu *vcpu,
+			    struct sys_reg_params *p,
+			    const struct sys_reg_desc const *r,
+			    bool raz)
+{
+	if (p->is_write) {
+		kvm_inject_undefined(vcpu);
+		return false;
+	}
+
+	p->regval = read_id_reg(r, raz);
+	return true;
+}
+
+static bool access_id_reg(struct kvm_vcpu *vcpu,
+			  struct sys_reg_params *p,
+			  const struct sys_reg_desc *r)
+{
+	return __access_id_reg(vcpu, p, r, false);
+}
+
+static bool access_raz_id_reg(struct kvm_vcpu *vcpu,
+			      struct sys_reg_params *p,
+			      const struct sys_reg_desc *r)
+{
+	return __access_id_reg(vcpu, p, r, true);
+}
+
+static int reg_from_user(u64 *val, const void __user *uaddr, u64 id);
+static int reg_to_user(void __user *uaddr, const u64 *val, u64 id);
+static u64 sys_reg_to_index(const struct sys_reg_desc *reg);
+
+/*
+ * cpufeature ID register user accessors
+ *
+ * For now, these registers are immutable for userspace, so no values
+ * are stored, and for set_id_reg() we don't allow the effective value
+ * to be changed.
+ */
+static int __get_id_reg(const struct sys_reg_desc *rd, void __user *uaddr,
+			bool raz)
+{
+	const u64 id = sys_reg_to_index(rd);
+	const u64 val = read_id_reg(rd, raz);
+
+	BUG_ON(KVM_REG_SIZE(id) != sizeof(val));
+	return reg_to_user(uaddr, &val, id);
+}
+
+static int __set_id_reg(const struct sys_reg_desc *rd, void __user *uaddr,
+			bool raz)
+{
+	const u64 id = sys_reg_to_index(rd);
+	int err;
+	u64 val;
+
+	BUG_ON(KVM_REG_SIZE(id) != sizeof(val));
+	err = reg_from_user(&val, uaddr, id);
+	if (err)
+		return err;
+
+	/* This is what we mean by invariant: you can't change it. */
+	if (val != read_id_reg(rd, raz))
+		return -EINVAL;
+
+	return 0;
+}
+
+static int get_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
+		      const struct kvm_one_reg *reg, void __user *uaddr)
+{
+	return __get_id_reg(rd, uaddr, false);
+}
+
+static int set_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
+		      const struct kvm_one_reg *reg, void __user *uaddr)
+{
+	return __set_id_reg(rd, uaddr, false);
+}
+
+static int get_raz_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
+			  const struct kvm_one_reg *reg, void __user *uaddr)
+{
+	return __get_id_reg(rd, uaddr, true);
+}
+
+static int set_raz_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
+			  const struct kvm_one_reg *reg, void __user *uaddr)
+{
+	return __set_id_reg(rd, uaddr, true);
+}
+
+/* sys_reg_desc initialiser for cpufeature ID register name_EL1 */
+#define _ID(name) {			\
+	SYS_DESC(SYS_##name##_EL1),	\
+	.access	= access_id_reg,	\
+	.get_user = get_id_reg,		\
+	.set_user = set_id_reg,		\
+}
+
+/*
+ * sys_reg_desc initialiser for cpufeature ID register ID_name_EL1
+ * (So we can get 4 regs to 1 line.)
+ */
+#define ID(name) _ID(ID_##name)
+
+/*
+ * sys_reg_desc initialiser for unknown (RAZ) cpufeature ID register
+ * Op0=3, Op1=0, CRn=0, CRm=crm, Op2=op2
+ * (1 <= crm < 8, 0 <= Op2 < 8).
+ */
+#define _ID_RAZ(crm, op2) {				\
+	Op0(3), Op1(0), CRn(0), CRm(crm), Op2(op2),	\
+	.access = access_raz_id_reg,			\
+	.get_user = get_raz_id_reg,			\
+	.set_user = set_raz_id_reg,			\
+}
+
 /*
  * Architected system registers.
  * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2
@@ -944,6 +1073,32 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	{ SYS_DESC(SYS_DBGVCR32_EL2), NULL, reset_val, DBGVCR32_EL2, 0 },
 
 	{ SYS_DESC(SYS_MPIDR_EL1), NULL, reset_mpidr, MPIDR_EL1 },
+
+	/*
+	 * All non-RAZ feature registers listed here must also be
+	 * present in arm64_ftr_regs[].
+	 */
+
+	/* AArch64 mappings of the AArch32 ID registers */
+	/* ID_AFR0_EL1 not exposed to guests for now */
+	ID(PFR0),	ID(PFR1),	ID(DFR0),	_ID_RAZ(1,3),
+	ID(MMFR0),	ID(MMFR1),	ID(MMFR2),	ID(MMFR3),
+	ID(ISAR0),	ID(ISAR1),	ID(ISAR2),	ID(ISAR3),
+	ID(ISAR4),	ID(ISAR5),	ID(MMFR4),	_ID_RAZ(2,7),
+	_ID(MVFR0),	_ID(MVFR1),	_ID(MVFR2),	_ID_RAZ(3,3),
+	_ID_RAZ(3,4),	_ID_RAZ(3,5),	_ID_RAZ(3,6),	_ID_RAZ(3,7),
+
+	/* AArch64 ID registers */
+	ID(AA64PFR0),	ID(AA64PFR1),	_ID_RAZ(4,2),	_ID_RAZ(4,3),
+	_ID_RAZ(4,4),	_ID_RAZ(4,5),	_ID_RAZ(4,6),	_ID_RAZ(4,7),
+	ID(AA64DFR0),	ID(AA64DFR1),	_ID_RAZ(5,2),	_ID_RAZ(5,3),
+	/* ID_AA64AFR0_EL1 and ID_AA64AFR0_EL1 not exposed to guests for now */
+	_ID_RAZ(5,4),	_ID_RAZ(5,5),	_ID_RAZ(5,6),	_ID_RAZ(5,7),
+	ID(AA64ISAR0),	ID(AA64ISAR1),	_ID_RAZ(6,2),	_ID_RAZ(6,3),
+	_ID_RAZ(6,4),	_ID_RAZ(6,5),	_ID_RAZ(6,6),	_ID_RAZ(6,7),
+	ID(AA64MMFR0),	ID(AA64MMFR1),	ID(AA64MMFR2),	_ID_RAZ(7,3),
+	_ID_RAZ(7,4),	_ID_RAZ(7,5),	_ID_RAZ(7,6),	_ID_RAZ(7,7),
+
 	{ SYS_DESC(SYS_SCTLR_EL1), access_vm_reg, reset_val, SCTLR_EL1, 0x00C50078 },
 	{ SYS_DESC(SYS_CPACR_EL1), NULL, reset_val, CPACR_EL1, 0 },
 	{ SYS_DESC(SYS_TTBR0_EL1), access_vm_reg, reset_unknown, TTBR0_EL1 },
@@ -1790,8 +1945,8 @@ static const struct sys_reg_desc *index_to_sys_reg_desc(struct kvm_vcpu *vcpu,
 	if (!r)
 		r = find_reg(&params, sys_reg_descs, ARRAY_SIZE(sys_reg_descs));
 
-	/* Not saved in the sys_reg array? */
-	if (r && !r->reg)
+	/* Not saved in the sys_reg array and not otherwise accessible? */
+	if (r && !(r->reg || r->get_user))
 		r = NULL;
 
 	return r;
@@ -1815,20 +1970,6 @@ static const struct sys_reg_desc *index_to_sys_reg_desc(struct kvm_vcpu *vcpu,
 FUNCTION_INVARIANT(midr_el1)
 FUNCTION_INVARIANT(ctr_el0)
 FUNCTION_INVARIANT(revidr_el1)
-FUNCTION_INVARIANT(id_pfr0_el1)
-FUNCTION_INVARIANT(id_pfr1_el1)
-FUNCTION_INVARIANT(id_dfr0_el1)
-FUNCTION_INVARIANT(id_afr0_el1)
-FUNCTION_INVARIANT(id_mmfr0_el1)
-FUNCTION_INVARIANT(id_mmfr1_el1)
-FUNCTION_INVARIANT(id_mmfr2_el1)
-FUNCTION_INVARIANT(id_mmfr3_el1)
-FUNCTION_INVARIANT(id_isar0_el1)
-FUNCTION_INVARIANT(id_isar1_el1)
-FUNCTION_INVARIANT(id_isar2_el1)
-FUNCTION_INVARIANT(id_isar3_el1)
-FUNCTION_INVARIANT(id_isar4_el1)
-FUNCTION_INVARIANT(id_isar5_el1)
 FUNCTION_INVARIANT(clidr_el1)
 FUNCTION_INVARIANT(aidr_el1)
 
@@ -1836,20 +1977,6 @@ FUNCTION_INVARIANT(aidr_el1)
 static struct sys_reg_desc invariant_sys_regs[] = {
 	{ SYS_DESC(SYS_MIDR_EL1), NULL, get_midr_el1 },
 	{ SYS_DESC(SYS_REVIDR_EL1), NULL, get_revidr_el1 },
-	{ SYS_DESC(SYS_ID_PFR0_EL1), NULL, get_id_pfr0_el1 },
-	{ SYS_DESC(SYS_ID_PFR1_EL1), NULL, get_id_pfr1_el1 },
-	{ SYS_DESC(SYS_ID_DFR0_EL1), NULL, get_id_dfr0_el1 },
-	{ SYS_DESC(SYS_ID_AFR0_EL1), NULL, get_id_afr0_el1 },
-	{ SYS_DESC(SYS_ID_MMFR0_EL1), NULL, get_id_mmfr0_el1 },
-	{ SYS_DESC(SYS_ID_MMFR1_EL1), NULL, get_id_mmfr1_el1 },
-	{ SYS_DESC(SYS_ID_MMFR2_EL1), NULL, get_id_mmfr2_el1 },
-	{ SYS_DESC(SYS_ID_MMFR3_EL1), NULL, get_id_mmfr3_el1 },
-	{ SYS_DESC(SYS_ID_ISAR0_EL1), NULL, get_id_isar0_el1 },
-	{ SYS_DESC(SYS_ID_ISAR1_EL1), NULL, get_id_isar1_el1 },
-	{ SYS_DESC(SYS_ID_ISAR2_EL1), NULL, get_id_isar2_el1 },
-	{ SYS_DESC(SYS_ID_ISAR3_EL1), NULL, get_id_isar3_el1 },
-	{ SYS_DESC(SYS_ID_ISAR4_EL1), NULL, get_id_isar4_el1 },
-	{ SYS_DESC(SYS_ID_ISAR5_EL1), NULL, get_id_isar5_el1 },
 	{ SYS_DESC(SYS_CLIDR_EL1), NULL, get_clidr_el1 },
 	{ SYS_DESC(SYS_AIDR_EL1), NULL, get_aidr_el1 },
 	{ SYS_DESC(SYS_CTR_EL0), NULL, get_ctr_el0 },
@@ -2079,12 +2206,31 @@ static bool copy_reg_to_user(const struct sys_reg_desc *reg, u64 __user **uind)
 	return true;
 }
 
+static int walk_one_sys_reg(const struct sys_reg_desc *rd,
+			    u64 __user **uind,
+			    unsigned int *total)
+{
+	/*
+	 * Ignore registers we trap but don't save,
+	 * and for which no custom user accessor is provided.
+	 */
+	if (!(rd->reg || rd->get_user))
+		return 0;
+
+	if (!copy_reg_to_user(rd, uind))
+		return -EFAULT;
+
+	(*total)++;
+	return 0;
+}
+
 /* Assumed ordered tables, see kvm_sys_reg_table_init. */
 static int walk_sys_regs(struct kvm_vcpu *vcpu, u64 __user *uind)
 {
 	const struct sys_reg_desc *i1, *i2, *end1, *end2;
 	unsigned int total = 0;
 	size_t num;
+	int err;
 
 	/* We check for duplicates here, to allow arch-specific overrides. */
 	i1 = get_target_table(vcpu->arch.target, true, &num);
@@ -2098,21 +2244,9 @@ static int walk_sys_regs(struct kvm_vcpu *vcpu, u64 __user *uind)
 	while (i1 || i2) {
 		int cmp = cmp_sys_reg(i1, i2);
 		/* target-specific overrides generic entry. */
-		if (cmp <= 0) {
-			/* Ignore registers we trap but don't save. */
-			if (i1->reg) {
-				if (!copy_reg_to_user(i1, &uind))
-					return -EFAULT;
-				total++;
-			}
-		} else {
-			/* Ignore registers we trap but don't save. */
-			if (i2->reg) {
-				if (!copy_reg_to_user(i2, &uind))
-					return -EFAULT;
-				total++;
-			}
-		}
+		err = walk_one_sys_reg(cmp <= 0 ? i1 : i2, &uind, &total);
+		if (err)
+			return err;
 
 		if (cmp <= 0 && ++i1 == end1)
 			i1 = NULL;
-- 
2.1.4

[PATCH 03/27] arm64: efi: Add missing Kconfig dependency on KERNEL_MODE_NEON

[Dave Martin]

The EFI runtime services ABI permits calls to EFI to clobber
certain FPSIMD/NEON registers, as per the AArch64 procedure call
standard.

Saving/restoring the clobbered registers around such calls needs
KERNEL_MODE_NEON, but the dependency is missing from Kconfig.

This patch adds the missing dependency.

This will aid bisection of the patches implementing support for the
ARM Scalable Vector Extension (SVE).

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index dfd9086..fdc302e 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1050,6 +1050,7 @@ config EFI_STUB
 config EFI
 	bool "UEFI runtime support"
 	depends on OF && !CPU_BIG_ENDIAN
+	depends on KERNEL_MODE_NEON
 	select LIBFDT
 	select UCS2_STRING
 	select EFI_PARAMS_FROM_FDT
-- 
2.1.4

[PATCH 04/27] arm64: Port deprecated instruction emulation to new sysctl interface

[Dave Martin]

Currently, armv8_deprected.c takes charge of the "abi" sysctl
directory, which makes life difficult for other code that wants to
register sysctls in the same directory.

There is a "new" [1] sysctl registration interface that removes the
need to define ctl_tables for parent directories explicitly, which
is ideal here.

This patch ports register_insn_emulation_sysctl() over to the
register_sysctl() interface and removes the redundant ctl_table for
"abi".

Signed-off-by: Dave Martin <Dave.Martin@arm.com>

[1] fea478d4101a (sysctl: Add register_sysctl for normal sysctl
users)
The commit message notes an intent to port users of the
pre-existing interfaces over to register_sysctl(), though the
number of users of the new interface currently appears negligible.
---
 arch/arm64/kernel/armv8_deprecated.c | 15 +++------------
 1 file changed, 3 insertions(+), 12 deletions(-)

diff --git a/arch/arm64/kernel/armv8_deprecated.c b/arch/arm64/kernel/armv8_deprecated.c
index f0e6d71..e15eb2d 100644
--- a/arch/arm64/kernel/armv8_deprecated.c
+++ b/arch/arm64/kernel/armv8_deprecated.c
@@ -228,15 +228,7 @@ static int emulation_proc_handler(struct ctl_table *table, int write,
 	return ret;
 }
 
-static struct ctl_table ctl_abi[] = {
-	{
-		.procname = "abi",
-		.mode = 0555,
-	},
-	{ }
-};
-
-static void __init register_insn_emulation_sysctl(struct ctl_table *table)
+static void __init register_insn_emulation_sysctl(void)
 {
 	unsigned long flags;
 	int i = 0;
@@ -262,8 +254,7 @@ static void __init register_insn_emulation_sysctl(struct ctl_table *table)
 	}
 	raw_spin_unlock_irqrestore(&insn_emulation_lock, flags);
 
-	table->child = insns_sysctl;
-	register_sysctl_table(table);
+	register_sysctl("abi", insns_sysctl);
 }
 
 /*
@@ -644,7 +635,7 @@ static int __init armv8_deprecated_init(void)
 	cpuhp_setup_state_nocalls(CPUHP_AP_ARM64_ISNDEP_STARTING,
 				  "arm64/isndep:starting",
 				  run_all_insn_set_hw_mode, NULL);
-	register_insn_emulation_sysctl(ctl_abi);
+	register_insn_emulation_sysctl();
 
 	return 0;
 }
-- 
2.1.4

[PATCH 05/27] arm64: fpsimd: Simplify uses of {set,clear}_ti_thread_flag()

[Dave Martin]

The existing FPSIMD context switch code contains a couple of
instances of {set,clear}_ti_thread(task_thread_info(task)).  Since
there are thread flag manipulators that operate directly on
task_struct, this verbosity isn't strictly needed.

For consistency, this patch simplifies the affected calls.  This
should have no impact on behaviour.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/kernel/fpsimd.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
index 138fcfa..9c1f268e 100644
--- a/arch/arm64/kernel/fpsimd.c
+++ b/arch/arm64/kernel/fpsimd.c
@@ -159,11 +159,9 @@ void fpsimd_thread_switch(struct task_struct *next)
 
 		if (__this_cpu_read(fpsimd_last_state) == st
 		    && st->cpu == smp_processor_id())
-			clear_ti_thread_flag(task_thread_info(next),
-					     TIF_FOREIGN_FPSTATE);
+			clear_tsk_thread_flag(next, TIF_FOREIGN_FPSTATE);
 		else
-			set_ti_thread_flag(task_thread_info(next),
-					   TIF_FOREIGN_FPSTATE);
+			set_tsk_thread_flag(next, TIF_FOREIGN_FPSTATE);
 	}
 }
 
-- 
2.1.4

[PATCH 06/27] arm64/sve: System register and exception syndrome definitions

[Dave Martin]

The SVE architecture adds some system registers, ID register fields
and a dedicated ESR exception class.

This patch adds the appropriate definitions that will be needed by
the kernel.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/include/asm/esr.h     |  3 ++-
 arch/arm64/include/asm/kvm_arm.h |  1 +
 arch/arm64/include/asm/sysreg.h  | 16 ++++++++++++++++
 arch/arm64/kernel/traps.c        |  1 +
 4 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h
index 8cabd57..813629e 100644
--- a/arch/arm64/include/asm/esr.h
+++ b/arch/arm64/include/asm/esr.h
@@ -43,7 +43,8 @@
 #define ESR_ELx_EC_HVC64	(0x16)
 #define ESR_ELx_EC_SMC64	(0x17)
 #define ESR_ELx_EC_SYS64	(0x18)
-/* Unallocated EC: 0x19 - 0x1E */
+#define ESR_ELx_EC_SVE		(0x19)
+/* Unallocated EC: 0x1A - 0x1E */
 #define ESR_ELx_EC_IMP_DEF	(0x1f)
 #define ESR_ELx_EC_IABT_LOW	(0x20)
 #define ESR_ELx_EC_IABT_CUR	(0x21)
diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
index 61d694c..dbf0537 100644
--- a/arch/arm64/include/asm/kvm_arm.h
+++ b/arch/arm64/include/asm/kvm_arm.h
@@ -185,6 +185,7 @@
 #define CPTR_EL2_TCPAC	(1 << 31)
 #define CPTR_EL2_TTA	(1 << 20)
 #define CPTR_EL2_TFP	(1 << CPTR_EL2_TFP_SHIFT)
+#define CPTR_EL2_TZ	(1 << 8)
 #define CPTR_EL2_DEFAULT	0x000033ff
 
 /* Hyp Debug Configuration Register bits */
diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
index 248339e..2d259e8 100644
--- a/arch/arm64/include/asm/sysreg.h
+++ b/arch/arm64/include/asm/sysreg.h
@@ -145,6 +145,7 @@
 
 #define SYS_ID_AA64PFR0_EL1		sys_reg(3, 0, 0, 4, 0)
 #define SYS_ID_AA64PFR1_EL1		sys_reg(3, 0, 0, 4, 1)
+#define SYS_ID_AA64ZFR0_EL1		sys_reg(3, 0, 0, 4, 4)
 
 #define SYS_ID_AA64DFR0_EL1		sys_reg(3, 0, 0, 5, 0)
 #define SYS_ID_AA64DFR1_EL1		sys_reg(3, 0, 0, 5, 1)
@@ -160,6 +161,8 @@
 #define SYS_ACTLR_EL1			sys_reg(3, 0, 1, 0, 1)
 #define SYS_CPACR_EL1			sys_reg(3, 0, 1, 0, 2)
 
+#define SYS_ZCR_EL1			sys_reg(3, 0, 1, 2, 0)
+
 #define SYS_TTBR0_EL1			sys_reg(3, 0, 2, 0, 0)
 #define SYS_TTBR1_EL1			sys_reg(3, 0, 2, 0, 1)
 #define SYS_TCR_EL1			sys_reg(3, 0, 2, 0, 2)
@@ -250,6 +253,8 @@
 
 #define SYS_PMCCFILTR_EL0		sys_reg (3, 3, 14, 15, 7)
 
+#define SYS_ZCR_EL2			sys_reg(3, 4, 1, 2, 0)
+
 #define SYS_DACR32_EL2			sys_reg(3, 4, 3, 0, 0)
 #define SYS_IFSR32_EL2			sys_reg(3, 4, 5, 0, 1)
 #define SYS_FPEXC32_EL2			sys_reg(3, 4, 5, 3, 0)
@@ -331,6 +336,7 @@
 #define ID_AA64ISAR1_JSCVT_SHIFT	12
 
 /* id_aa64pfr0 */
+#define ID_AA64PFR0_SVE_SHIFT		32
 #define ID_AA64PFR0_GIC_SHIFT		24
 #define ID_AA64PFR0_ASIMD_SHIFT		20
 #define ID_AA64PFR0_FP_SHIFT		16
@@ -339,6 +345,7 @@
 #define ID_AA64PFR0_EL1_SHIFT		4
 #define ID_AA64PFR0_EL0_SHIFT		0
 
+#define ID_AA64PFR0_SVE			0x1
 #define ID_AA64PFR0_FP_NI		0xf
 #define ID_AA64PFR0_FP_SUPPORTED	0x0
 #define ID_AA64PFR0_ASIMD_NI		0xf
@@ -440,6 +447,15 @@
 #endif
 
 
+#define ZCR_ELx_LEN_SHIFT	0
+#define ZCR_ELx_LEN_SIZE	9
+#define ZCR_ELx_LEN_MASK	0x1ff
+
+#define CPACR_EL1_ZEN_EL1EN	(1 << 16)
+#define CPACR_EL1_ZEN_EL0EN	(1 << 17)
+#define CPACR_EL1_ZEN		(CPACR_EL1_ZEN_EL1EN | CPACR_EL1_ZEN_EL0EN)
+
+
 /* Safe value for MPIDR_EL1: Bit31:RES1, Bit30:U:0, Bit24:MT:0 */
 #define SYS_MPIDR_SAFE_VAL		(1UL << 31)
 
diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
index 0f047e9..8964795 100644
--- a/arch/arm64/kernel/traps.c
+++ b/arch/arm64/kernel/traps.c
@@ -621,6 +621,7 @@ static const char *esr_class_str[] = {
 	[ESR_ELx_EC_HVC64]		= "HVC (AArch64)",
 	[ESR_ELx_EC_SMC64]		= "SMC (AArch64)",
 	[ESR_ELx_EC_SYS64]		= "MSR/MRS (AArch64)",
+	[ESR_ELx_EC_SVE]		= "SVE",
 	[ESR_ELx_EC_IMP_DEF]		= "EL3 IMP DEF",
 	[ESR_ELx_EC_IABT_LOW]		= "IABT (lower EL)",
 	[ESR_ELx_EC_IABT_CUR]		= "IABT (current EL)",
-- 
2.1.4

[PATCH 07/27] arm64/sve: Low-level SVE architectural state manipulation functions

[Dave Martin]

Manipulating the SVE architectural state, including the vector and
predicate registers, first-fault register and the vector length,
requires the use of dedicated instructions added by SVE.

This patch adds suitable assembly functions for saving and
restoring the SVE registers and querying the vector length.
Setting of the vector length is done as part of register restore.

Since people building kernels may not all get an SVE-enabled
toolchain for a while, this patch uses macros that generate
explicit opcodes in place of assembler mnemonics.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/include/asm/fpsimd.h       |   5 ++
 arch/arm64/include/asm/fpsimdmacros.h | 137 ++++++++++++++++++++++++++++++++++
 arch/arm64/kernel/entry-fpsimd.S      |  17 +++++
 3 files changed, 159 insertions(+)

diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
index 410c481..026a7c7 100644
--- a/arch/arm64/include/asm/fpsimd.h
+++ b/arch/arm64/include/asm/fpsimd.h
@@ -67,6 +67,11 @@ extern void fpsimd_update_current_state(struct fpsimd_state *state);
 
 extern void fpsimd_flush_task_state(struct task_struct *target);
 
+extern void sve_save_state(void *state, u32 *pfpsr);
+extern void sve_load_state(void const *state, u32 const *pfpsr,
+			   unsigned long vq_minus_1);
+extern unsigned int sve_get_vl(void);
+
 /* For use by EFI runtime services calls only */
 extern void __efi_fpsimd_begin(void);
 extern void __efi_fpsimd_end(void);
diff --git a/arch/arm64/include/asm/fpsimdmacros.h b/arch/arm64/include/asm/fpsimdmacros.h
index 0f5fdd3..5023064 100644
--- a/arch/arm64/include/asm/fpsimdmacros.h
+++ b/arch/arm64/include/asm/fpsimdmacros.h
@@ -75,3 +75,140 @@
 	ldr	w\tmpnr, [\state, #16 * 2 + 4]
 	fpsimd_restore_fpcr x\tmpnr, \state
 .endm
+
+.macro _check_general_reg nr
+	.if (\nr) < 0 || (\nr) > 30
+		.error "Bad register number \nr."
+	.endif
+.endm
+
+.macro _sve_check_zreg znr
+	.if (\znr) < 0 || (\znr) > 31
+		.error "Bad Scalable Vector Extension vector register number \znr."
+	.endif
+.endm
+
+.macro _sve_check_preg pnr
+	.if (\pnr) < 0 || (\pnr) > 15
+		.error "Bad Scalable Vector Extension predicate register number \pnr."
+	.endif
+.endm
+
+.macro _check_num n, min, max
+	.if (\n) < (\min) || (\n) > (\max)
+		.error "Number \n out of range [\min,\max]"
+	.endif
+.endm
+
+.macro _sve_str_v nz, nxbase, offset=0
+	_sve_check_zreg \nz
+	_check_general_reg \nxbase
+	_check_num (\offset), -0x100, 0xff
+	.inst	0xe5804000			\
+		| (\nz)				\
+		| ((\nxbase) << 5)		\
+		| (((\offset) & 7) << 10)	\
+		| (((\offset) & 0x1f8) << 13)
+.endm
+
+.macro _sve_ldr_v nz, nxbase, offset=0
+	_sve_check_zreg \nz
+	_check_general_reg \nxbase
+	_check_num (\offset), -0x100, 0xff
+	.inst	0x85804000			\
+		| (\nz)				\
+		| ((\nxbase) << 5)		\
+		| (((\offset) & 7) << 10)	\
+		| (((\offset) & 0x1f8) << 13)
+.endm
+
+.macro _sve_str_p np, nxbase, offset=0
+	_sve_check_preg \np
+	_check_general_reg \nxbase
+	_check_num (\offset), -0x100, 0xff
+	.inst	0xe5800000			\
+		| (\np)				\
+		| ((\nxbase) << 5)		\
+		| (((\offset) & 7) << 10)	\
+		| (((\offset) & 0x1f8) << 13)
+.endm
+
+.macro _sve_ldr_p np, nxbase, offset=0
+	_sve_check_preg \np
+	_check_general_reg \nxbase
+	_check_num (\offset), -0x100, 0xff
+	.inst	0x85800000			\
+		| (\np)				\
+		| ((\nxbase) << 5)		\
+		| (((\offset) & 7) << 10)	\
+		| (((\offset) & 0x1f8) << 13)
+.endm
+
+.macro _sve_rdvl nx, imm
+	_check_general_reg \nx
+	_check_num (\imm), -0x20, 0x1f
+	.inst	0x04bf5000			\
+		| (\nx)				\
+		| (((\imm) & 0x3f) << 5)
+.endm
+
+.macro _sve_rdffr np
+	_sve_check_preg \np
+	.inst	0x2519f000			\
+		| (\np)
+.endm
+
+.macro _sve_wrffr np
+	_sve_check_preg \np
+	.inst	0x25289000			\
+		| ((\np) << 5)
+.endm
+
+.macro __for from:req, to:req
+	.if (\from) == (\to)
+		_for__body \from
+	.else
+		__for \from, (\from) + ((\to) - (\from)) / 2
+		__for (\from) + ((\to) - (\from)) / 2 + 1, \to
+	.endif
+.endm
+
+.macro _for var:req, from:req, to:req, insn:vararg
+	.macro _for__body \var:req
+		\insn
+	.endm
+
+	__for \from, \to
+
+	.purgem _for__body
+.endm
+
+.macro sve_save nxbase, xpfpsr, nxtmp
+ _for n, 0, 31,	_sve_str_v	\n, \nxbase, \n - 34
+ _for n, 0, 15,	_sve_str_p	\n, \nxbase, \n - 16
+		_sve_rdffr	0
+		_sve_str_p	0, \nxbase
+		_sve_ldr_p	0, \nxbase, -16
+
+		mrs		x\nxtmp, fpsr
+		str		w\nxtmp, [\xpfpsr]
+		mrs		x\nxtmp, fpcr
+		str		w\nxtmp, [\xpfpsr, #4]
+.endm
+
+.macro sve_load nxbase, xpfpsr, xvqminus1, nxtmp
+		mrs_s		x\nxtmp, SYS_ZCR_EL1
+		bic		x\nxtmp, x\nxtmp, ZCR_ELx_LEN_MASK
+		orr		x\nxtmp, x\nxtmp, \xvqminus1
+		msr_s		SYS_ZCR_EL1, x\nxtmp	// self-synchronising
+
+ _for n, 0, 31,	_sve_ldr_v	\n, \nxbase, \n - 34
+		_sve_ldr_p	0, \nxbase
+		_sve_wrffr	0
+ _for n, 0, 15,	_sve_ldr_p	\n, \nxbase, \n - 16
+
+		ldr		w\nxtmp, [\xpfpsr]
+		msr		fpsr, x\nxtmp
+		ldr		w\nxtmp, [\xpfpsr, #4]
+		msr		fpcr, x\nxtmp
+.endm
diff --git a/arch/arm64/kernel/entry-fpsimd.S b/arch/arm64/kernel/entry-fpsimd.S
index 6a27cd6..73f17bf 100644
--- a/arch/arm64/kernel/entry-fpsimd.S
+++ b/arch/arm64/kernel/entry-fpsimd.S
@@ -41,3 +41,20 @@ ENTRY(fpsimd_load_state)
 	fpsimd_restore x0, 8
 	ret
 ENDPROC(fpsimd_load_state)
+
+#ifdef CONFIG_ARM64_SVE
+ENTRY(sve_save_state)
+	sve_save 0, x1, 2
+	ret
+ENDPROC(sve_save_state)
+
+ENTRY(sve_load_state)
+	sve_load 0, x1, x2, 3
+	ret
+ENDPROC(sve_load_state)
+
+ENTRY(sve_get_vl)
+	_sve_rdvl	0, 1
+	ret
+ENDPROC(sve_get_vl)
+#endif /* CONFIG_ARM64_SVE */
-- 
2.1.4

[PATCH 08/27] arm64/sve: Kconfig update and conditional compilation support

[Dave Martin]

This patch adds CONFIG_ARM64_SVE to control building of SVE support
into the kernel, and adds a stub predicate system_supports_sve() to
control conditional compilation and runtime SVE support.

system_supports_sve() just returns false for now: it will be
replaced with a non-trivial implementation in a later patch, once
SVE support is complete enough to be enabled safely.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/Kconfig                  | 11 +++++++++++
 arch/arm64/include/asm/cpufeature.h |  5 +++++
 2 files changed, 16 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index fdc302e..fbf7a5c 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -962,6 +962,17 @@ config ARM64_UAO
 
 endmenu
 
+config ARM64_SVE
+	bool "ARM Scalable Vector Extension support"
+	default y
+	help
+	  The Scalable Vector Extension (SVE) is an extension to the AArch64
+	  execution state which complements and extends the SIMD functionality
+	  of the base architecture to support much larger vectors and to enable
+	  additional vectorisation opportunities.
+
+	  To enable use of this extension on CPUs that implement it, say Y.
+
 config ARM64_MODULE_CMODEL_LARGE
 	bool
 
diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index 428ee1f..4ea3441 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -262,6 +262,11 @@ static inline bool system_uses_ttbr0_pan(void)
 		!cpus_have_const_cap(ARM64_HAS_PAN);
 }
 
+static inline bool system_supports_sve(void)
+{
+	return false;
+}
+
 #endif /* __ASSEMBLY__ */
 
 #endif
-- 
2.1.4

[PATCH 09/27] arm64/sve: Signal frame and context structure definition

[Dave Martin]

This patch defines the representation that will be used for the SVE
register state in the signal frame, and implements support for
saving and restoring the SVE registers around signals.

The same layout will also be used for the in-kernel task state.

Due to the variability of the SVE vector length, it is not possible
to define a fixed C struct to describe all the registers.  Instead,
Macros are defined in sigcontext.h to facilitate access to the
parts of the structure.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/include/uapi/asm/sigcontext.h | 113 ++++++++++++++++++++++++++++++-
 1 file changed, 112 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/uapi/asm/sigcontext.h b/arch/arm64/include/uapi/asm/sigcontext.h
index f0a76b9..0533bdf 100644
--- a/arch/arm64/include/uapi/asm/sigcontext.h
+++ b/arch/arm64/include/uapi/asm/sigcontext.h
@@ -16,6 +16,8 @@
 #ifndef _UAPI__ASM_SIGCONTEXT_H
 #define _UAPI__ASM_SIGCONTEXT_H
 
+#ifndef __ASSEMBLY__
+
 #include <linux/types.h>
 
 /*
@@ -41,10 +43,11 @@ struct sigcontext {
  *
  *	0x210		fpsimd_context
  *	 0x10		esr_context
+ *	0x8a0		sve_context (vl <= 64) (optional)
  *	 0x20		extra_context (optional)
  *	 0x10		terminator (null _aarch64_ctx)
  *
- *	0xdb0		(reserved for future allocation)
+ *	0x510		(reserved for future allocation)
  *
  * New records that can exceed this space need to be opt-in for userspace, so
  * that an expanded signal frame is not generated unexpectedly.  The mechanism
@@ -116,4 +119,112 @@ struct extra_context {
 	__u32 __reserved[3];
 };
 
+#define SVE_MAGIC	0x53564501
+
+struct sve_context {
+	struct _aarch64_ctx head;
+	__u16 vl;
+	__u16 __reserved[3];
+};
+
+#endif /* !__ASSEMBLY__ */
+
+/*
+ * The SVE architecture leaves space for future expansion of the
+ * vector length beyond its initial architectural limit of 2048 bits
+ * (16 quadwords).
+ */
+#define SVE_VQ_MIN		1
+#define SVE_VQ_MAX		0x200
+
+#define SVE_VL_MIN		(SVE_VQ_MIN * 0x10)
+#define SVE_VL_MAX		(SVE_VQ_MAX * 0x10)
+
+#define SVE_NUM_ZREGS		32
+#define SVE_NUM_PREGS		16
+
+#define sve_vl_valid(vl) \
+	((vl) % 0x10 == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
+#define sve_vq_from_vl(vl)	((vl) / 0x10)
+#define sve_vl_from_vq(vq)	((vq) * 0x10)
+
+/*
+ * If the SVE registers are currently live for the thread at signal delivery,
+ * sve_context.head.size >=
+ *	SVE_SIG_CONTEXT_SIZE(sve_vq_from_vl(sve_context.vl))
+ * and the register data may be accessed using the SVE_SIG_*() macros.
+ *
+ * If sve_context.head.size <
+ *	SVE_SIG_CONTEXT_SIZE(sve_vq_from_vl(sve_context.vl)),
+ * the SVE registers were not live for the thread and no register data
+ * is included: in this case, the SVE_SIG_*() macros should not be
+ * used except for this check.
+ *
+ * The same convention applies when returning from a signal: a caller
+ * will need to remove or resize the sve_context block if it wants to
+ * make the SVE registers live when they were previously non-live or
+ * vice-versa.  This may require the the caller to allocate fresh
+ * memory and/or move other context blocks in the signal frame.
+ *
+ * Changing the vector length during signal return is not permitted:
+ * sve_context.vl must equal the thread's current vector length when
+ * doing a sigreturn.
+ *
+ *
+ * Note: for all these macros, the "vq" argument denotes the SVE
+ * vector length in quadwords (i.e., units of 128 bits).
+ *
+ * The correct way to obtain vq is to use sve_vq_from_vl(vl).  The
+ * result is valid if and only if sve_vl_valid(vl) is true.  This is
+ * guaranteed for a struct sve_context written by the kernel.
+ *
+ *
+ * Additional macros describe the contents and layout of the payload.
+ * For each, SVE_SIG_x_OFFSET(args) is the start offset relative to
+ * the start of struct sve_context, and SVE_SIG_x_SIZE(args) is the
+ * size in bytes:
+ *
+ *	x	type				description
+ *	-	----				-----------
+ *	REGS					the entire SVE context
+ *
+ *	ZREGS	__uint128_t[SVE_NUM_ZREGS][vq]	all Z-registers
+ *	ZREG	__uint128_t[vq]			individual Z-register Zn
+ *
+ *	PREGS	uint16_t[SVE_NUM_PREGS][vq]	all P-registers
+ *	PREG	uint16_t[vq]			individual P-register Pn
+ *
+ *	FFR	uint16_t[vq]			first-fault status register
+ *
+ * Additional data might be appended in the future.
+ */
+
+#define SVE_SIG_ZREG_SIZE(vq)	((__u32)(vq) * 16)
+#define SVE_SIG_PREG_SIZE(vq)	((__u32)(vq) * 2)
+#define SVE_SIG_FFR_SIZE(vq)	SVE_SIG_PREG_SIZE(vq)
+
+#define SVE_SIG_REGS_OFFSET	((sizeof(struct sve_context) + 15) / 16 * 16)
+
+#define SVE_SIG_ZREGS_OFFSET	SVE_SIG_REGS_OFFSET
+#define SVE_SIG_ZREG_OFFSET(vq, n) \
+	(SVE_SIG_ZREGS_OFFSET + SVE_SIG_ZREG_SIZE(vq) * (n))
+#define SVE_SIG_ZREGS_SIZE(vq) \
+	(SVE_SIG_ZREG_OFFSET(vq, SVE_NUM_ZREGS) - SVE_SIG_ZREGS_OFFSET)
+
+#define SVE_SIG_PREGS_OFFSET(vq) \
+	(SVE_SIG_ZREGS_OFFSET + SVE_SIG_ZREGS_SIZE(vq))
+#define SVE_SIG_PREG_OFFSET(vq, n) \
+	(SVE_SIG_PREGS_OFFSET(vq) + SVE_SIG_PREG_SIZE(vq) * (n))
+#define SVE_SIG_PREGS_SIZE(vq) \
+	(SVE_SIG_PREG_OFFSET(vq, SVE_NUM_PREGS) - SVE_SIG_PREGS_OFFSET(vq))
+
+#define SVE_SIG_FFR_OFFSET(vq) \
+	(SVE_SIG_PREGS_OFFSET(vq) + SVE_SIG_PREGS_SIZE(vq))
+
+#define SVE_SIG_REGS_SIZE(vq) \
+	(SVE_SIG_FFR_OFFSET(vq) + SVE_SIG_FFR_SIZE(vq) - SVE_SIG_REGS_OFFSET)
+
+#define SVE_SIG_CONTEXT_SIZE(vq) (SVE_SIG_REGS_OFFSET + SVE_SIG_REGS_SIZE(vq))
+
+
 #endif /* _UAPI__ASM_SIGCONTEXT_H */
-- 
2.1.4

[PATCH 09/27] arm64/sve: Signal frame and context structure definition

[Dave Martin]

This patch defines the representation that will be used for the SVE
register state in the signal frame, and implements support for
saving and restoring the SVE registers around signals.

The same layout will also be used for the in-kernel task state.

Due to the variability of the SVE vector length, it is not possible
to define a fixed C struct to describe all the registers.  Instead,
Macros are defined in sigcontext.h to facilitate access to the
parts of the structure.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/include/uapi/asm/sigcontext.h | 113 ++++++++++++++++++++++++++++++-
 1 file changed, 112 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/uapi/asm/sigcontext.h b/arch/arm64/include/uapi/asm/sigcontext.h
index f0a76b9..0533bdf 100644
--- a/arch/arm64/include/uapi/asm/sigcontext.h
+++ b/arch/arm64/include/uapi/asm/sigcontext.h
@@ -16,6 +16,8 @@
 #ifndef _UAPI__ASM_SIGCONTEXT_H
 #define _UAPI__ASM_SIGCONTEXT_H
 
+#ifndef __ASSEMBLY__
+
 #include <linux/types.h>
 
 /*
@@ -41,10 +43,11 @@ struct sigcontext {
  *
  *	0x210		fpsimd_context
  *	 0x10		esr_context
+ *	0x8a0		sve_context (vl <= 64) (optional)
  *	 0x20		extra_context (optional)
  *	 0x10		terminator (null _aarch64_ctx)
  *
- *	0xdb0		(reserved for future allocation)
+ *	0x510		(reserved for future allocation)
  *
  * New records that can exceed this space need to be opt-in for userspace, so
  * that an expanded signal frame is not generated unexpectedly.  The mechanism
@@ -116,4 +119,112 @@ struct extra_context {
 	__u32 __reserved[3];
 };
 
+#define SVE_MAGIC	0x53564501
+
+struct sve_context {
+	struct _aarch64_ctx head;
+	__u16 vl;
+	__u16 __reserved[3];
+};
+
+#endif /* !__ASSEMBLY__ */
+
+/*
+ * The SVE architecture leaves space for future expansion of the
+ * vector length beyond its initial architectural limit of 2048 bits
+ * (16 quadwords).
+ */
+#define SVE_VQ_MIN		1
+#define SVE_VQ_MAX		0x200
+
+#define SVE_VL_MIN		(SVE_VQ_MIN * 0x10)
+#define SVE_VL_MAX		(SVE_VQ_MAX * 0x10)
+
+#define SVE_NUM_ZREGS		32
+#define SVE_NUM_PREGS		16
+
+#define sve_vl_valid(vl) \
+	((vl) % 0x10 == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
+#define sve_vq_from_vl(vl)	((vl) / 0x10)
+#define sve_vl_from_vq(vq)	((vq) * 0x10)
+
+/*
+ * If the SVE registers are currently live for the thread at signal delivery,
+ * sve_context.head.size >=
+ *	SVE_SIG_CONTEXT_SIZE(sve_vq_from_vl(sve_context.vl))
+ * and the register data may be accessed using the SVE_SIG_*() macros.
+ *
+ * If sve_context.head.size <
+ *	SVE_SIG_CONTEXT_SIZE(sve_vq_from_vl(sve_context.vl)),
+ * the SVE registers were not live for the thread and no register data
+ * is included: in this case, the SVE_SIG_*() macros should not be
+ * used except for this check.
+ *
+ * The same convention applies when returning from a signal: a caller
+ * will need to remove or resize the sve_context block if it wants to
+ * make the SVE registers live when they were previously non-live or
+ * vice-versa.  This may require the the caller to allocate fresh
+ * memory and/or move other context blocks in the signal frame.
+ *
+ * Changing the vector length during signal return is not permitted:
+ * sve_context.vl must equal the thread's current vector length when
+ * doing a sigreturn.
+ *
+ *
+ * Note: for all these macros, the "vq" argument denotes the SVE
+ * vector length in quadwords (i.e., units of 128 bits).
+ *
+ * The correct way to obtain vq is to use sve_vq_from_vl(vl).  The
+ * result is valid if and only if sve_vl_valid(vl) is true.  This is
+ * guaranteed for a struct sve_context written by the kernel.
+ *
+ *
+ * Additional macros describe the contents and layout of the payload.
+ * For each, SVE_SIG_x_OFFSET(args) is the start offset relative to
+ * the start of struct sve_context, and SVE_SIG_x_SIZE(args) is the
+ * size in bytes:
+ *
+ *	x	type				description
+ *	-	----				-----------
+ *	REGS					the entire SVE context
+ *
+ *	ZREGS	__uint128_t[SVE_NUM_ZREGS][vq]	all Z-registers
+ *	ZREG	__uint128_t[vq]			individual Z-register Zn
+ *
+ *	PREGS	uint16_t[SVE_NUM_PREGS][vq]	all P-registers
+ *	PREG	uint16_t[vq]			individual P-register Pn
+ *
+ *	FFR	uint16_t[vq]			first-fault status register
+ *
+ * Additional data might be appended in the future.
+ */
+
+#define SVE_SIG_ZREG_SIZE(vq)	((__u32)(vq) * 16)
+#define SVE_SIG_PREG_SIZE(vq)	((__u32)(vq) * 2)
+#define SVE_SIG_FFR_SIZE(vq)	SVE_SIG_PREG_SIZE(vq)
+
+#define SVE_SIG_REGS_OFFSET	((sizeof(struct sve_context) + 15) / 16 * 16)
+
+#define SVE_SIG_ZREGS_OFFSET	SVE_SIG_REGS_OFFSET
+#define SVE_SIG_ZREG_OFFSET(vq, n) \
+	(SVE_SIG_ZREGS_OFFSET + SVE_SIG_ZREG_SIZE(vq) * (n))
+#define SVE_SIG_ZREGS_SIZE(vq) \
+	(SVE_SIG_ZREG_OFFSET(vq, SVE_NUM_ZREGS) - SVE_SIG_ZREGS_OFFSET)
+
+#define SVE_SIG_PREGS_OFFSET(vq) \
+	(SVE_SIG_ZREGS_OFFSET + SVE_SIG_ZREGS_SIZE(vq))
+#define SVE_SIG_PREG_OFFSET(vq, n) \
+	(SVE_SIG_PREGS_OFFSET(vq) + SVE_SIG_PREG_SIZE(vq) * (n))
+#define SVE_SIG_PREGS_SIZE(vq) \
+	(SVE_SIG_PREG_OFFSET(vq, SVE_NUM_PREGS) - SVE_SIG_PREGS_OFFSET(vq))
+
+#define SVE_SIG_FFR_OFFSET(vq) \
+	(SVE_SIG_PREGS_OFFSET(vq) + SVE_SIG_PREGS_SIZE(vq))
+
+#define SVE_SIG_REGS_SIZE(vq) \
+	(SVE_SIG_FFR_OFFSET(vq) + SVE_SIG_FFR_SIZE(vq) - SVE_SIG_REGS_OFFSET)
+
+#define SVE_SIG_CONTEXT_SIZE(vq) (SVE_SIG_REGS_OFFSET + SVE_SIG_REGS_SIZE(vq))
+
+
 #endif /* _UAPI__ASM_SIGCONTEXT_H */
-- 
2.1.4

[PATCH 10/27] arm64/sve: Low-level CPU setup

[Dave Martin]

To enable the kernel to use SVE, all SVE traps from EL1 must be
disabled.  To take maximum advantage of the hardware, the full
available vector length also needs to be enabled for EL1 by
programming ZCR_EL2.LEN.  (The kernel will program ZCR_EL1.LEN as
required, but this cannot override the limit set by ZCR_EL2.)

In advance of full SVE support being implemented for userspace, it
also necessary to ensure that SVE traps from EL0 are enabled.

This patch makes the appropriate changes to the primary and
secondary CPU initialisation code.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/kernel/head.S | 13 ++++++++++++-
 arch/arm64/mm/proc.S     | 14 ++++++++++++--
 2 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 973df7d..0ae1713 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -514,8 +514,19 @@ CPU_LE(	movk	x0, #0x30d0, lsl #16	)	// Clear EE and E0E on LE systems
 	mov	x0, #0x33ff
 	msr	cptr_el2, x0			// Disable copro. traps to EL2
 
+	/* SVE register access */
+	mrs	x1, id_aa64pfr0_el1
+	ubfx	x1, x1, #ID_AA64PFR0_SVE_SHIFT, #4
+	cbz	x1, 7f
+
+	bic	x0, x0, #CPTR_EL2_TZ		// Also disable SVE traps
+	msr	cptr_el2, x0			// Disable copro. traps to EL2
+	isb
+	mov	x1, #ZCR_ELx_LEN_MASK		// SVE: Enable full vector
+	msr_s	SYS_ZCR_EL2, x1			// length for EL1.
+
 	/* Hypervisor stub */
-	adr_l	x0, __hyp_stub_vectors
+7:	adr_l	x0, __hyp_stub_vectors
 	msr	vbar_el2, x0
 
 	/* spsr */
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index 877d42f..dd22ef2 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -27,6 +27,7 @@
 #include <asm/pgtable-hwdef.h>
 #include <asm/cpufeature.h>
 #include <asm/alternative.h>
+#include <asm/sysreg.h>
 
 #ifdef CONFIG_ARM64_64K_PAGES
 #define TCR_TG_FLAGS	TCR_TG0_64K | TCR_TG1_64K
@@ -186,8 +187,17 @@ ENTRY(__cpu_setup)
 	tlbi	vmalle1				// Invalidate local TLB
 	dsb	nsh
 
-	mov	x0, #3 << 20
-	msr	cpacr_el1, x0			// Enable FP/ASIMD
+	mov	x0, #3 << 20			// FEN
+
+	/* SVE */
+	mrs	x5, id_aa64pfr0_el1
+	ubfx	x5, x5, #ID_AA64PFR0_SVE_SHIFT, #4
+	cbz	x5, 1f
+
+	bic	x0, x0, #CPACR_EL1_ZEN
+	orr	x0, x0, #CPACR_EL1_ZEN_EL1EN	// SVE: trap for EL0, not EL1
+1:	msr	cpacr_el1, x0			// Enable FP/ASIMD
+
 	mov	x0, #1 << 12			// Reset mdscr_el1 and disable
 	msr	mdscr_el1, x0			// access to the DCC from EL0
 	isb					// Unmask debug exceptions now,
-- 
2.1.4

[PATCH 10/27] arm64/sve: Low-level CPU setup

[Dave Martin]

To enable the kernel to use SVE, all SVE traps from EL1 must be
disabled.  To take maximum advantage of the hardware, the full
available vector length also needs to be enabled for EL1 by
programming ZCR_EL2.LEN.  (The kernel will program ZCR_EL1.LEN as
required, but this cannot override the limit set by ZCR_EL2.)

In advance of full SVE support being implemented for userspace, it
also necessary to ensure that SVE traps from EL0 are enabled.

This patch makes the appropriate changes to the primary and
secondary CPU initialisation code.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/kernel/head.S | 13 ++++++++++++-
 arch/arm64/mm/proc.S     | 14 ++++++++++++--
 2 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 973df7d..0ae1713 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -514,8 +514,19 @@ CPU_LE(	movk	x0, #0x30d0, lsl #16	)	// Clear EE and E0E on LE systems
 	mov	x0, #0x33ff
 	msr	cptr_el2, x0			// Disable copro. traps to EL2
 
+	/* SVE register access */
+	mrs	x1, id_aa64pfr0_el1
+	ubfx	x1, x1, #ID_AA64PFR0_SVE_SHIFT, #4
+	cbz	x1, 7f
+
+	bic	x0, x0, #CPTR_EL2_TZ		// Also disable SVE traps
+	msr	cptr_el2, x0			// Disable copro. traps to EL2
+	isb
+	mov	x1, #ZCR_ELx_LEN_MASK		// SVE: Enable full vector
+	msr_s	SYS_ZCR_EL2, x1			// length for EL1.
+
 	/* Hypervisor stub */
-	adr_l	x0, __hyp_stub_vectors
+7:	adr_l	x0, __hyp_stub_vectors
 	msr	vbar_el2, x0
 
 	/* spsr */
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index 877d42f..dd22ef2 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -27,6 +27,7 @@
 #include <asm/pgtable-hwdef.h>
 #include <asm/cpufeature.h>
 #include <asm/alternative.h>
+#include <asm/sysreg.h>
 
 #ifdef CONFIG_ARM64_64K_PAGES
 #define TCR_TG_FLAGS	TCR_TG0_64K | TCR_TG1_64K
@@ -186,8 +187,17 @@ ENTRY(__cpu_setup)
 	tlbi	vmalle1				// Invalidate local TLB
 	dsb	nsh
 
-	mov	x0, #3 << 20
-	msr	cpacr_el1, x0			// Enable FP/ASIMD
+	mov	x0, #3 << 20			// FEN
+
+	/* SVE */
+	mrs	x5, id_aa64pfr0_el1
+	ubfx	x5, x5, #ID_AA64PFR0_SVE_SHIFT, #4
+	cbz	x5, 1f
+
+	bic	x0, x0, #CPACR_EL1_ZEN
+	orr	x0, x0, #CPACR_EL1_ZEN_EL1EN	// SVE: trap for EL0, not EL1
+1:	msr	cpacr_el1, x0			// Enable FP/ASIMD
+
 	mov	x0, #1 << 12			// Reset mdscr_el1 and disable
 	msr	mdscr_el1, x0			// access to the DCC from EL0
 	isb					// Unmask debug exceptions now,
-- 
2.1.4

[PATCH 11/27] arm64/sve: Core task context handling

[Dave Martin]

This patch adds the core support for switching and managing the SVE
architectural state of user tasks.

Calls to the existing FPSIMD low-level save/restore functions are
factored out as new functions task_fpsimd_{save,load}(), since SVE
now dynamically may or may not need to be handled at these points
depending on the kernel configuration, hardware features discovered
at boot, and the runtime state of the task.  To make these
decisions as fast as possible, const cpucaps are used where
feasible, via the system_supports_sve() helper.

The SVE registers are only tracked for threads that have explicitly
used SVE, indicated by the new thread flag TIF_SVE.  Otherwise, the
FPSIMD view of the architectural state is stored in
thread.fpsimd_state as usual.

When in use, the SVE registers are not stored directly in
thread_struct due to their potentially large and variable size.
Because the task_struct slab allocator must be configured very
early during kernel boot, it is also tricky to configure it
correctly to match the maximum vector length provided by the
hardware, since this depends on examining secondary CPUs as well as
the primary.  Instead, a pointer sve_state in thread_struct points
to a dynamically allocated buffer containing the SVE register data,
and code is added to allocate, duplicate and free this buffer at
appropriate times.

TIF_SVE is set when taking an SVE access trap from userspace, if
suitable hardware support has been detected.  This enables SVE for
the thread: a subsequent return to userspace will disable the trap
accordingly.  If such a trap is taken without sufficient hardware
support, SIGILL is sent to the thread instead as if an undefined
instruction had been executed: this may happen if userspace tries
to use SVE in a system where not all CPUs support it for example.

The kernel may clear TIF_SVE and disable SVE for the thread
whenever an explicit syscall is made by userspace, though this is
considered an optimisation opportunity rather than a deterministic
guarantee: the kernel may not do this on every syscall, but it is
permitted to do so.  For backwards compatibility reasons and
conformance with the spirit of the base AArch64 procedure call
standard, the subset of the SVE register state that aliases the
FPSIMD registers is still preserved across a syscall even if this
happens.

TIF_SVE is also cleared, and SVE disabled, on exec: this is an
obvious slow path and a hint that we are running a new binary that
may not use SVE.

Code is added to sync data between thread.fpsimd_state and
thread.sve_state whenever enabling/disabling SVE, in a manner
consistent with the SVE architectural programmer's model.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/include/asm/fpsimd.h      |  19 +++
 arch/arm64/include/asm/processor.h   |   2 +
 arch/arm64/include/asm/thread_info.h |   1 +
 arch/arm64/include/asm/traps.h       |   2 +
 arch/arm64/kernel/entry.S            |  14 +-
 arch/arm64/kernel/fpsimd.c           | 241 ++++++++++++++++++++++++++++++++++-
 arch/arm64/kernel/process.c          |   6 +-
 arch/arm64/kernel/traps.c            |   4 +-
 8 files changed, 279 insertions(+), 10 deletions(-)

diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
index 026a7c7..72090a1 100644
--- a/arch/arm64/include/asm/fpsimd.h
+++ b/arch/arm64/include/asm/fpsimd.h
@@ -20,6 +20,8 @@
 
 #ifndef __ASSEMBLY__
 
+#include <linux/stddef.h>
+
 /*
  * FP/SIMD storage area has:
  *  - FPSR and FPCR
@@ -72,6 +74,23 @@ extern void sve_load_state(void const *state, u32 const *pfpsr,
 			   unsigned long vq_minus_1);
 extern unsigned int sve_get_vl(void);
 
+#ifdef CONFIG_ARM64_SVE
+
+extern size_t sve_state_size(struct task_struct const *task);
+
+extern void sve_alloc(struct task_struct *task);
+extern void fpsimd_release_thread(struct task_struct *task);
+extern void fpsimd_dup_sve(struct task_struct *dst,
+			   struct task_struct const *src);
+
+#else /* ! CONFIG_ARM64_SVE */
+
+static void __maybe_unused sve_alloc(struct task_struct *task) { }
+static void __maybe_unused fpsimd_release_thread(struct task_struct *task) { }
+static void __maybe_unused fpsimd_dup_sve(struct task_struct *dst,
+					  struct task_struct const *src) { }
+#endif /* ! CONFIG_ARM64_SVE */
+
 /* For use by EFI runtime services calls only */
 extern void __efi_fpsimd_begin(void);
 extern void __efi_fpsimd_end(void);
diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
index b7334f1..969feed 100644
--- a/arch/arm64/include/asm/processor.h
+++ b/arch/arm64/include/asm/processor.h
@@ -85,6 +85,8 @@ struct thread_struct {
 	unsigned long		tp2_value;
 #endif
 	struct fpsimd_state	fpsimd_state;
+	void			*sve_state;	/* SVE registers, if any */
+	u16			sve_vl;		/* SVE vector length */
 	unsigned long		fault_address;	/* fault info */
 	unsigned long		fault_code;	/* ESR_EL1 value */
 	struct debug_info	debug;		/* debugging */
diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h
index 46c3b93..1a4b30b 100644
--- a/arch/arm64/include/asm/thread_info.h
+++ b/arch/arm64/include/asm/thread_info.h
@@ -96,6 +96,7 @@ struct thread_info {
 #define TIF_RESTORE_SIGMASK	20
 #define TIF_SINGLESTEP		21
 #define TIF_32BIT		22	/* 32bit process */
+#define TIF_SVE			23	/* Scalable Vector Extension in use */
 
 #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
 #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
diff --git a/arch/arm64/include/asm/traps.h b/arch/arm64/include/asm/traps.h
index 02e9035..f058c07 100644
--- a/arch/arm64/include/asm/traps.h
+++ b/arch/arm64/include/asm/traps.h
@@ -34,6 +34,8 @@ struct undef_hook {
 
 void register_undef_hook(struct undef_hook *hook);
 void unregister_undef_hook(struct undef_hook *hook);
+void force_signal_inject(int signal, int code, struct pt_regs *regs,
+			 unsigned long address);
 
 void arm64_notify_segfault(struct pt_regs *regs, unsigned long addr);
 
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index cace76d..c33069c 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -524,6 +524,8 @@ el0_sync:
 	b.eq	el0_ia
 	cmp	x24, #ESR_ELx_EC_FP_ASIMD	// FP/ASIMD access
 	b.eq	el0_fpsimd_acc
+	cmp	x24, #ESR_ELx_EC_SVE		// SVE access
+	b.eq	el0_sve_acc
 	cmp	x24, #ESR_ELx_EC_FP_EXC64	// FP/ASIMD exception
 	b.eq	el0_fpsimd_exc
 	cmp	x24, #ESR_ELx_EC_SYS64		// configurable trap
@@ -622,9 +624,19 @@ el0_fpsimd_acc:
 	mov	x1, sp
 	bl	do_fpsimd_acc
 	b	ret_to_user
+el0_sve_acc:
+	/*
+	 * Scalable Vector Extension access
+	 */
+	enable_dbg
+	ct_user_exit
+	mov	x0, x25
+	mov	x1, sp
+	bl	do_sve_acc
+	b	ret_to_user
 el0_fpsimd_exc:
 	/*
-	 * Floating Point or Advanced SIMD exception
+	 * Floating Point, Advanced SIMD or SVE exception
 	 */
 	enable_dbg
 	ct_user_exit
diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
index 9c1f268e..37dd1b2 100644
--- a/arch/arm64/kernel/fpsimd.c
+++ b/arch/arm64/kernel/fpsimd.c
@@ -24,12 +24,17 @@
 #include <linux/init.h>
 #include <linux/percpu.h>
 #include <linux/preempt.h>
+#include <linux/ptrace.h>
 #include <linux/sched/signal.h>
 #include <linux/signal.h>
+#include <linux/slab.h>
 
 #include <asm/fpsimd.h>
 #include <asm/cputype.h>
 #include <asm/simd.h>
+#include <asm/sigcontext.h>
+#include <asm/sysreg.h>
+#include <asm/traps.h>
 
 #define FPEXC_IOF	(1 << 0)
 #define FPEXC_DZF	(1 << 1)
@@ -38,6 +43,10 @@
 #define FPEXC_IXF	(1 << 4)
 #define FPEXC_IDF	(1 << 7)
 
+/* Forward declarations for local functions used by both SVE and FPSIMD */
+static void task_fpsimd_load(void);
+static void task_fpsimd_save(void);
+
 /*
  * In order to reduce the number of times the FPSIMD state is needlessly saved
  * and restored, we need to keep track of two things:
@@ -99,6 +108,160 @@
  */
 static DEFINE_PER_CPU(struct fpsimd_state *, fpsimd_last_state);
 
+static void sve_free(struct task_struct *task)
+{
+	kfree(task->thread.sve_state);
+	task->thread.sve_state = NULL;
+}
+
+/* Offset of FFR in the SVE register dump */
+static size_t sve_ffr_offset(int vl)
+{
+	BUG_ON(!sve_vl_valid(vl));
+	return SVE_SIG_FFR_OFFSET(sve_vq_from_vl(vl)) - SVE_SIG_REGS_OFFSET;
+}
+
+static void *sve_pffr(struct task_struct *task)
+{
+	unsigned int vl = task->thread.sve_vl;
+
+	BUG_ON(!sve_vl_valid(vl) || !task->thread.sve_state);
+	return (char *)task->thread.sve_state + sve_ffr_offset(vl);
+}
+
+static u64 sve_cpacr_trap_on(u64 cpacr)
+{
+	return cpacr & ~(u64)CPACR_EL1_ZEN_EL0EN;
+}
+
+static u64 sve_cpacr_trap_off(u64 cpacr)
+{
+	return cpacr | CPACR_EL1_ZEN_EL0EN;
+}
+
+static void change_cpacr(u64 old, u64 new)
+{
+	if (old != new)
+		write_sysreg(new, CPACR_EL1);
+}
+
+#ifdef CONFIG_ARM64_SVE
+
+#define ZREG(sve_state, vq, n) ((char *)(sve_state) +		\
+	(SVE_SIG_ZREG_OFFSET(vq, n) - SVE_SIG_REGS_OFFSET))
+
+static void sve_to_fpsimd(struct task_struct *task)
+{
+	unsigned int vl = task->thread.sve_vl;
+	unsigned int vq;
+	void const *sst = task->thread.sve_state;
+	struct fpsimd_state *fst = &task->thread.fpsimd_state;
+	unsigned int i;
+
+	if (!system_supports_sve())
+		return;
+
+	BUG_ON(!sve_vl_valid(vl));
+	vq = sve_vq_from_vl(vl);
+
+	for (i = 0; i < 32; ++i)
+		memcpy(&fst->vregs[i], ZREG(sst, vq, i),
+		       sizeof(fst->vregs[i]));
+}
+
+static void fpsimd_to_sve(struct task_struct *task)
+{
+	unsigned int vl = task->thread.sve_vl;
+	unsigned int vq;
+	void *sst = task->thread.sve_state;
+	struct fpsimd_state const *fst = &task->thread.fpsimd_state;
+	unsigned int i;
+
+	if (!system_supports_sve())
+		return;
+
+	BUG_ON(!sve_vl_valid(vl));
+	vq = sve_vq_from_vl(vl);
+
+	for (i = 0; i < 32; ++i)
+		memcpy(ZREG(sst, vq, i), &fst->vregs[i],
+		       sizeof(fst->vregs[i]));
+}
+
+size_t sve_state_size(struct task_struct const *task)
+{
+	unsigned int vl = task->thread.sve_vl;
+
+	BUG_ON(!sve_vl_valid(vl));
+	return SVE_SIG_REGS_SIZE(sve_vq_from_vl(vl));
+}
+
+void sve_alloc(struct task_struct *task)
+{
+	if (task->thread.sve_state) {
+		memset(task->thread.sve_state, 0, sve_state_size(current));
+		return;
+	}
+
+	/* This is a small allocation (maximum ~8KB) and Should Not Fail. */
+	task->thread.sve_state =
+		kzalloc(sve_state_size(task), GFP_KERNEL);
+
+	/*
+	 * If future SVE revisions can have larger vectors though,
+	 * this may cease to be true:
+	 */
+	BUG_ON(!task->thread.sve_state);
+}
+
+/*
+ * Handle SVE state across fork():
+ *
+ * dst and src must not end up with aliases of the same sve_state.
+ * Because a task cannot fork except in a syscall, we can discard SVE
+ * state for dst here: reallocation will be deferred until dst tries
+ * to use SVE.
+ */
+void fpsimd_dup_sve(struct task_struct *dst, struct task_struct const *src)
+{
+	BUG_ON(!in_syscall(task_pt_regs(dst)));
+
+	if (test_and_clear_tsk_thread_flag(dst, TIF_SVE)) {
+		sve_to_fpsimd(dst);
+		dst->thread.sve_state = NULL;
+	}
+}
+
+void fpsimd_release_thread(struct task_struct *dead_task)
+{
+	sve_free(dead_task);
+}
+
+#endif /* CONFIG_ARM64_SVE */
+
+/*
+ * Trapped SVE access
+ */
+void do_sve_acc(unsigned int esr, struct pt_regs *regs)
+{
+	BUG_ON(is_compat_task());
+
+	/* Even if we chose not to use SVE, the hardware could still trap: */
+	if (unlikely(!system_supports_sve())) {
+		force_signal_inject(SIGILL, ILL_ILLOPC, regs, 0);
+		return;
+	}
+
+	task_fpsimd_save();
+
+	sve_alloc(current);
+	fpsimd_to_sve(current);
+	if (test_and_set_thread_flag(TIF_SVE))
+		WARN_ON(1); /* SVE access shouldn't have trapped */
+
+	task_fpsimd_load();
+}
+
 /*
  * Trapped FP/ASIMD access.
  */
@@ -135,6 +298,55 @@ void do_fpsimd_exc(unsigned int esr, struct pt_regs *regs)
 	send_sig_info(SIGFPE, &info, current);
 }
 
+static void task_fpsimd_load(void)
+{
+	if (system_supports_sve() && test_thread_flag(TIF_SVE)) {
+		unsigned int vl = current->thread.sve_vl;
+
+		BUG_ON(!sve_vl_valid(vl));
+		sve_load_state(sve_pffr(current),
+			       &current->thread.fpsimd_state.fpsr,
+			       sve_vq_from_vl(vl) - 1);
+	} else
+		fpsimd_load_state(&current->thread.fpsimd_state);
+
+	if (system_supports_sve()) {
+		u64 cpacr = read_sysreg(CPACR_EL1);
+		u64 new_cpacr;
+
+		/* Toggle SVE trapping for userspace if needed */
+		if (test_thread_flag(TIF_SVE))
+			new_cpacr = sve_cpacr_trap_off(cpacr);
+		else
+			new_cpacr = sve_cpacr_trap_on(cpacr);
+
+		change_cpacr(cpacr, new_cpacr);
+		/* Serialised by exception return to user */
+	}
+}
+
+static void task_fpsimd_save(void)
+{
+	if (system_supports_sve() &&
+	    in_syscall(current_pt_regs()) &&
+	    test_thread_flag(TIF_SVE)) {
+		u64 cpacr = read_sysreg(CPACR_EL1);
+
+		clear_thread_flag(TIF_SVE);
+
+		/* Trap if the task tries to use SVE again: */
+		change_cpacr(cpacr, sve_cpacr_trap_on(cpacr));
+	}
+
+	if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) {
+		if (system_supports_sve() && test_thread_flag(TIF_SVE))
+			sve_save_state(sve_pffr(current),
+				       &current->thread.fpsimd_state.fpsr);
+		else
+			fpsimd_save_state(&current->thread.fpsimd_state);
+	}
+}
+
 void fpsimd_thread_switch(struct task_struct *next)
 {
 	if (!system_supports_fpsimd())
@@ -144,8 +356,8 @@ void fpsimd_thread_switch(struct task_struct *next)
 	 * the registers is in fact the most recent userland FPSIMD state of
 	 * 'current'.
 	 */
-	if (current->mm && !test_thread_flag(TIF_FOREIGN_FPSTATE))
-		fpsimd_save_state(&current->thread.fpsimd_state);
+	if (current->mm)
+		task_fpsimd_save();
 
 	if (next->mm) {
 		/*
@@ -172,8 +384,25 @@ void fpsimd_flush_thread(void)
 
 	local_bh_disable();
 
-	memset(&current->thread.fpsimd_state, 0, sizeof(struct fpsimd_state));
 	fpsimd_flush_task_state(current);
+
+	memset(&current->thread.fpsimd_state, 0, sizeof(struct fpsimd_state));
+
+	if (system_supports_sve()) {
+		clear_thread_flag(TIF_SVE);
+		sve_free(current);
+
+		/*
+		 * User tasks must have a valid vector length set, but tasks
+		 * forked early (e.g., init) may not initially have one.
+		 * By now, we will know what the hardware supports, so
+		 * sve_default_vl should be valid, and thus the above
+		 * assignment should ensure a valid VL for the task.
+		 * If not, something went badly wrong.
+		 */
+		BUG_ON(!sve_vl_valid(current->thread.sve_vl));
+	}
+
 	set_thread_flag(TIF_FOREIGN_FPSTATE);
 
 	local_bh_enable();
@@ -211,7 +440,7 @@ void fpsimd_restore_current_state(void)
 	if (test_and_clear_thread_flag(TIF_FOREIGN_FPSTATE)) {
 		struct fpsimd_state *st = &current->thread.fpsimd_state;
 
-		fpsimd_load_state(st);
+		task_fpsimd_load();
 		__this_cpu_write(fpsimd_last_state, st);
 		st->cpu = smp_processor_id();
 	}
@@ -375,8 +604,8 @@ static int fpsimd_cpu_pm_notifier(struct notifier_block *self,
 {
 	switch (cmd) {
 	case CPU_PM_ENTER:
-		if (current->mm && !test_thread_flag(TIF_FOREIGN_FPSTATE))
-			fpsimd_save_state(&current->thread.fpsimd_state);
+		if (current->mm)
+			task_fpsimd_save();
 		this_cpu_write(fpsimd_last_state, NULL);
 		break;
 	case CPU_PM_EXIT:
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 659ae80..e51cb1f 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -239,13 +239,17 @@ void flush_thread(void)
 
 void release_thread(struct task_struct *dead_task)
 {
+	fpsimd_release_thread(dead_task);
 }
 
 int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
 {
 	if (current->mm)
 		fpsimd_preserve_current_state();
-	*dst = *src;
+	memcpy(dst, src, arch_task_struct_size);
+
+	fpsimd_dup_sve(dst, src);
+
 	return 0;
 }
 
diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
index 8964795..286064e 100644
--- a/arch/arm64/kernel/traps.c
+++ b/arch/arm64/kernel/traps.c
@@ -379,8 +379,8 @@ static int call_undef_hook(struct pt_regs *regs)
 	return fn ? fn(regs, instr) : 1;
 }
 
-static void force_signal_inject(int signal, int code, struct pt_regs *regs,
-				unsigned long address)
+void force_signal_inject(int signal, int code, struct pt_regs *regs,
+			 unsigned long address)
 {
 	siginfo_t info;
 	void __user *pc = (void __user *)instruction_pointer(regs);
-- 
2.1.4

[PATCH 12/27] arm64/sve: Support vector length resetting for new processes

[Dave Martin]

It's desirable to be able to reset the vector length to some sane
default for new processes, since the new binary and its libraries
processes may or may not be SVE-aware.

This patch tracks the desired post-exec vector length (if any) in a
new thread member sve_vl_onexec, and adds a new thread flag
TIF_SVE_VL_INHERIT to control whether to inherit or reset the
vector length.  Currently these are inactive.  Subsequent patches
will provide the capability to configure them.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/include/asm/processor.h   |  1 +
 arch/arm64/include/asm/thread_info.h |  1 +
 arch/arm64/kernel/fpsimd.c           | 13 +++++++++++++
 3 files changed, 15 insertions(+)

diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
index 969feed..da8802a 100644
--- a/arch/arm64/include/asm/processor.h
+++ b/arch/arm64/include/asm/processor.h
@@ -87,6 +87,7 @@ struct thread_struct {
 	struct fpsimd_state	fpsimd_state;
 	void			*sve_state;	/* SVE registers, if any */
 	u16			sve_vl;		/* SVE vector length */
+	u16			sve_vl_onexec;	/* SVE vl after next exec */
 	unsigned long		fault_address;	/* fault info */
 	unsigned long		fault_code;	/* ESR_EL1 value */
 	struct debug_info	debug;		/* debugging */
diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h
index 1a4b30b..bf9c552 100644
--- a/arch/arm64/include/asm/thread_info.h
+++ b/arch/arm64/include/asm/thread_info.h
@@ -97,6 +97,7 @@ struct thread_info {
 #define TIF_SINGLESTEP		21
 #define TIF_32BIT		22	/* 32bit process */
 #define TIF_SVE			23	/* Scalable Vector Extension in use */
+#define TIF_SVE_VL_INHERIT	24	/* Inherit sve_vl_onexec across exec */
 
 #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
 #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
index 37dd1b2..80ecb2d 100644
--- a/arch/arm64/kernel/fpsimd.c
+++ b/arch/arm64/kernel/fpsimd.c
@@ -108,6 +108,9 @@ static void task_fpsimd_save(void);
  */
 static DEFINE_PER_CPU(struct fpsimd_state *, fpsimd_last_state);
 
+/* Default VL for tasks that don't set it explicitly: */
+static int sve_default_vl = -1;
+
 static void sve_free(struct task_struct *task)
 {
 	kfree(task->thread.sve_state);
@@ -392,6 +395,9 @@ void fpsimd_flush_thread(void)
 		clear_thread_flag(TIF_SVE);
 		sve_free(current);
 
+		current->thread.sve_vl = current->thread.sve_vl_onexec ?
+			current->thread.sve_vl_onexec : sve_default_vl;
+
 		/*
 		 * User tasks must have a valid vector length set, but tasks
 		 * forked early (e.g., init) may not initially have one.
@@ -401,6 +407,13 @@ void fpsimd_flush_thread(void)
 		 * If not, something went badly wrong.
 		 */
 		BUG_ON(!sve_vl_valid(current->thread.sve_vl));
+
+		/*
+		 * If the task is not set to inherit, ensure that the vector
+		 * length will be reset by a subsequent exec:
+		 */
+		if (!test_thread_flag(TIF_SVE_VL_INHERIT))
+			current->thread.sve_vl_onexec = 0;
 	}
 
 	set_thread_flag(TIF_FOREIGN_FPSTATE);
-- 
2.1.4

[PATCH 12/27] arm64/sve: Support vector length resetting for new processes

[Dave Martin]

It's desirable to be able to reset the vector length to some sane
default for new processes, since the new binary and its libraries
processes may or may not be SVE-aware.

This patch tracks the desired post-exec vector length (if any) in a
new thread member sve_vl_onexec, and adds a new thread flag
TIF_SVE_VL_INHERIT to control whether to inherit or reset the
vector length.  Currently these are inactive.  Subsequent patches
will provide the capability to configure them.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/include/asm/processor.h   |  1 +
 arch/arm64/include/asm/thread_info.h |  1 +
 arch/arm64/kernel/fpsimd.c           | 13 +++++++++++++
 3 files changed, 15 insertions(+)

diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
index 969feed..da8802a 100644
--- a/arch/arm64/include/asm/processor.h
+++ b/arch/arm64/include/asm/processor.h
@@ -87,6 +87,7 @@ struct thread_struct {
 	struct fpsimd_state	fpsimd_state;
 	void			*sve_state;	/* SVE registers, if any */
 	u16			sve_vl;		/* SVE vector length */
+	u16			sve_vl_onexec;	/* SVE vl after next exec */
 	unsigned long		fault_address;	/* fault info */
 	unsigned long		fault_code;	/* ESR_EL1 value */
 	struct debug_info	debug;		/* debugging */
diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h
index 1a4b30b..bf9c552 100644
--- a/arch/arm64/include/asm/thread_info.h
+++ b/arch/arm64/include/asm/thread_info.h
@@ -97,6 +97,7 @@ struct thread_info {
 #define TIF_SINGLESTEP		21
 #define TIF_32BIT		22	/* 32bit process */
 #define TIF_SVE			23	/* Scalable Vector Extension in use */
+#define TIF_SVE_VL_INHERIT	24	/* Inherit sve_vl_onexec across exec */
 
 #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
 #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
index 37dd1b2..80ecb2d 100644
--- a/arch/arm64/kernel/fpsimd.c
+++ b/arch/arm64/kernel/fpsimd.c
@@ -108,6 +108,9 @@ static void task_fpsimd_save(void);
  */
 static DEFINE_PER_CPU(struct fpsimd_state *, fpsimd_last_state);
 
+/* Default VL for tasks that don't set it explicitly: */
+static int sve_default_vl = -1;
+
 static void sve_free(struct task_struct *task)
 {
 	kfree(task->thread.sve_state);
@@ -392,6 +395,9 @@ void fpsimd_flush_thread(void)
 		clear_thread_flag(TIF_SVE);
 		sve_free(current);
 
+		current->thread.sve_vl = current->thread.sve_vl_onexec ?
+			current->thread.sve_vl_onexec : sve_default_vl;
+
 		/*
 		 * User tasks must have a valid vector length set, but tasks
 		 * forked early (e.g., init) may not initially have one.
@@ -401,6 +407,13 @@ void fpsimd_flush_thread(void)
 		 * If not, something went badly wrong.
 		 */
 		BUG_ON(!sve_vl_valid(current->thread.sve_vl));
+
+		/*
+		 * If the task is not set to inherit, ensure that the vector
+		 * length will be reset by a subsequent exec:
+		 */
+		if (!test_thread_flag(TIF_SVE_VL_INHERIT))
+			current->thread.sve_vl_onexec = 0;
 	}
 
 	set_thread_flag(TIF_FOREIGN_FPSTATE);
-- 
2.1.4

[PATCH 13/27] arm64/sve: Signal handling support

[Dave Martin]

This patch implements support for saving and restoring the SVE
registers around signals.

A fixed-size header struct sve_context is always included in the
signal frame encoding the thread's vector length at the time of
signal delivery, optionally followed by a variable-layout structure
encoding the SVE registers.

Because of the need to preserve backwards compatibility, the FPSIMD
view of the SVE registers is always dumped as a struct
fpsimd_context in the usual way, in addition to any sve_context.

The SVE vector registers are dumped in full, including bits 127:0
of each register which alias the corresponding FPSIMD vector
registers in the hardware.  To avoid any ambiguity about which
alias to restore during sigreturn, the kernel always restores bits
127:0 of each SVE vector register from the fpsimd_context in the
signal frame (which must be present): userspace needs to take this
into account if it wants to modify the SVE vector register contents
on return from a signal.

FPSR and FPCR, which are used by both FPSIMD and SVE, are not
included in sve_context because they are always present in
fpsimd_context anyway.

For signal delivery, a new helper
fpsimd_signal_preserve_current_state() is added to update _both_
the FPSIMD and SVE views in the task struct, to make it easier to
populate this information into the signal frame.  Because of the
redundancy between the two views of the state, only one is updated
otherwise.  In order to avoid racing with a pending discard of the
SVE state, this flush is hoisted before the sigframe layout phase,
so that the layout and population phases see a consistent view of
the thread.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/include/asm/fpsimd.h |   1 +
 arch/arm64/kernel/fpsimd.c      |  23 ++++--
 arch/arm64/kernel/signal.c      | 169 ++++++++++++++++++++++++++++++++++++++--
 arch/arm64/kernel/signal32.c    |   2 +-
 4 files changed, 179 insertions(+), 16 deletions(-)

diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
index 72090a1..7efd04e 100644
--- a/arch/arm64/include/asm/fpsimd.h
+++ b/arch/arm64/include/asm/fpsimd.h
@@ -63,6 +63,7 @@ extern void fpsimd_load_state(struct fpsimd_state *state);
 extern void fpsimd_thread_switch(struct task_struct *next);
 extern void fpsimd_flush_thread(void);
 
+extern void fpsimd_signal_preserve_current_state(void);
 extern void fpsimd_preserve_current_state(void);
 extern void fpsimd_restore_current_state(void);
 extern void fpsimd_update_current_state(struct fpsimd_state *state);
diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
index 80ecb2d..e8674f6 100644
--- a/arch/arm64/kernel/fpsimd.c
+++ b/arch/arm64/kernel/fpsimd.c
@@ -148,8 +148,6 @@ static void change_cpacr(u64 old, u64 new)
 		write_sysreg(new, CPACR_EL1);
 }
 
-#ifdef CONFIG_ARM64_SVE
-
 #define ZREG(sve_state, vq, n) ((char *)(sve_state) +		\
 	(SVE_SIG_ZREG_OFFSET(vq, n) - SVE_SIG_REGS_OFFSET))
 
@@ -191,6 +189,8 @@ static void fpsimd_to_sve(struct task_struct *task)
 		       sizeof(fst->vregs[i]));
 }
 
+#ifdef CONFIG_ARM64_SVE
+
 size_t sve_state_size(struct task_struct const *task)
 {
 	unsigned int vl = task->thread.sve_vl;
@@ -431,13 +431,17 @@ void fpsimd_preserve_current_state(void)
 		return;
 
 	local_bh_disable();
-
-	if (!test_thread_flag(TIF_FOREIGN_FPSTATE))
-		fpsimd_save_state(&current->thread.fpsimd_state);
-
+	task_fpsimd_save();
 	local_bh_enable();
 }
 
+void fpsimd_signal_preserve_current_state(void)
+{
+	fpsimd_preserve_current_state();
+	if (system_supports_sve() && test_thread_flag(TIF_SVE))
+		sve_to_fpsimd(current);
+}
+
 /*
  * Load the userland FPSIMD state of 'current' from memory, but only if the
  * FPSIMD state already held in the registers is /not/ the most recent FPSIMD
@@ -473,7 +477,12 @@ void fpsimd_update_current_state(struct fpsimd_state *state)
 
 	local_bh_disable();
 
-	fpsimd_load_state(state);
+	if (system_supports_sve() && test_thread_flag(TIF_SVE)) {
+		current->thread.fpsimd_state = *state;
+		fpsimd_to_sve(current);
+	}
+	task_fpsimd_load();
+
 	if (test_and_clear_thread_flag(TIF_FOREIGN_FPSTATE)) {
 		struct fpsimd_state *st = &current->thread.fpsimd_state;
 
diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c
index 4991e87..2694143 100644
--- a/arch/arm64/kernel/signal.c
+++ b/arch/arm64/kernel/signal.c
@@ -62,6 +62,7 @@ struct rt_sigframe_user_layout {
 
 	unsigned long fpsimd_offset;
 	unsigned long esr_offset;
+	unsigned long sve_offset;
 	unsigned long extra_offset;
 	unsigned long end_offset;
 };
@@ -178,9 +179,6 @@ static int preserve_fpsimd_context(struct fpsimd_context __user *ctx)
 	struct fpsimd_state *fpsimd = &current->thread.fpsimd_state;
 	int err;
 
-	/* dump the hardware registers to the fpsimd_state structure */
-	fpsimd_preserve_current_state();
-
 	/* copy the FP and status/control registers */
 	err = __copy_to_user(ctx->vregs, fpsimd->vregs, sizeof(fpsimd->vregs));
 	__put_user_error(fpsimd->fpsr, &ctx->fpsr, err);
@@ -213,6 +211,8 @@ static int restore_fpsimd_context(struct fpsimd_context __user *ctx)
 	__get_user_error(fpsimd.fpsr, &ctx->fpsr, err);
 	__get_user_error(fpsimd.fpcr, &ctx->fpcr, err);
 
+	clear_thread_flag(TIF_SVE);
+
 	/* load the hardware registers from the fpsimd_state structure */
 	if (!err)
 		fpsimd_update_current_state(&fpsimd);
@@ -220,10 +220,113 @@ static int restore_fpsimd_context(struct fpsimd_context __user *ctx)
 	return err ? -EFAULT : 0;
 }
 
+
 struct user_ctxs {
 	struct fpsimd_context __user *fpsimd;
+	struct sve_context __user *sve;
 };
 
+#ifdef CONFIG_ARM64_SVE
+
+static int preserve_sve_context(struct sve_context __user *ctx)
+{
+	int err = 0;
+	u16 reserved[ARRAY_SIZE(ctx->__reserved)];
+	unsigned int vl = current->thread.sve_vl;
+	unsigned int vq = 0;
+
+	BUG_ON(!sve_vl_valid(vl));
+	if (test_thread_flag(TIF_SVE))
+		vq = sve_vq_from_vl(vl);
+
+	memset(reserved, 0, sizeof(reserved));
+
+	__put_user_error(SVE_MAGIC, &ctx->head.magic, err);
+	__put_user_error(round_up(SVE_SIG_CONTEXT_SIZE(vq), 16),
+			 &ctx->head.size, err);
+	__put_user_error(vl, &ctx->vl, err);
+	BUILD_BUG_ON(sizeof(ctx->__reserved) != sizeof(reserved));
+	err |= copy_to_user(&ctx->__reserved, reserved, sizeof(reserved));
+
+	if (vq) {
+		/*
+		 * This assumes that the SVE state has already been saved to
+		 * the task struct by calling preserve_fpsimd_context().
+		 */
+		BUG_ON(SVE_SIG_REGS_SIZE(vq) != sve_state_size(current));
+		err |= copy_to_user((char __user *)ctx + SVE_SIG_REGS_OFFSET,
+				    current->thread.sve_state,
+				    SVE_SIG_REGS_SIZE(vq));
+	}
+
+	return err ? -EFAULT : 0;
+}
+
+static int restore_sve_fpsimd_context(struct user_ctxs *user)
+{
+	int err;
+	unsigned int vq;
+	struct fpsimd_state fpsimd;
+	struct sve_context sve;
+
+	if (__copy_from_user(&sve, user->sve, sizeof(sve)))
+		return -EFAULT;
+
+	if (sve.vl != current->thread.sve_vl)
+		return -EINVAL;
+
+	if (sve.head.size <= sizeof(*user->sve)) {
+		clear_thread_flag(TIF_SVE);
+		goto fpsimd_only;
+	}
+
+	BUG_ON(!sve_vl_valid(sve.vl));
+	vq = sve_vq_from_vl(sve.vl);
+
+	if (sve.head.size < SVE_SIG_CONTEXT_SIZE(vq))
+		return -EINVAL;
+
+	fpsimd_flush_task_state(current);
+	barrier();
+	set_thread_flag(TIF_FOREIGN_FPSTATE);
+	barrier();
+
+	sve_alloc(current);
+	BUG_ON(SVE_SIG_REGS_SIZE(vq) != sve_state_size(current));
+	err = __copy_from_user(current->thread.sve_state,
+			       (char __user const *)user->sve +
+					SVE_SIG_REGS_OFFSET,
+			       SVE_SIG_REGS_SIZE(vq));
+	if (err)
+		return err;
+
+	barrier();
+	set_thread_flag(TIF_SVE);
+
+fpsimd_only:
+	/* copy the FP and status/control registers */
+	/* restore_sigframe() already checked that user->fpsimd != NULL. */
+	err = __copy_from_user(fpsimd.vregs, user->fpsimd->vregs,
+			       sizeof(fpsimd.vregs));
+	__get_user_error(fpsimd.fpsr, &user->fpsimd->fpsr, err);
+	__get_user_error(fpsimd.fpcr, &user->fpsimd->fpcr, err);
+
+	/* load the hardware registers from the fpsimd_state structure */
+	if (!err)
+		fpsimd_update_current_state(&fpsimd);
+
+	return err;
+}
+
+#else /* ! CONFIG_ARM64_SVE */
+
+/* Turn any non-optimised out attempts to use these into a link error: */
+extern int preserve_sve_context(void __user *ctx);
+extern int restore_sve_fpsimd_context(struct user_ctxs *user);
+
+#endif /* ! CONFIG_ARM64_SVE */
+
+
 static int parse_user_sigframe(struct user_ctxs *user,
 			       struct rt_sigframe __user *sf)
 {
@@ -236,6 +339,7 @@ static int parse_user_sigframe(struct user_ctxs *user,
 	char const __user *const sfp = (char const __user *)sf;
 
 	user->fpsimd = NULL;
+	user->sve = NULL;
 
 	if (!IS_ALIGNED((unsigned long)base, 16))
 		goto invalid;
@@ -286,6 +390,19 @@ static int parse_user_sigframe(struct user_ctxs *user,
 			/* ignore */
 			break;
 
+		case SVE_MAGIC:
+			if (!system_supports_sve())
+				goto invalid;
+
+			if (user->sve)
+				goto invalid;
+
+			if (size < sizeof(*user->sve))
+				goto invalid;
+
+			user->sve = (struct sve_context __user *)head;
+			break;
+
 		case EXTRA_MAGIC:
 			if (have_extra_context)
 				goto invalid;
@@ -358,9 +475,6 @@ static int parse_user_sigframe(struct user_ctxs *user,
 	}
 
 done:
-	if (!user->fpsimd)
-		goto invalid;
-
 	return 0;
 
 invalid:
@@ -394,8 +508,18 @@ static int restore_sigframe(struct pt_regs *regs,
 	if (err == 0)
 		err = parse_user_sigframe(&user, sf);
 
-	if (err == 0)
-		err = restore_fpsimd_context(user.fpsimd);
+	if (err == 0) {
+		if (!user.fpsimd)
+			return -EINVAL;
+
+		if (user.sve) {
+			if (!system_supports_sve())
+				return -EINVAL;
+
+			err = restore_sve_fpsimd_context(&user);
+		} else
+			err = restore_fpsimd_context(user.fpsimd);
+	}
 
 	return err;
 }
@@ -454,6 +578,20 @@ static int setup_sigframe_layout(struct rt_sigframe_user_layout *user)
 			return err;
 	}
 
+	if (system_supports_sve()) {
+		unsigned int vq = 0;
+
+		if (test_thread_flag(TIF_SVE)) {
+			BUG_ON(!sve_vl_valid(current->thread.sve_vl));
+			vq = sve_vq_from_vl(current->thread.sve_vl);
+		}
+
+		err = sigframe_alloc(user, &user->sve_offset,
+				     SVE_SIG_CONTEXT_SIZE(vq));
+		if (err)
+			return err;
+	}
+
 	return sigframe_alloc_end(user);
 }
 
@@ -495,6 +633,13 @@ static int setup_sigframe(struct rt_sigframe_user_layout *user,
 		__put_user_error(current->thread.fault_code, &esr_ctx->esr, err);
 	}
 
+	/* Scalable Vector Extension state, if present */
+	if (system_supports_sve() && err == 0 && user->sve_offset) {
+		struct sve_context __user *sve_ctx =
+			apply_user_offset(user, user->sve_offset);
+		err |= preserve_sve_context(sve_ctx);
+	}
+
 	if (err == 0 && user->extra_offset) {
 		char __user *sfp = (char __user *)user->sigframe;
 		char __user *userp =
@@ -594,6 +739,14 @@ static int setup_rt_frame(int usig, struct ksignal *ksig, sigset_t *set,
 	struct rt_sigframe __user *frame;
 	int err = 0;
 
+	/*
+	 * Ensure FPSIMD/SVE state in task_struct is up-to-date.
+	 * This is needed here in order to complete any pending SVE discard:
+	 * otherwise, discard may occur between deciding on the sigframe
+	 * layout and dumping the register data.
+	 */
+	fpsimd_signal_preserve_current_state();
+
 	if (get_sigframe(&user, ksig, regs))
 		return 1;
 
diff --git a/arch/arm64/kernel/signal32.c b/arch/arm64/kernel/signal32.c
index 4e5a664..202337d 100644
--- a/arch/arm64/kernel/signal32.c
+++ b/arch/arm64/kernel/signal32.c
@@ -244,7 +244,7 @@ static int compat_preserve_vfp_context(struct compat_vfp_sigframe __user *frame)
 	 * Note that this also saves V16-31, which aren't visible
 	 * in AArch32.
 	 */
-	fpsimd_preserve_current_state();
+	fpsimd_signal_preserve_current_state();
 
 	/* Place structure header on the stack */
 	__put_user_error(magic, &frame->magic, err);
-- 
2.1.4

[PATCH 14/27] arm64/sve: Backend logic for setting the vector length

[Dave Martin]

This patch implements the core logic for changing a task's vector
length on request from userspace.  This will be used by the ptrace
and prctl frontends that are implemented in later patches.

The SVE architecture permits, but does not require, implementations
to support vector lengths that are not a power of two.  To handle
this, logic is added to check a requested vector length against a
possibly sparse bitmap of available vector lengths at runtime, so
that the best supported value can be chosen.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/include/asm/fpsimd.h |   6 +++
 arch/arm64/kernel/fpsimd.c      | 116 ++++++++++++++++++++++++++++++++++++++++
 include/uapi/linux/prctl.h      |   5 ++
 3 files changed, 127 insertions(+)

diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
index 7efd04e..39b26d2 100644
--- a/arch/arm64/include/asm/fpsimd.h
+++ b/arch/arm64/include/asm/fpsimd.h
@@ -70,11 +70,15 @@ extern void fpsimd_update_current_state(struct fpsimd_state *state);
 
 extern void fpsimd_flush_task_state(struct task_struct *target);
 
+#define SVE_VL_ARCH_MAX 0x100
+
 extern void sve_save_state(void *state, u32 *pfpsr);
 extern void sve_load_state(void const *state, u32 const *pfpsr,
 			   unsigned long vq_minus_1);
 extern unsigned int sve_get_vl(void);
 
+extern int sve_max_vl;
+
 #ifdef CONFIG_ARM64_SVE
 
 extern size_t sve_state_size(struct task_struct const *task);
@@ -83,6 +87,8 @@ extern void sve_alloc(struct task_struct *task);
 extern void fpsimd_release_thread(struct task_struct *task);
 extern void fpsimd_dup_sve(struct task_struct *dst,
 			   struct task_struct const *src);
+extern int sve_set_vector_length(struct task_struct *task,
+				 unsigned long vl, unsigned long flags);
 
 #else /* ! CONFIG_ARM64_SVE */
 
diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
index e8674f6..bce95de 100644
--- a/arch/arm64/kernel/fpsimd.c
+++ b/arch/arm64/kernel/fpsimd.c
@@ -18,12 +18,14 @@
  */
 
 #include <linux/bottom_half.h>
+#include <linux/bitmap.h>
 #include <linux/cpu.h>
 #include <linux/cpu_pm.h>
 #include <linux/kernel.h>
 #include <linux/init.h>
 #include <linux/percpu.h>
 #include <linux/preempt.h>
+#include <linux/prctl.h>
 #include <linux/ptrace.h>
 #include <linux/sched/signal.h>
 #include <linux/signal.h>
@@ -111,6 +113,20 @@ static DEFINE_PER_CPU(struct fpsimd_state *, fpsimd_last_state);
 /* Default VL for tasks that don't set it explicitly: */
 static int sve_default_vl = -1;
 
+#ifdef CONFIG_ARM64_SVE
+
+/* Maximum supported vector length across all CPUs (initially poisoned) */
+int sve_max_vl = -1;
+/* Set of available vector lengths, as vq_to_bit(vq): */
+static DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX);
+
+#else /* ! CONFIG_ARM64_SVE */
+
+/* Dummy declaration for code that will be optimised out: */
+extern DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX);
+
+#endif /* ! CONFIG_ARM64_SVE */
+
 static void sve_free(struct task_struct *task)
 {
 	kfree(task->thread.sve_state);
@@ -148,6 +164,37 @@ static void change_cpacr(u64 old, u64 new)
 		write_sysreg(new, CPACR_EL1);
 }
 
+static unsigned int vq_to_bit(unsigned int vq)
+{
+	BUILD_BUG_ON(vq < 1 || vq > SVE_VQ_MAX);
+
+	return SVE_VQ_MAX - vq;
+}
+
+static unsigned int bit_to_vq(unsigned int bit)
+{
+	BUILD_BUG_ON(bit >= SVE_VQ_MAX);
+
+	return SVE_VQ_MAX - bit;
+}
+
+static unsigned int find_supported_vector_length(unsigned int vl)
+{
+	int bit;
+
+	BUG_ON(!sve_vl_valid(vl));
+
+	BUG_ON(!sve_vl_valid(sve_max_vl));
+	if (vl > sve_max_vl)
+		vl = sve_max_vl;
+
+	bit = find_next_bit(sve_vq_map, SVE_VQ_MAX,
+			    vq_to_bit(sve_vq_from_vl(vl)));
+	BUG_ON(bit < 0 || bit >= SVE_VQ_MAX);
+
+	return 16 * bit_to_vq(bit);
+}
+
 #define ZREG(sve_state, vq, n) ((char *)(sve_state) +		\
 	(SVE_SIG_ZREG_OFFSET(vq, n) - SVE_SIG_REGS_OFFSET))
 
@@ -235,6 +282,73 @@ void fpsimd_dup_sve(struct task_struct *dst, struct task_struct const *src)
 	}
 }
 
+int sve_set_vector_length(struct task_struct *task,
+			  unsigned long vl, unsigned long flags)
+{
+	BUG_ON(task == current && preemptible());
+
+	if (flags & ~(unsigned long)(PR_SVE_VL_INHERIT |
+				     PR_SVE_SET_VL_ONEXEC))
+		return -EINVAL;
+
+	if (!sve_vl_valid(vl))
+		return -EINVAL;
+
+	/*
+	 * Clamp to the maximum vector length that VL-agnostic SVE code can
+	 * work with.  A flag may be assigned in the future to allow setting
+	 * of larger vector lengths without confusing older software.
+	 */
+	if (vl > SVE_VL_ARCH_MAX)
+		vl = SVE_VL_ARCH_MAX;
+
+	vl = find_supported_vector_length(vl);
+
+	if (flags & (PR_SVE_VL_INHERIT |
+		     PR_SVE_SET_VL_ONEXEC))
+		task->thread.sve_vl_onexec = vl;
+	else
+		/* Reset VL to system default on next exec: */
+		task->thread.sve_vl_onexec = 0;
+
+	/* Only actually set the VL if not deferred: */
+	if (flags & PR_SVE_SET_VL_ONEXEC)
+		goto out;
+
+	/*
+	 * To ensure the FPSIMD bits of the SVE vector registers are preserved,
+	 * write any live register state back to task_struct, and convert to a
+	 * non-SVE thread.
+	 */
+	if (vl != task->thread.sve_vl) {
+		if (task == current) {
+			task_fpsimd_save();
+			set_thread_flag(TIF_FOREIGN_FPSTATE);
+		}
+
+		if (test_and_clear_tsk_thread_flag(task, TIF_SVE))
+			sve_to_fpsimd(task);
+
+		/*
+		 * Force reallocation of task SVE state to the correct size
+		 * on next use:
+		 */
+		sve_free(task);
+	}
+
+	task->thread.sve_vl = vl;
+
+	fpsimd_flush_task_state(task);
+
+out:
+	if (flags & PR_SVE_VL_INHERIT)
+		set_thread_flag(TIF_SVE_VL_INHERIT);
+	else
+		clear_thread_flag(TIF_SVE_VL_INHERIT);
+
+	return 0;
+}
+
 void fpsimd_release_thread(struct task_struct *dead_task)
 {
 	sve_free(dead_task);
@@ -407,6 +521,8 @@ void fpsimd_flush_thread(void)
 		 * If not, something went badly wrong.
 		 */
 		BUG_ON(!sve_vl_valid(current->thread.sve_vl));
+		BUG_ON(find_supported_vector_length(current->thread.sve_vl) !=
+		       current->thread.sve_vl);
 
 		/*
 		 * If the task is not set to inherit, ensure that the vector
diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h
index a8d0759..1b64901 100644
--- a/include/uapi/linux/prctl.h
+++ b/include/uapi/linux/prctl.h
@@ -197,4 +197,9 @@ struct prctl_mm_map {
 # define PR_CAP_AMBIENT_LOWER		3
 # define PR_CAP_AMBIENT_CLEAR_ALL	4
 
+/* arm64 Scalable Vector Extension controls */
+# define PR_SVE_SET_VL_ONEXEC		(1 << 18) /* defer effect until exec */
+# define PR_SVE_VL_LEN_MASK		0xffff
+# define PR_SVE_VL_INHERIT		(1 << 17) /* inherit across exec */
+
 #endif /* _LINUX_PRCTL_H */
-- 
2.1.4

[PATCH 15/27] arm64/sve: Probe SVE capabilities and usable vector lengths

[Dave Martin]

This patch uses the cpufeatures framework to determine common SVE
capabilities and vector lengths, and configures the runtime SVE
support code appropriately.

ZCR_ELx is not really a feature register, but it is convenient to
use it as a template for recording the maximum vector length
supported by a CPU, using the LEN field.  This field is similar to
a feature field in that it is a contiguous bitfield for which we
want to determine the minimum system-wide value.  This patch adds
ZCR as a pseudo-register in cpuinfo/cpufeatures, with appropriate
custom code to populate it.  Finding the minimum supported value of
the LEN field is left to the cpufeatures framework in the usual
way.

The meaning of ID_AA64ZFR0_EL1 is not architecturally defined yet,
so for now we just require it to be zero.

Note that much of this code is dormant and SVE still won't be used
yet, since system_supports_sve() remains hardwired to false.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/include/asm/cpu.h        |   4 ++
 arch/arm64/include/asm/cpufeature.h |  28 ++++++++++
 arch/arm64/include/asm/fpsimd.h     |  10 ++++
 arch/arm64/kernel/cpufeature.c      |  48 ++++++++++++++++
 arch/arm64/kernel/cpuinfo.c         |   6 ++
 arch/arm64/kernel/fpsimd.c          | 108 ++++++++++++++++++++++++++++++++++++
 6 files changed, 204 insertions(+)

diff --git a/arch/arm64/include/asm/cpu.h b/arch/arm64/include/asm/cpu.h
index 889226b..8839227 100644
--- a/arch/arm64/include/asm/cpu.h
+++ b/arch/arm64/include/asm/cpu.h
@@ -41,6 +41,7 @@ struct cpuinfo_arm64 {
 	u64		reg_id_aa64mmfr2;
 	u64		reg_id_aa64pfr0;
 	u64		reg_id_aa64pfr1;
+	u64		reg_id_aa64zfr0;
 
 	u32		reg_id_dfr0;
 	u32		reg_id_isar0;
@@ -59,6 +60,9 @@ struct cpuinfo_arm64 {
 	u32		reg_mvfr0;
 	u32		reg_mvfr1;
 	u32		reg_mvfr2;
+
+	/* pseudo-ZCR for recording maximum ZCR_EL1 LEN value: */
+	u64		reg_zcr;
 };
 
 DECLARE_PER_CPU(struct cpuinfo_arm64, cpu_data);
diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index 4ea3441..05eec27 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -10,6 +10,7 @@
 #define __ASM_CPUFEATURE_H
 
 #include <asm/cpucaps.h>
+#include <asm/fpsimd.h>
 #include <asm/hwcap.h>
 #include <asm/sysreg.h>
 
@@ -223,6 +224,13 @@ static inline bool id_aa64pfr0_32bit_el0(u64 pfr0)
 	return val == ID_AA64PFR0_EL0_32BIT_64BIT;
 }
 
+static inline bool id_aa64pfr0_sve(u64 pfr0)
+{
+	u32 val = cpuid_feature_extract_unsigned_field(pfr0, ID_AA64PFR0_SVE_SHIFT);
+
+	return val > 0;
+}
+
 void __init setup_cpu_features(void);
 
 void update_cpu_capabilities(const struct arm64_cpu_capabilities *caps,
@@ -267,6 +275,26 @@ static inline bool system_supports_sve(void)
 	return false;
 }
 
+/*
+ * Read the pseudo-ZCR used by cpufeatures to identify the supported SVE
+ * vector length.
+ * Use only if SVE is present.  This function clobbers the SVE vector length.
+ */
+static u64 __maybe_unused read_zcr_features(void)
+{
+	u64 zcr;
+	unsigned int vq_max;
+
+	write_sysreg_s(ZCR_ELx_LEN_MASK, SYS_ZCR_EL1);
+
+	zcr = read_sysreg_s(SYS_ZCR_EL1);
+	zcr &= ~(u64)ZCR_ELx_LEN_MASK;
+	vq_max = sve_get_vl() / 16;
+	zcr |= vq_max - 1;
+
+	return zcr;
+}
+
 #endif /* __ASSEMBLY__ */
 
 #endif
diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
index 39b26d2..f43f573 100644
--- a/arch/arm64/include/asm/fpsimd.h
+++ b/arch/arm64/include/asm/fpsimd.h
@@ -90,12 +90,22 @@ extern void fpsimd_dup_sve(struct task_struct *dst,
 extern int sve_set_vector_length(struct task_struct *task,
 				 unsigned long vl, unsigned long flags);
 
+extern void __init sve_init_vq_map(void);
+extern void sve_update_vq_map(void);
+extern int sve_verify_vq_map(void);
+extern void __init sve_setup(void);
+
 #else /* ! CONFIG_ARM64_SVE */
 
 static void __maybe_unused sve_alloc(struct task_struct *task) { }
 static void __maybe_unused fpsimd_release_thread(struct task_struct *task) { }
 static void __maybe_unused fpsimd_dup_sve(struct task_struct *dst,
 					  struct task_struct const *src) { }
+static void __maybe_unused sve_init_vq_map(void) { }
+static void __maybe_unused sve_update_vq_map(void) { }
+static int __maybe_unused sve_verify_vq_map(void) { return 0; }
+static void __maybe_unused sve_setup(void) { }
+
 #endif /* ! CONFIG_ARM64_SVE */
 
 /* For use by EFI runtime services calls only */
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 9f9e0064..f322c90 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -27,6 +27,7 @@
 #include <asm/cpu.h>
 #include <asm/cpufeature.h>
 #include <asm/cpu_ops.h>
+#include <asm/fpsimd.h>
 #include <asm/mmu_context.h>
 #include <asm/processor.h>
 #include <asm/sysreg.h>
@@ -267,6 +268,12 @@ static const struct arm64_ftr_bits ftr_id_dfr0[] = {
 	ARM64_FTR_END,
 };
 
+static const struct arm64_ftr_bits ftr_zcr[] = {
+	ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE,
+		ZCR_ELx_LEN_SHIFT, ZCR_ELx_LEN_SIZE, 0),	/* LEN */
+	ARM64_FTR_END,
+};
+
 /*
  * Common ftr bits for a 32bit register with all hidden, strict
  * attributes, with 4bit feature fields and a default safe value of
@@ -333,6 +340,7 @@ static const struct __ftr_reg_entry {
 	/* Op1 = 0, CRn = 0, CRm = 4 */
 	ARM64_FTR_REG(SYS_ID_AA64PFR0_EL1, ftr_id_aa64pfr0),
 	ARM64_FTR_REG(SYS_ID_AA64PFR1_EL1, ftr_raz),
+	ARM64_FTR_REG(SYS_ID_AA64ZFR0_EL1, ftr_raz),
 
 	/* Op1 = 0, CRn = 0, CRm = 5 */
 	ARM64_FTR_REG(SYS_ID_AA64DFR0_EL1, ftr_id_aa64dfr0),
@@ -347,6 +355,9 @@ static const struct __ftr_reg_entry {
 	ARM64_FTR_REG(SYS_ID_AA64MMFR1_EL1, ftr_id_aa64mmfr1),
 	ARM64_FTR_REG(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2),
 
+	/* Op1 = 0, CRn = 1, CRm = 2 */
+	ARM64_FTR_REG(SYS_ZCR_EL1, ftr_zcr),
+
 	/* Op1 = 3, CRn = 0, CRm = 0 */
 	{ SYS_CTR_EL0, &arm64_ftr_reg_ctrel0 },
 	ARM64_FTR_REG(SYS_DCZID_EL0, ftr_dczid),
@@ -484,6 +495,7 @@ void __init init_cpu_features(struct cpuinfo_arm64 *info)
 	init_cpu_ftr_reg(SYS_ID_AA64MMFR2_EL1, info->reg_id_aa64mmfr2);
 	init_cpu_ftr_reg(SYS_ID_AA64PFR0_EL1, info->reg_id_aa64pfr0);
 	init_cpu_ftr_reg(SYS_ID_AA64PFR1_EL1, info->reg_id_aa64pfr1);
+	init_cpu_ftr_reg(SYS_ID_AA64ZFR0_EL1, info->reg_id_aa64zfr0);
 
 	if (id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0)) {
 		init_cpu_ftr_reg(SYS_ID_DFR0_EL1, info->reg_id_dfr0);
@@ -504,6 +516,10 @@ void __init init_cpu_features(struct cpuinfo_arm64 *info)
 		init_cpu_ftr_reg(SYS_MVFR2_EL1, info->reg_mvfr2);
 	}
 
+	if (id_aa64pfr0_sve(info->reg_id_aa64pfr0)) {
+		init_cpu_ftr_reg(SYS_ZCR_EL1, info->reg_zcr);
+		sve_init_vq_map();
+	}
 }
 
 static void update_cpu_ftr_reg(struct arm64_ftr_reg *reg, u64 new)
@@ -607,6 +623,9 @@ void update_cpu_features(int cpu,
 	taint |= check_update_ftr_reg(SYS_ID_AA64PFR1_EL1, cpu,
 				      info->reg_id_aa64pfr1, boot->reg_id_aa64pfr1);
 
+	taint |= check_update_ftr_reg(SYS_ID_AA64ZFR0_EL1, cpu,
+				      info->reg_id_aa64zfr0, boot->reg_id_aa64zfr0);
+
 	/*
 	 * If we have AArch32, we care about 32-bit features for compat.
 	 * If the system doesn't support AArch32, don't update them.
@@ -654,6 +673,12 @@ void update_cpu_features(int cpu,
 					info->reg_mvfr2, boot->reg_mvfr2);
 	}
 
+	if (id_aa64pfr0_sve(info->reg_id_aa64pfr0)) {
+		taint |= check_update_ftr_reg(SYS_ZCR_EL1, cpu,
+					info->reg_zcr, boot->reg_zcr);
+		sve_update_vq_map();
+	}
+
 	/*
 	 * Mismatched CPU features are a recipe for disaster. Don't even
 	 * pretend to support them.
@@ -1084,6 +1109,23 @@ verify_local_cpu_features(const struct arm64_cpu_capabilities *caps)
 	}
 }
 
+static void verify_sve_features(void)
+{
+	u64 safe_zcr = read_sanitised_ftr_reg(SYS_ZCR_EL1);
+	u64 zcr = read_zcr_features();
+
+	unsigned int safe_len = safe_zcr & ZCR_ELx_LEN_MASK;
+	unsigned int len = zcr & ZCR_ELx_LEN_MASK;
+
+	if (len < safe_len || sve_verify_vq_map()) {
+		pr_crit("CPU%d: SVE: required vector length(s) missing\n",
+			smp_processor_id());
+		cpu_die_early();
+	}
+
+	/* Add checks on other ZCR bits here if necessary */
+}
+
 /*
  * Run through the enabled system capabilities and enable() it on this CPU.
  * The capabilities were decided based on the available CPUs at the boot time.
@@ -1097,8 +1139,12 @@ static void verify_local_cpu_capabilities(void)
 	verify_local_cpu_errata_workarounds();
 	verify_local_cpu_features(arm64_features);
 	verify_local_elf_hwcaps(arm64_elf_hwcaps);
+
 	if (system_supports_32bit_el0())
 		verify_local_elf_hwcaps(compat_elf_hwcaps);
+
+	if (system_supports_sve())
+		verify_sve_features();
 }
 
 void check_local_cpu_capabilities(void)
@@ -1176,6 +1222,8 @@ void __init setup_cpu_features(void)
 	if (system_supports_32bit_el0())
 		setup_elf_hwcaps(compat_elf_hwcaps);
 
+	sve_setup();
+
 	/* Advertise that we have computed the system capabilities */
 	set_sys_caps_initialised();
 
diff --git a/arch/arm64/kernel/cpuinfo.c b/arch/arm64/kernel/cpuinfo.c
index f495ee5..0e087d7 100644
--- a/arch/arm64/kernel/cpuinfo.c
+++ b/arch/arm64/kernel/cpuinfo.c
@@ -19,6 +19,7 @@
 #include <asm/cpu.h>
 #include <asm/cputype.h>
 #include <asm/cpufeature.h>
+#include <asm/fpsimd.h>
 
 #include <linux/bitops.h>
 #include <linux/bug.h>
@@ -325,6 +326,7 @@ static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info)
 	info->reg_id_aa64mmfr2 = read_cpuid(ID_AA64MMFR2_EL1);
 	info->reg_id_aa64pfr0 = read_cpuid(ID_AA64PFR0_EL1);
 	info->reg_id_aa64pfr1 = read_cpuid(ID_AA64PFR1_EL1);
+	info->reg_id_aa64zfr0 = read_cpuid(ID_AA64ZFR0_EL1);
 
 	/* Update the 32bit ID registers only if AArch32 is implemented */
 	if (id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0)) {
@@ -347,6 +349,10 @@ static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info)
 		info->reg_mvfr2 = read_cpuid(MVFR2_EL1);
 	}
 
+	if (IS_ENABLED(CONFIG_ARM64_SVE) &&
+	    id_aa64pfr0_sve(info->reg_id_aa64pfr0))
+		info->reg_zcr = read_zcr_features();
+
 	cpuinfo_detect_icache_policy(info);
 }
 
diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
index bce95de..955c873 100644
--- a/arch/arm64/kernel/fpsimd.c
+++ b/arch/arm64/kernel/fpsimd.c
@@ -119,11 +119,13 @@ static int sve_default_vl = -1;
 int sve_max_vl = -1;
 /* Set of available vector lengths, as vq_to_bit(vq): */
 static DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX);
+static bool sve_vq_map_finalised;
 
 #else /* ! CONFIG_ARM64_SVE */
 
 /* Dummy declaration for code that will be optimised out: */
 extern DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX);
+extern bool sve_vq_map_finalised;
 
 #endif /* ! CONFIG_ARM64_SVE */
 
@@ -349,6 +351,112 @@ int sve_set_vector_length(struct task_struct *task,
 	return 0;
 }
 
+static unsigned long *sve_alloc_vq_map(void)
+{
+	return kzalloc(BITS_TO_LONGS(SVE_VQ_MAX) * sizeof(unsigned long),
+		       GFP_KERNEL);
+}
+
+static void sve_probe_vqs(DECLARE_BITMAP(map, SVE_VQ_MAX))
+{
+	unsigned int vq, vl;
+	unsigned long zcr;
+
+	zcr = ZCR_ELx_LEN_MASK;
+	zcr = read_sysreg_s(SYS_ZCR_EL1) & ~zcr;
+
+	for (vq = SVE_VQ_MAX; vq >= 1; --vq) {
+		write_sysreg_s(zcr | (vq - 1), SYS_ZCR_EL1); /* self-syncing */
+		vl = sve_get_vl();
+
+		BUG_ON(!sve_vl_valid(vl));
+		vq = sve_vq_from_vl(vl);
+		set_bit(vq_to_bit(vq), map);
+	}
+}
+
+void __init sve_init_vq_map(void)
+{
+	sve_probe_vqs(sve_vq_map);
+}
+
+/*
+ * If we haven't committed to the set of supported VQs yet, filter out
+ * those not supported by the current CPU:
+ */
+void sve_update_vq_map(void)
+{
+	unsigned long *map;
+
+	if (sve_vq_map_finalised)
+		return;
+
+	map = sve_alloc_vq_map();
+	sve_probe_vqs(map);
+	bitmap_and(sve_vq_map, sve_vq_map, map, SVE_VQ_MAX);
+	kfree(map);
+}
+
+/* Check whether the current CPU supports all VQs in the committed set */
+int sve_verify_vq_map(void)
+{
+	int ret = 0;
+	unsigned long *map = sve_alloc_vq_map();
+
+	sve_probe_vqs(map);
+	bitmap_andnot(map, sve_vq_map, map, SVE_VQ_MAX);
+	if (!bitmap_empty(map, SVE_VQ_MAX)) {
+		pr_warn("SVE: cpu%d: Required vector length(s) missing\n",
+			smp_processor_id());
+		ret = -EINVAL;
+	}
+
+	kfree(map);
+
+	return ret;
+}
+
+void __init sve_setup(void)
+{
+	u64 zcr;
+	unsigned int max_vl;
+
+	if (!system_supports_sve())
+		return;
+
+	/*
+	 * The architecture mandates 128-bit vectors be supported, and
+	 * the code assumes elsewhere that sve_vq_map is non-empty:
+	 */
+	BUG_ON(!test_bit(vq_to_bit(1), sve_vq_map));
+
+	sve_vq_map_finalised = true;
+
+
+	zcr = read_sanitised_ftr_reg(SYS_ZCR_EL1);
+	max_vl = sve_vl_from_vq((zcr & ZCR_ELx_LEN_MASK) + 1);
+
+	/*
+	 * Sanity-check that the max VL we determined through CPU features
+	 * corresponds properly to sve_vq_map:
+	 */
+	sve_max_vl = SVE_VL_MAX;
+	BUG_ON(find_supported_vector_length(SVE_VL_MAX) != max_vl);
+	BUG_ON(find_supported_vector_length(max_vl) != max_vl);
+
+	sve_max_vl = max_vl;
+
+	/* For the default VL, pick the maximum supported value <= 64: */
+	sve_default_vl = find_supported_vector_length(64);
+	BUG_ON(!sve_vl_valid(sve_default_vl));
+	BUG_ON(sve_default_vl > 64);
+
+	pr_info("SVE: maximum available vector length %u bytes per vector\n",
+		sve_max_vl);
+	pr_info("SVE: default vector length %u bytes per vector\n",
+		sve_default_vl);
+}
+
 void fpsimd_release_thread(struct task_struct *dead_task)
 {
 	sve_free(dead_task);
-- 
2.1.4

[PATCH 16/27] arm64/sve: Preserve SVE registers around kernel-mode NEON use

[Dave Martin]

Kernel-mode NEON will corrupt the SVE vector registers, due to the
way they alias the FPSIMD vector registers in the hardware.

This patch ensures that any live SVE register content for the task
is saved by kernel_neon_begin().  The data will be restored in the
usual way on return to userspace.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/kernel/fpsimd.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
index 955c873..b7fb836 100644
--- a/arch/arm64/kernel/fpsimd.c
+++ b/arch/arm64/kernel/fpsimd.c
@@ -758,8 +758,10 @@ void kernel_neon_begin(void)
 	__this_cpu_write(kernel_neon_busy, true);
 
 	/* Save unsaved task fpsimd state, if any: */
-	if (current->mm && !test_and_set_thread_flag(TIF_FOREIGN_FPSTATE))
-		fpsimd_save_state(&current->thread.fpsimd_state);
+	if (current->mm) {
+		task_fpsimd_save();
+		set_thread_flag(TIF_FOREIGN_FPSTATE);
+	}
 
 	/* Invalidate any task state remaining in the fpsimd regs: */
 	__this_cpu_write(fpsimd_last_state, NULL);
-- 
2.1.4

[PATCH 16/27] arm64/sve: Preserve SVE registers around kernel-mode NEON use

[Dave Martin]

Kernel-mode NEON will corrupt the SVE vector registers, due to the
way they alias the FPSIMD vector registers in the hardware.

This patch ensures that any live SVE register content for the task
is saved by kernel_neon_begin().  The data will be restored in the
usual way on return to userspace.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/kernel/fpsimd.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
index 955c873..b7fb836 100644
--- a/arch/arm64/kernel/fpsimd.c
+++ b/arch/arm64/kernel/fpsimd.c
@@ -758,8 +758,10 @@ void kernel_neon_begin(void)
 	__this_cpu_write(kernel_neon_busy, true);
 
 	/* Save unsaved task fpsimd state, if any: */
-	if (current->mm && !test_and_set_thread_flag(TIF_FOREIGN_FPSTATE))
-		fpsimd_save_state(&current->thread.fpsimd_state);
+	if (current->mm) {
+		task_fpsimd_save();
+		set_thread_flag(TIF_FOREIGN_FPSTATE);
+	}
 
 	/* Invalidate any task state remaining in the fpsimd regs: */
 	__this_cpu_write(fpsimd_last_state, NULL);
-- 
2.1.4

[PATCH 17/27] arm64/sve: Preserve SVE registers around EFI runtime service calls

[Dave Martin]

The EFI runtime services ABI allows EFI to make free use of the
FPSIMD registers during EFI runtime service calls, subject to the
callee-save requirements of the AArch64 procedure call standard.

However, the SVE architecture allows upper bits of the SVE vector
registers to be zeroed as a side-effect of FPSIMD V-register
writes.  This means that the SVE vector registers must be saved in
their entirety in order to avoid data loss: non-SVE-aware EFI
implementations cannot restore them correctly.

The non-IRQ case is already handled gracefully by
kernel_neon_begin().  For the IRQ case, this patch allocates a
suitable per-CPU stash buffer for the full SVE register state and
uses it to preserve the affected registers around EFI calls.  It is
currently unclear how the EFI runtime services ABI will be
clarified with respect to SVE, so it safest to assume that the
predicate registers and FFR must be saved and restored too.

No attempt is made to restore the restore the vector length after
a call, for now.  It is deemed rather insane for EFI to change it,
and contemporary EFI implementations certainly won't.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/kernel/fpsimd.c | 53 ++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 49 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
index b7fb836..c727b47 100644
--- a/arch/arm64/kernel/fpsimd.c
+++ b/arch/arm64/kernel/fpsimd.c
@@ -120,12 +120,14 @@ int sve_max_vl = -1;
 /* Set of available vector lengths, as vq_to_bit(vq): */
 static DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX);
 static bool sve_vq_map_finalised;
+static void __percpu *efi_sve_state;
 
 #else /* ! CONFIG_ARM64_SVE */
 
 /* Dummy declaration for code that will be optimised out: */
 extern DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX);
 extern bool sve_vq_map_finalised;
+extern void __percpu *efi_sve_state;
 
 #endif /* ! CONFIG_ARM64_SVE */
 
@@ -416,6 +418,23 @@ int sve_verify_vq_map(void)
 	return ret;
 }
 
+static void __init sve_kernel_mode_neon_setup(void)
+{
+	if (!IS_ENABLED(CONFIG_KERNEL_MODE_NEON))
+		return;
+
+	/*
+	 * alloc_percpu() warns and prints a backtrace if this goes wrong.
+	 * This is evidence of a crippled system and we are returning void,
+	 * so no attempt is made to handle this situation here.
+	 */
+	BUG_ON(!sve_vl_valid(sve_max_vl));
+	efi_sve_state = __alloc_percpu(
+		SVE_SIG_REGS_SIZE(sve_vq_from_vl(sve_max_vl)), 16);
+	if (!efi_sve_state)
+		panic("Cannot allocate percpu memory for EFI SVE save/restore");
+}
+
 void __init sve_setup(void)
 {
 	u64 zcr;
@@ -455,6 +474,8 @@ void __init sve_setup(void)
 		sve_max_vl);
 	pr_info("SVE: default vector length %u bytes per vector\n",
 		sve_default_vl);
+
+	sve_kernel_mode_neon_setup();
 }
 
 void fpsimd_release_thread(struct task_struct *dead_task)
@@ -797,6 +818,7 @@ EXPORT_SYMBOL(kernel_neon_end);
 
 DEFINE_PER_CPU(struct fpsimd_state, efi_fpsimd_state);
 DEFINE_PER_CPU(bool, efi_fpsimd_state_used);
+DEFINE_PER_CPU(bool, efi_sve_state_used);
 
 /*
  * EFI runtime services support functions
@@ -825,7 +847,20 @@ void __efi_fpsimd_begin(void)
 	if (may_use_simd())
 		kernel_neon_begin();
 	else {
-		fpsimd_save_state(this_cpu_ptr(&efi_fpsimd_state));
+		/*
+		 * If !efi_sve_state, SVE can't be in use yet and doesn't need
+		 * preserving:
+		 */
+		if (system_supports_sve() && likely(efi_sve_state)) {
+			char *sve_state = this_cpu_ptr(efi_sve_state);
+
+			__this_cpu_write(efi_sve_state_used, true);
+
+			sve_save_state(sve_state + sve_ffr_offset(sve_max_vl),
+				       &this_cpu_ptr(&efi_fpsimd_state)->fpsr);
+		} else
+			fpsimd_save_state(this_cpu_ptr(&efi_fpsimd_state));
+
 		__this_cpu_write(efi_fpsimd_state_used, true);
 	}
 }
@@ -838,10 +873,20 @@ void __efi_fpsimd_end(void)
 	if (!system_supports_fpsimd())
 		return;
 
-	if (__this_cpu_xchg(efi_fpsimd_state_used, false))
-		fpsimd_load_state(this_cpu_ptr(&efi_fpsimd_state));
-	else
+	if (!__this_cpu_xchg(efi_fpsimd_state_used, false))
 		kernel_neon_end();
+	else
+		if (system_supports_sve() &&
+		    likely(__this_cpu_read(efi_sve_state_used))) {
+			char const *sve_state = this_cpu_ptr(efi_sve_state);
+
+			sve_load_state(sve_state + sve_ffr_offset(sve_max_vl),
+				       &this_cpu_ptr(&efi_fpsimd_state)->fpsr,
+				       sve_vq_from_vl(sve_get_vl()) - 1);
+
+			__this_cpu_write(efi_sve_state_used, false);
+		} else
+			fpsimd_load_state(this_cpu_ptr(&efi_fpsimd_state));
 }
 
 #endif /* CONFIG_KERNEL_MODE_NEON */
-- 
2.1.4

[PATCH 18/27] arm64/sve: ptrace and ELF coredump support

[Dave Martin]

This patch defines and implements a new regset NT_ARM_SVE, which
describes a thread's SVE register state.  This allows a debugger to
manipulate the SVE state, as well as being included in ELF
coredumps for post-mortem debugging.

Because the regset size and layout are dependent on the thread's
current vector length, it is not possible to define a C struct to
describe the regset contents as is done for existing regsets.
Instead, and for the same reasons, NT_ARM_SVE is based on the
freeform variable-layout approach used for the SVE signal frame.

Additionally, to reduce debug overhead when debugging threads that
might or might not have live SVE register state, NT_ARM_SVE may be
presented in one of two different formats: the old struct
user_fpsimd_state format is embedded for describing the state of a
thread with no live SVE state, whereas a new variable-layout
structure is embedded for describing live SVE state.  This avoids a
debugger needing to poll NT_PRFPREG in addition to NT_ARM_SVE, and
allows existing userspace code to handle the non-SVE case without
too much modification.

For this to work, NT_ARM_SVE is defined with a fixed-format header
of type struct user_sve_header, which the recipient can use to
figure out the content, size and layout of the reset of the regset.
Accessor macros are defined to allow the vector-length-dependent
parts of the regset to be manipulated.

Signed-off-by: Alan Hayward <alan.hayward@arm.com>
Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/include/asm/fpsimd.h      |  13 +-
 arch/arm64/include/uapi/asm/ptrace.h | 130 ++++++++++++++++
 arch/arm64/kernel/fpsimd.c           |  34 +++++
 arch/arm64/kernel/ptrace.c           | 288 ++++++++++++++++++++++++++++++++++-
 include/uapi/linux/elf.h             |   1 +
 5 files changed, 457 insertions(+), 9 deletions(-)

diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
index f43f573..ffb8a50 100644
--- a/arch/arm64/include/asm/fpsimd.h
+++ b/arch/arm64/include/asm/fpsimd.h
@@ -37,13 +37,16 @@ struct fpsimd_state {
 			__uint128_t vregs[32];
 			u32 fpsr;
 			u32 fpcr;
+			/*
+			 * For ptrace compatibility, pad to next 128-bit
+			 * boundary here if extending this struct.
+			 */
 		};
 	};
 	/* the id of the last cpu to have restored this state */
 	unsigned int cpu;
 };
 
-
 #if defined(__KERNEL__) && defined(CONFIG_COMPAT)
 /* Masks for extracting the FPSR and FPCR from the FPSCR */
 #define VFP_FPSCR_STAT_MASK	0xf800009f
@@ -87,6 +90,10 @@ extern void sve_alloc(struct task_struct *task);
 extern void fpsimd_release_thread(struct task_struct *task);
 extern void fpsimd_dup_sve(struct task_struct *dst,
 			   struct task_struct const *src);
+extern void fpsimd_sync_to_sve(struct task_struct *task);
+extern void sve_sync_to_fpsimd(struct task_struct *task);
+extern void sve_sync_from_fpsimd_zeropad(struct task_struct *task);
+
 extern int sve_set_vector_length(struct task_struct *task,
 				 unsigned long vl, unsigned long flags);
 
@@ -101,6 +108,10 @@ static void __maybe_unused sve_alloc(struct task_struct *task) { }
 static void __maybe_unused fpsimd_release_thread(struct task_struct *task) { }
 static void __maybe_unused fpsimd_dup_sve(struct task_struct *dst,
 					  struct task_struct const *src) { }
+static void __maybe_unused sve_sync_to_fpsimd(struct task_struct *task) { }
+static void __maybe_unused sve_sync_from_fpsimd_zeropad(
+	struct task_struct *task) { }
+
 static void __maybe_unused sve_init_vq_map(void) { }
 static void __maybe_unused sve_update_vq_map(void) { }
 static int __maybe_unused sve_verify_vq_map(void) { return 0; }
diff --git a/arch/arm64/include/uapi/asm/ptrace.h b/arch/arm64/include/uapi/asm/ptrace.h
index d1ff83d..16d9344 100644
--- a/arch/arm64/include/uapi/asm/ptrace.h
+++ b/arch/arm64/include/uapi/asm/ptrace.h
@@ -22,6 +22,7 @@
 #include <linux/types.h>
 
 #include <asm/hwcap.h>
+#include <asm/sigcontext.h>
 
 
 /*
@@ -63,6 +64,8 @@
 
 #ifndef __ASSEMBLY__
 
+#include <linux/prctl.h>
+
 /*
  * User structures for general purpose, floating point and debug registers.
  */
@@ -90,6 +93,133 @@ struct user_hwdebug_state {
 	}		dbg_regs[16];
 };
 
+/* SVE/FP/SIMD state (NT_ARM_SVE) */
+
+struct user_sve_header {
+	__u32 size; /* total meaningful regset content in bytes */
+	__u32 max_size; /* maxmium possible size for this thread */
+	__u16 vl; /* current vector length */
+	__u16 max_vl; /* maximum possible vector length */
+	__u16 flags;
+	__u16 __reserved;
+};
+
+/* Definitions for user_sve_header.flags: */
+#define SVE_PT_REGS_MASK		(1 << 0)
+
+/* Flags: must be kept in sync with prctl interface in <linux/ptrace.h> */
+#define SVE_PT_REGS_FPSIMD		0
+#define SVE_PT_REGS_SVE			SVE_PT_REGS_MASK
+
+#define SVE_PT_VL_INHERIT		(PR_SVE_VL_INHERIT >> 16)
+#define SVE_PT_VL_ONEXEC		(PR_SVE_SET_VL_ONEXEC >> 16)
+
+
+/*
+ * The remainder of the SVE state follows struct user_sve_header.  The
+ * total size of the SVE state (including header) depends on the
+ * metadata in the header:  SVE_PT_SIZE(vq, flags) gives the total size
+ * of the state in bytes, including the header.
+ *
+ * Refer to <asm/sigcontext.h> for details of how to pass the correct
+ * "vq" argument to these macros.
+ */
+
+/* Offset from the start of struct user_sve_header to the register data */
+#define SVE_PT_REGS_OFFSET	((sizeof(struct sve_context) + 15) / 16 * 16)
+
+/*
+ * The register data content and layout depends on the value of the
+ * flags field.
+ */
+
+/*
+ * (flags & SVE_PT_REGS_MASK) == SVE_PT_REGS_FPSIMD case:
+ *
+ * The payload starts at offset SVE_PT_FPSIMD_OFFSET, and is of type
+ * struct user_fpsimd_state.  Additional data might be appended in the
+ * future: use SVE_PT_FPSIMD_SIZE(vq, flags) to compute the total size.
+ * SVE_PT_FPSIMD_SIZE(vq, flags) will never be less than
+ * sizeof(struct user_fpsimd_state).
+ */
+
+#define SVE_PT_FPSIMD_OFFSET		SVE_PT_REGS_OFFSET
+
+#define SVE_PT_FPSIMD_SIZE(vq, flags)	(sizeof(struct user_fpsimd_state))
+
+/*
+ * (flags & SVE_PT_REGS_MASK) == SVE_PT_REGS_SVE case:
+ *
+ * The payload starts at offset SVE_PT_SVE_OFFSET, and is of size
+ * SVE_PT_SVE_SIZE(vq, flags).
+ *
+ * Additional macros describe the contents and layout of the payload.
+ * For each, SVE_PT_SVE_x_OFFSET(args) is the start offset relative to
+ * the start of struct user_sve_header, and SVE_PT_SVE_x_SIZE(args) is
+ * the size in bytes:
+ *
+ *	x	type				description
+ *	-	----				-----------
+ *	ZREGS		\
+ *	ZREG		|
+ *	PREGS		| refer to <asm/sigcontext.h>
+ *	PREG		|
+ *	FFR		/
+ *
+ *	FPSR	uint32_t			FPSR
+ *	FPCR	uint32_t			FPCR
+ *
+ * Additional data might be appended in the future.
+ */
+
+#define SVE_PT_SVE_ZREG_SIZE(vq)	SVE_SIG_ZREG_SIZE(vq)
+#define SVE_PT_SVE_PREG_SIZE(vq)	SVE_SIG_PREG_SIZE(vq)
+#define SVE_PT_SVE_FFR_SIZE(vq)		SVE_SIG_FFR_SIZE(vq)
+#define SVE_PT_SVE_FPSR_SIZE		sizeof(__u32)
+#define SVE_PT_SVE_FPCR_SIZE		sizeof(__u32)
+
+#define __SVE_SIG_TO_PT(offset) \
+	((offset) - SVE_SIG_REGS_OFFSET + SVE_PT_REGS_OFFSET)
+
+#define SVE_PT_SVE_OFFSET		SVE_PT_REGS_OFFSET
+
+#define SVE_PT_SVE_ZREGS_OFFSET \
+	__SVE_SIG_TO_PT(SVE_SIG_ZREGS_OFFSET)
+#define SVE_PT_SVE_ZREG_OFFSET(vq, n) \
+	__SVE_SIG_TO_PT(SVE_SIG_ZREG_OFFSET(vq, n))
+#define SVE_PT_SVE_ZREGS_SIZE(vq) \
+	(SVE_PT_SVE_ZREG_OFFSET(vq, SVE_NUM_ZREGS) - SVE_PT_SVE_ZREGS_OFFSET)
+
+#define SVE_PT_SVE_PREGS_OFFSET(vq) \
+	__SVE_SIG_TO_PT(SVE_SIG_PREGS_OFFSET(vq))
+#define SVE_PT_SVE_PREG_OFFSET(vq, n) \
+	__SVE_SIG_TO_PT(SVE_SIG_PREG_OFFSET(vq, n))
+#define SVE_PT_SVE_PREGS_SIZE(vq) \
+	(SVE_PT_SVE_PREG_OFFSET(vq, SVE_NUM_PREGS) - \
+		SVE_PT_SVE_PREGS_OFFSET(vq))
+
+#define SVE_PT_SVE_FFR_OFFSET(vq) \
+	__SVE_SIG_TO_PT(SVE_SIG_FFR_OFFSET(vq))
+
+#define SVE_PT_SVE_FPSR_OFFSET(vq) \
+	((SVE_PT_SVE_FFR_OFFSET(vq) + SVE_PT_SVE_FFR_SIZE(vq) + 15) / 16 * 16)
+#define SVE_PT_SVE_FPCR_OFFSET(vq) \
+	(SVE_PT_SVE_FPSR_OFFSET(vq) + SVE_PT_SVE_FPSR_SIZE)
+
+/*
+ * Any future extension appended after FPCR must be aligned to the next
+ * 128-bit boundary.
+ */
+
+#define SVE_PT_SVE_SIZE(vq, flags)				\
+	((SVE_PT_SVE_FPCR_OFFSET(vq) + SVE_PT_SVE_FPCR_SIZE -	\
+		SVE_PT_SVE_OFFSET + 15) / 16 * 16)
+
+#define SVE_PT_SIZE(vq, flags)						\
+	 (((flags) & SVE_PT_REGS_MASK) == SVE_PT_REGS_SVE ?		\
+		  SVE_PT_SVE_OFFSET + SVE_PT_SVE_SIZE(vq, flags)	\
+		: SVE_PT_FPSIMD_OFFSET + SVE_PT_FPSIMD_SIZE(vq, flags))
+
 #endif /* __ASSEMBLY__ */
 
 #endif /* _UAPI__ASM_PTRACE_H */
diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
index c727b47..5a7c870 100644
--- a/arch/arm64/kernel/fpsimd.c
+++ b/arch/arm64/kernel/fpsimd.c
@@ -268,6 +268,40 @@ void sve_alloc(struct task_struct *task)
 	BUG_ON(!task->thread.sve_state);
 }
 
+void fpsimd_sync_to_sve(struct task_struct *task)
+{
+	if (!test_tsk_thread_flag(task, TIF_SVE))
+		fpsimd_to_sve(task);
+}
+
+void sve_sync_to_fpsimd(struct task_struct *task)
+{
+	if (test_tsk_thread_flag(task, TIF_SVE))
+		sve_to_fpsimd(task);
+}
+
+void sve_sync_from_fpsimd_zeropad(struct task_struct *task)
+{
+	unsigned int vl, vq;
+	void *sst = task->thread.sve_state;
+	struct fpsimd_state const *fst = &task->thread.fpsimd_state;
+	unsigned int i;
+
+	if (!test_tsk_thread_flag(task, TIF_SVE))
+		return;
+
+	vl = task->thread.sve_vl;
+
+	BUG_ON(!sve_vl_valid(vl));
+	vq = sve_vq_from_vl(vl);
+
+	memset(sst, 0, SVE_SIG_REGS_SIZE(vq));
+
+	for (i = 0; i < 32; ++i)
+		memcpy(ZREG(sst, vq, i), &fst->vregs[i],
+		       sizeof(fst->vregs[i]));
+}
+
 /*
  * Handle SVE state across fork():
  *
diff --git a/arch/arm64/kernel/ptrace.c b/arch/arm64/kernel/ptrace.c
index 28619b5..abc05ed 100644
--- a/arch/arm64/kernel/ptrace.c
+++ b/arch/arm64/kernel/ptrace.c
@@ -32,6 +32,7 @@
 #include <linux/security.h>
 #include <linux/init.h>
 #include <linux/signal.h>
+#include <linux/string.h>
 #include <linux/uaccess.h>
 #include <linux/perf_event.h>
 #include <linux/hw_breakpoint.h>
@@ -40,6 +41,7 @@
 #include <linux/elf.h>
 
 #include <asm/compat.h>
+#include <asm/cpufeature.h>
 #include <asm/debug-monitors.h>
 #include <asm/pgtable.h>
 #include <asm/syscall.h>
@@ -617,33 +619,66 @@ static int gpr_set(struct task_struct *target, const struct user_regset *regset,
 /*
  * TODO: update fp accessors for lazy context switching (sync/flush hwstate)
  */
-static int fpr_get(struct task_struct *target, const struct user_regset *regset,
-		   unsigned int pos, unsigned int count,
-		   void *kbuf, void __user *ubuf)
+static int __fpr_get(struct task_struct *target,
+		     const struct user_regset *regset,
+		     unsigned int pos, unsigned int count,
+		     void *kbuf, void __user *ubuf, unsigned int start_pos)
 {
 	struct user_fpsimd_state *uregs;
+
+	sve_sync_to_fpsimd(target);
+
 	uregs = &target->thread.fpsimd_state.user_fpsimd;
 
+	return user_regset_copyout(&pos, &count, &kbuf, &ubuf, uregs,
+				   start_pos, start_pos + sizeof(*uregs));
+}
+
+static int fpr_get(struct task_struct *target, const struct user_regset *regset,
+		   unsigned int pos, unsigned int count,
+		   void *kbuf, void __user *ubuf)
+{
 	if (target == current)
 		fpsimd_preserve_current_state();
 
-	return user_regset_copyout(&pos, &count, &kbuf, &ubuf, uregs, 0, -1);
+	return __fpr_get(target, regset, pos, count, kbuf, ubuf, 0);
 }
 
-static int fpr_set(struct task_struct *target, const struct user_regset *regset,
-		   unsigned int pos, unsigned int count,
-		   const void *kbuf, const void __user *ubuf)
+static int __fpr_set(struct task_struct *target,
+		     const struct user_regset *regset,
+		     unsigned int pos, unsigned int count,
+		     const void *kbuf, const void __user *ubuf,
+		     unsigned int start_pos)
 {
 	int ret;
 	struct user_fpsimd_state newstate =
 		target->thread.fpsimd_state.user_fpsimd;
 
-	ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, &newstate, 0, -1);
+	sve_sync_to_fpsimd(target);
+
+	ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, &newstate,
+				 start_pos, start_pos + sizeof(newstate));
 	if (ret)
 		return ret;
 
 	target->thread.fpsimd_state.user_fpsimd = newstate;
+
+	return ret;
+}
+
+static int fpr_set(struct task_struct *target, const struct user_regset *regset,
+		   unsigned int pos, unsigned int count,
+		   const void *kbuf, const void __user *ubuf)
+{
+	int ret;
+
+	ret = __fpr_set(target, regset, pos, count, kbuf, ubuf, 0);
+	if (ret)
+		return ret;
+
+	sve_sync_from_fpsimd_zeropad(target);
 	fpsimd_flush_task_state(target);
+
 	return ret;
 }
 
@@ -701,6 +736,229 @@ static int system_call_set(struct task_struct *target,
 	return ret;
 }
 
+#ifdef CONFIG_ARM64_SVE
+
+static void sve_init_header_from_task(struct user_sve_header *header,
+				      struct task_struct *target)
+{
+	unsigned int vq;
+
+	memset(header, 0, sizeof(*header));
+
+	header->flags = test_tsk_thread_flag(target, TIF_SVE) ?
+		SVE_PT_REGS_SVE : SVE_PT_REGS_FPSIMD;
+	if (test_tsk_thread_flag(target, TIF_SVE_VL_INHERIT))
+		header->flags |= SVE_PT_VL_INHERIT;
+
+	header->vl = target->thread.sve_vl;
+
+	BUG_ON(!sve_vl_valid(header->vl));
+	vq = sve_vq_from_vl(header->vl);
+
+	BUG_ON(!sve_vl_valid(sve_max_vl));
+	header->max_vl = sve_max_vl;
+
+	header->size = SVE_PT_SIZE(vq, header->flags);
+	header->max_size = SVE_PT_SIZE(sve_vq_from_vl(header->max_vl),
+				      SVE_PT_REGS_SVE);
+}
+
+static unsigned int sve_size_from_header(struct user_sve_header const *header)
+{
+	return (header->size + 15) / 16 * 16;
+}
+
+static unsigned int sve_get_size(struct task_struct *target,
+				 const struct user_regset *regset)
+{
+	struct user_sve_header header;
+
+	if (!system_supports_sve())
+		return 0;
+
+	sve_init_header_from_task(&header, target);
+	return sve_size_from_header(&header);
+}
+
+static int sve_get(struct task_struct *target,
+		   const struct user_regset *regset,
+		   unsigned int pos, unsigned int count,
+		   void *kbuf, void __user *ubuf)
+{
+	int ret;
+	struct user_sve_header header;
+	unsigned int vq;
+	unsigned long start, end;
+
+	if (!system_supports_sve())
+		return -EINVAL;
+
+	/* Header */
+	sve_init_header_from_task(&header, target);
+
+	BUG_ON(!sve_vl_valid(header.vl));
+	vq = sve_vq_from_vl(header.vl);
+
+	ret = user_regset_copyout(&pos, &count, &kbuf, &ubuf, &header,
+				  0, sizeof(header));
+	if (ret)
+		return ret;
+
+	if (target == current)
+		fpsimd_preserve_current_state();
+
+	/* Registers: FPSIMD-only case */
+
+	BUILD_BUG_ON(SVE_PT_FPSIMD_OFFSET != sizeof(header));
+
+	if ((header.flags & SVE_PT_REGS_MASK) == SVE_PT_REGS_FPSIMD)
+		return __fpr_get(target, regset, pos, count, kbuf, ubuf,
+				 SVE_PT_FPSIMD_OFFSET);
+
+	/* Otherwise: full SVE case */
+
+	BUILD_BUG_ON(SVE_PT_SVE_OFFSET != sizeof(header));
+
+	start = SVE_PT_SVE_OFFSET;
+	end = SVE_PT_SVE_FFR_OFFSET(vq) + SVE_PT_SVE_FFR_SIZE(vq);
+
+	BUG_ON(end < start);
+	BUG_ON(end - start > sve_state_size(target));
+	ret = user_regset_copyout(&pos, &count, &kbuf, &ubuf,
+				  target->thread.sve_state,
+				  start, end);
+	if (ret)
+		return ret;
+
+	start = end;
+	end = SVE_PT_SVE_FPSR_OFFSET(vq);
+
+	BUG_ON(end < start);
+	ret = user_regset_copyout_zero(&pos, &count, &kbuf, &ubuf,
+				       start, end);
+	if (ret)
+		return ret;
+
+	start = end;
+	end = SVE_PT_SVE_FPCR_OFFSET(vq) + SVE_PT_SVE_FPCR_SIZE;
+
+	BUG_ON((char *)(&target->thread.fpsimd_state.fpcr + 1) <
+	       (char *)&target->thread.fpsimd_state.fpsr);
+	BUG_ON(end < start);
+	BUG_ON((char *)(&target->thread.fpsimd_state.fpcr + 1) -
+	       (char *)&target->thread.fpsimd_state.fpsr !=
+		end - start);
+
+	ret = user_regset_copyout(&pos, &count, &kbuf, &ubuf,
+				  &target->thread.fpsimd_state.fpsr,
+				  start, end);
+	if (ret)
+		return ret;
+
+	start = end;
+	end = sve_size_from_header(&header);
+	BUG_ON(end < start);
+
+	return user_regset_copyout_zero(&pos, &count, &kbuf, &ubuf,
+					start, end);
+}
+
+static int sve_set(struct task_struct *target,
+		   const struct user_regset *regset,
+		   unsigned int pos, unsigned int count,
+		   const void *kbuf, const void __user *ubuf)
+{
+	int ret;
+	struct user_sve_header header;
+	unsigned int vq;
+	unsigned long start, end;
+
+	if (!system_supports_sve())
+		return -EINVAL;
+
+	/* Header */
+	if (count < sizeof(header))
+		return -EINVAL;
+	ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf, &header,
+				 0, sizeof(header));
+	if (ret)
+		goto out;
+
+	/*
+	 * Apart from PT_SVE_REGS_MASK, all PT_SVE_* flags are consumed by
+	 * sve_set_vector_length(), which will also validate them for us:
+	 */
+	ret = sve_set_vector_length(target, header.vl,
+				    header.flags & ~SVE_PT_REGS_MASK);
+	if (ret)
+		goto out;
+
+	/* Actual VL set may be less than the user asked for: */
+	BUG_ON(!sve_vl_valid(target->thread.sve_vl));
+	vq = sve_vq_from_vl(target->thread.sve_vl);
+
+	/* Registers: FPSIMD-only case */
+
+	BUILD_BUG_ON(SVE_PT_FPSIMD_OFFSET != sizeof(header));
+
+	if ((header.flags & SVE_PT_REGS_MASK) == SVE_PT_REGS_FPSIMD) {
+		sve_sync_to_fpsimd(target);
+
+		ret = __fpr_set(target, regset, pos, count, kbuf, ubuf,
+				SVE_PT_FPSIMD_OFFSET);
+		clear_tsk_thread_flag(target, TIF_SVE);
+		goto out;
+	}
+
+	/* Otherwise: full SVE case */
+
+	sve_alloc(target);
+	fpsimd_sync_to_sve(target);
+	set_tsk_thread_flag(target, TIF_SVE);
+
+	BUILD_BUG_ON(SVE_PT_SVE_OFFSET != sizeof(header));
+
+	start = SVE_PT_SVE_OFFSET;
+	end = SVE_PT_SVE_FFR_OFFSET(vq) + SVE_PT_SVE_FFR_SIZE(vq);
+
+	BUG_ON(end < start);
+	BUG_ON(end - start > sve_state_size(target));
+	ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf,
+				 target->thread.sve_state,
+				 start, end);
+	if (ret)
+		goto out;
+
+	start = end;
+	end = SVE_PT_SVE_FPSR_OFFSET(vq);
+
+	BUG_ON(end < start);
+	ret = user_regset_copyin_ignore(&pos, &count, &kbuf, &ubuf,
+					start, end);
+	if (ret)
+		goto out;
+
+	start = end;
+	end = SVE_PT_SVE_FPCR_OFFSET(vq) + SVE_PT_SVE_FPCR_SIZE;
+
+	BUG_ON((char *)(&target->thread.fpsimd_state.fpcr + 1) <
+		(char *)&target->thread.fpsimd_state.fpsr);
+	BUG_ON(end < start);
+	BUG_ON((char *)(&target->thread.fpsimd_state.fpcr + 1) -
+	       (char *)&target->thread.fpsimd_state.fpsr !=
+		end - start);
+
+	ret = user_regset_copyin(&pos, &count, &kbuf, &ubuf,
+				 &target->thread.fpsimd_state.fpsr,
+				 start, end);
+
+out:
+	fpsimd_flush_task_state(target);
+	return ret;
+}
+
+#endif /* CONFIG_ARM64_SVE */
+
 enum aarch64_regset {
 	REGSET_GPR,
 	REGSET_FPR,
@@ -710,6 +968,9 @@ enum aarch64_regset {
 	REGSET_HW_WATCH,
 #endif
 	REGSET_SYSTEM_CALL,
+#ifdef CONFIG_ARM64_SVE
+	REGSET_SVE,
+#endif
 };
 
 static const struct user_regset aarch64_regsets[] = {
@@ -767,6 +1028,17 @@ static const struct user_regset aarch64_regsets[] = {
 		.get = system_call_get,
 		.set = system_call_set,
 	},
+#ifdef CONFIG_ARM64_SVE
+	[REGSET_SVE] = { /* Scalable Vector Extension */
+		.core_note_type = NT_ARM_SVE,
+		.n = (SVE_PT_SIZE(SVE_VQ_MAX, SVE_PT_REGS_SVE) + 15) / 16,
+		.size = 16,
+		.align = 16,
+		.get = sve_get,
+		.set = sve_set,
+		.get_size = sve_get_size,
+	},
+#endif
 };
 
 static const struct user_regset_view user_aarch64_view = {
diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h
index b5280db..735b8f4 100644
--- a/include/uapi/linux/elf.h
+++ b/include/uapi/linux/elf.h
@@ -416,6 +416,7 @@ typedef struct elf64_shdr {
 #define NT_ARM_HW_BREAK	0x402		/* ARM hardware breakpoint registers */
 #define NT_ARM_HW_WATCH	0x403		/* ARM hardware watchpoint registers */
 #define NT_ARM_SYSTEM_CALL	0x404	/* ARM system call number */
+#define NT_ARM_SVE	0x405		/* ARM Scalable Vector Extension registers */
 #define NT_METAG_CBUF	0x500		/* Metag catch buffer registers */
 #define NT_METAG_RPIPE	0x501		/* Metag read pipeline state */
 #define NT_METAG_TLS	0x502		/* Metag TLS pointer */
-- 
2.1.4

[PATCH 19/27] arm64/sve: Add prctl controls for userspace vector length management

[Dave Martin]

This patch adds two arm64-specific prctls, to permit userspace to
control its vector length:

 * PR_SVE_SET_VL: set the thread's SVE vector length and vector
   length inheritance mode.

 * PR_SVE_GET_VL: get the same information.

Although these calls shadow instruction set features in the SVE
architecture, these prctls provide additional control: the vector
length inheritance mode is Linux-specific and nothing to do with
the architecture, and the architecture does not permit EL0 to set
its own vector length directly.  Both can be used in portable tools
without requiring the use of SVE instructions.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/include/asm/fpsimd.h    | 14 ++++++++++++
 arch/arm64/include/asm/processor.h |  4 ++++
 arch/arm64/kernel/fpsimd.c         | 46 ++++++++++++++++++++++++++++++++++++++
 include/uapi/linux/prctl.h         |  4 ++++
 kernel/sys.c                       | 12 ++++++++++
 5 files changed, 80 insertions(+)

diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
index ffb8a50..b58aa45 100644
--- a/arch/arm64/include/asm/fpsimd.h
+++ b/arch/arm64/include/asm/fpsimd.h
@@ -17,6 +17,7 @@
 #define __ASM_FP_H
 
 #include <asm/ptrace.h>
+#include <asm/errno.h>
 
 #ifndef __ASSEMBLY__
 
@@ -97,6 +98,9 @@ extern void sve_sync_from_fpsimd_zeropad(struct task_struct *task);
 extern int sve_set_vector_length(struct task_struct *task,
 				 unsigned long vl, unsigned long flags);
 
+extern int sve_set_current_vl(unsigned long arg);
+extern int sve_get_current_vl(void);
+
 extern void __init sve_init_vq_map(void);
 extern void sve_update_vq_map(void);
 extern int sve_verify_vq_map(void);
@@ -112,6 +116,16 @@ static void __maybe_unused sve_sync_to_fpsimd(struct task_struct *task) { }
 static void __maybe_unused sve_sync_from_fpsimd_zeropad(
 	struct task_struct *task) { }
 
+static int __maybe_unused sve_set_current_vl(unsigned long arg)
+{
+	return -EINVAL;
+}
+
+static int __maybe_unused sve_get_current_vl(void)
+{
+	return -EINVAL;
+}
+
 static void __maybe_unused sve_init_vq_map(void) { }
 static void __maybe_unused sve_update_vq_map(void) { }
 static int __maybe_unused sve_verify_vq_map(void) { return 0; }
diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
index da8802a..70b03b4 100644
--- a/arch/arm64/include/asm/processor.h
+++ b/arch/arm64/include/asm/processor.h
@@ -197,4 +197,8 @@ static inline void spin_lock_prefetch(const void *ptr)
 int cpu_enable_pan(void *__unused);
 int cpu_enable_cache_maint_trap(void *__unused);
 
+/* Userspace interface for PR_SVE_{SET,GET}_VL prctl()s: */
+#define SVE_SET_VL(arg)	sve_set_current_vl(arg)
+#define SVE_GET_VL()	sve_get_current_vl()
+
 #endif /* __ASM_PROCESSOR_H */
diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
index 5a7c870..4fe1675 100644
--- a/arch/arm64/kernel/fpsimd.c
+++ b/arch/arm64/kernel/fpsimd.c
@@ -24,6 +24,7 @@
 #include <linux/kernel.h>
 #include <linux/init.h>
 #include <linux/percpu.h>
+#include <linux/prctl.h>
 #include <linux/preempt.h>
 #include <linux/prctl.h>
 #include <linux/ptrace.h>
@@ -387,6 +388,51 @@ int sve_set_vector_length(struct task_struct *task,
 	return 0;
 }
 
+/*
+ * Encode the current vector length and flags for return.
+ * This is only required for prctl(): ptrace has separate fields
+ */
+static int sve_prctl_status(void)
+{
+	int ret = current->thread.sve_vl;
+
+	if (test_thread_flag(TIF_SVE_VL_INHERIT))
+		ret |= PR_SVE_VL_INHERIT;
+
+	return ret;
+}
+
+/* PR_SVE_SET_VL */
+int sve_set_current_vl(unsigned long arg)
+{
+	unsigned long vl, flags;
+	int ret;
+
+	vl = arg & PR_SVE_VL_LEN_MASK;
+	flags = arg & ~vl;
+
+	if (!system_supports_sve())
+		return -EINVAL;
+
+	preempt_disable();
+	ret = sve_set_vector_length(current, vl, flags);
+	preempt_enable();
+
+	if (ret)
+		return ret;
+
+	return sve_prctl_status();
+}
+
+/* PR_SVE_GET_VL */
+int sve_get_current_vl(void)
+{
+	if (!system_supports_sve())
+		return -EINVAL;
+
+	return sve_prctl_status();
+}
+
 static unsigned long *sve_alloc_vq_map(void)
 {
 	return kzalloc(BITS_TO_LONGS(SVE_VQ_MAX) * sizeof(unsigned long),
diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h
index 1b64901..1ef9370 100644
--- a/include/uapi/linux/prctl.h
+++ b/include/uapi/linux/prctl.h
@@ -198,7 +198,11 @@ struct prctl_mm_map {
 # define PR_CAP_AMBIENT_CLEAR_ALL	4
 
 /* arm64 Scalable Vector Extension controls */
+/* Flag values must be kept in sync with ptrace NT_ARM_SVE interface */
+#define PR_SVE_SET_VL			48	/* set task vector length */
 # define PR_SVE_SET_VL_ONEXEC		(1 << 18) /* defer effect until exec */
+#define PR_SVE_GET_VL			49	/* get task vector length */
+/* Bits common to PR_SVE_SET_VL and PR_SVE_GET_VL */
 # define PR_SVE_VL_LEN_MASK		0xffff
 # define PR_SVE_VL_INHERIT		(1 << 17) /* inherit across exec */
 
diff --git a/kernel/sys.c b/kernel/sys.c
index 2855ee7..f8215a6 100644
--- a/kernel/sys.c
+++ b/kernel/sys.c
@@ -110,6 +110,12 @@
 #ifndef SET_FP_MODE
 # define SET_FP_MODE(a,b)	(-EINVAL)
 #endif
+#ifndef SVE_SET_VL
+# define SVE_SET_VL(a)		(-EINVAL)
+#endif
+#ifndef SVE_GET_VL
+# define SVE_GET_VL()		(-EINVAL)
+#endif
 
 /*
  * this is where the system-wide overflow UID and GID are defined, for
@@ -2389,6 +2395,12 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
 	case PR_GET_FP_MODE:
 		error = GET_FP_MODE(me);
 		break;
+	case PR_SVE_SET_VL:
+		error = SVE_SET_VL(arg2);
+		break;
+	case PR_SVE_GET_VL:
+		error = SVE_GET_VL();
+		break;
 	default:
 		error = -EINVAL;
 		break;
-- 
2.1.4

[PATCH 20/27] arm64/sve: Add sysctl to set the default vector length for new processes

[Dave Martin]

Because of the effect of SVE on the size of the signal frame, the
default vector length used for new processes involves a tradeoff
between performance of SVE-enabled software on the one hand, and
reliability of non-SVE-aware software on the other hand.

For this reason, the best choice depends on the repertoire of
userspace software in use and is thus best left up to distro
maintainers, sysadmins and developers.

If CONFIG_SYSCTL is enabled, this patch exposes the default vector
length in /proc/sys/abi/sve_default_vector_length, where boot
scripts or the adventurous can poke it.

In common with other arm64 ABI sysctls, this control is currently
global: setting it requires CAP_SYS_ADMIN in the root user
namespace, but the value set is effective for subsequent execs in
all namespaces.  The control only affects _new_ processes, however:
changing it does not affect the vector length of any existing
process.

The intended usage model is that if userspace is known to be fully
SVE-tolerant (or a developer is curious to find out) then init
scripts can crank this up during startup.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/kernel/fpsimd.c | 57 +++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 56 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
index 4fe1675..d45fcfb 100644
--- a/arch/arm64/kernel/fpsimd.c
+++ b/arch/arm64/kernel/fpsimd.c
@@ -31,6 +31,7 @@
 #include <linux/sched/signal.h>
 #include <linux/signal.h>
 #include <linux/slab.h>
+#include <linux/sysctl.h>
 
 #include <asm/fpsimd.h>
 #include <asm/cputype.h>
@@ -200,6 +201,60 @@ static unsigned int find_supported_vector_length(unsigned int vl)
 	return 16 * bit_to_vq(bit);
 }
 
+#ifdef CONFIG_SYSCTL
+
+static int sve_proc_do_default_vl(struct ctl_table *table, int write,
+				  void __user *buffer, size_t *lenp,
+				  loff_t *ppos)
+{
+	int ret;
+	int vl = sve_default_vl;
+	struct ctl_table tmp_table = {
+		.data = &vl,
+		.maxlen = sizeof(vl),
+	};
+
+	ret = proc_dointvec(&tmp_table, write, buffer, lenp, ppos);
+	if (ret || !write)
+		return ret;
+
+	/* Writing -1 has the special meaning "set to max": */
+	if (vl == -1) {
+		BUG_ON(find_supported_vector_length(sve_max_vl) != sve_max_vl);
+		sve_default_vl = sve_max_vl;
+
+		return 0;
+	}
+
+	if (!sve_vl_valid(vl))
+		return -EINVAL;
+
+	sve_default_vl = find_supported_vector_length(vl);
+	return 0;
+}
+
+static struct ctl_table sve_default_vl_table[] = {
+	{
+		.procname	= "sve_default_vector_length",
+		.mode		= 0644,
+		.proc_handler	= sve_proc_do_default_vl,
+	},
+	{ }
+};
+
+static int __init sve_sysctl_init(void)
+{
+	if (system_supports_sve())
+		if (!register_sysctl("abi", sve_default_vl_table))
+			return -EINVAL;
+
+	return 0;
+}
+
+#else /* ! CONFIG_SYSCTL */
+static int __init sve_sysctl_init(void) { return 0; }
+#endif /* ! CONFIG_SYSCTL */
+
 #define ZREG(sve_state, vq, n) ((char *)(sve_state) +		\
 	(SVE_SIG_ZREG_OFFSET(vq, n) - SVE_SIG_REGS_OFFSET))
 
@@ -1037,6 +1092,6 @@ static int __init fpsimd_init(void)
 	if (!(elf_hwcap & HWCAP_ASIMD))
 		pr_notice("Advanced SIMD is not implemented\n");
 
-	return 0;
+	return sve_sysctl_init();
 }
 late_initcall(fpsimd_init);
-- 
2.1.4

[PATCH 21/27] arm64/sve: KVM: Prevent guests from using SVE

[Dave Martin]

Until KVM has full SVE support, guests must not be allowed to
execute SVE instructions.

This patch enables the necessary traps, and also ensures that the
traps are disabled again on exit from the guest so that the host
can still use SVE if it wants to.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/include/asm/kvm_arm.h | 3 ++-
 arch/arm64/kvm/hyp/switch.c      | 6 +++---
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
index dbf0537..8a19651 100644
--- a/arch/arm64/include/asm/kvm_arm.h
+++ b/arch/arm64/include/asm/kvm_arm.h
@@ -186,7 +186,7 @@
 #define CPTR_EL2_TTA	(1 << 20)
 #define CPTR_EL2_TFP	(1 << CPTR_EL2_TFP_SHIFT)
 #define CPTR_EL2_TZ	(1 << 8)
-#define CPTR_EL2_DEFAULT	0x000033ff
+#define CPTR_EL2_DEFAULT	(0x000033ff & ~CPTR_EL2_TZ)
 
 /* Hyp Debug Configuration Register bits */
 #define MDCR_EL2_TPMS		(1 << 14)
@@ -237,5 +237,6 @@
 
 #define CPACR_EL1_FPEN		(3 << 20)
 #define CPACR_EL1_TTA		(1 << 28)
+#define CPACR_EL1_DEFAULT	(CPACR_EL1_FPEN | CPACR_EL1_ZEN_EL1EN)
 
 #endif /* __ARM64_KVM_ARM_H__ */
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 35a90b8..951f3eb 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -48,7 +48,7 @@ static void __hyp_text __activate_traps_vhe(void)
 
 	val = read_sysreg(cpacr_el1);
 	val |= CPACR_EL1_TTA;
-	val &= ~CPACR_EL1_FPEN;
+	val &= ~(CPACR_EL1_FPEN | CPACR_EL1_ZEN);
 	write_sysreg(val, cpacr_el1);
 
 	write_sysreg(__kvm_hyp_vector, vbar_el1);
@@ -59,7 +59,7 @@ static void __hyp_text __activate_traps_nvhe(void)
 	u64 val;
 
 	val = CPTR_EL2_DEFAULT;
-	val |= CPTR_EL2_TTA | CPTR_EL2_TFP;
+	val |= CPTR_EL2_TTA | CPTR_EL2_TFP | CPTR_EL2_TZ;
 	write_sysreg(val, cptr_el2);
 }
 
@@ -117,7 +117,7 @@ static void __hyp_text __deactivate_traps_vhe(void)
 
 	write_sysreg(mdcr_el2, mdcr_el2);
 	write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);
-	write_sysreg(CPACR_EL1_FPEN, cpacr_el1);
+	write_sysreg(CPACR_EL1_DEFAULT, cpacr_el1);
 	write_sysreg(vectors, vbar_el1);
 }
 
-- 
2.1.4

[PATCH 21/27] arm64/sve: KVM: Prevent guests from using SVE

[Dave Martin]

Until KVM has full SVE support, guests must not be allowed to
execute SVE instructions.

This patch enables the necessary traps, and also ensures that the
traps are disabled again on exit from the guest so that the host
can still use SVE if it wants to.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/include/asm/kvm_arm.h | 3 ++-
 arch/arm64/kvm/hyp/switch.c      | 6 +++---
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
index dbf0537..8a19651 100644
--- a/arch/arm64/include/asm/kvm_arm.h
+++ b/arch/arm64/include/asm/kvm_arm.h
@@ -186,7 +186,7 @@
 #define CPTR_EL2_TTA	(1 << 20)
 #define CPTR_EL2_TFP	(1 << CPTR_EL2_TFP_SHIFT)
 #define CPTR_EL2_TZ	(1 << 8)
-#define CPTR_EL2_DEFAULT	0x000033ff
+#define CPTR_EL2_DEFAULT	(0x000033ff & ~CPTR_EL2_TZ)
 
 /* Hyp Debug Configuration Register bits */
 #define MDCR_EL2_TPMS		(1 << 14)
@@ -237,5 +237,6 @@
 
 #define CPACR_EL1_FPEN		(3 << 20)
 #define CPACR_EL1_TTA		(1 << 28)
+#define CPACR_EL1_DEFAULT	(CPACR_EL1_FPEN | CPACR_EL1_ZEN_EL1EN)
 
 #endif /* __ARM64_KVM_ARM_H__ */
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 35a90b8..951f3eb 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -48,7 +48,7 @@ static void __hyp_text __activate_traps_vhe(void)
 
 	val = read_sysreg(cpacr_el1);
 	val |= CPACR_EL1_TTA;
-	val &= ~CPACR_EL1_FPEN;
+	val &= ~(CPACR_EL1_FPEN | CPACR_EL1_ZEN);
 	write_sysreg(val, cpacr_el1);
 
 	write_sysreg(__kvm_hyp_vector, vbar_el1);
@@ -59,7 +59,7 @@ static void __hyp_text __activate_traps_nvhe(void)
 	u64 val;
 
 	val = CPTR_EL2_DEFAULT;
-	val |= CPTR_EL2_TTA | CPTR_EL2_TFP;
+	val |= CPTR_EL2_TTA | CPTR_EL2_TFP | CPTR_EL2_TZ;
 	write_sysreg(val, cptr_el2);
 }
 
@@ -117,7 +117,7 @@ static void __hyp_text __deactivate_traps_vhe(void)
 
 	write_sysreg(mdcr_el2, mdcr_el2);
 	write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);
-	write_sysreg(CPACR_EL1_FPEN, cpacr_el1);
+	write_sysreg(CPACR_EL1_DEFAULT, cpacr_el1);
 	write_sysreg(vectors, vbar_el1);
 }
 
-- 
2.1.4

[PATCH 22/27] arm64/sve: KVM: Treat guest SVE use as undefined instruction execution

[Dave Martin]

When trapping forbidden attempts by a guest to use SVE, we want the
guest to see a trap consistent with SVE not being implemented.

This patch injects an undefined instruction exception into the
guest in response to such an exception.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/kvm/handle_exit.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c
index 17d8a16..e3e42d0 100644
--- a/arch/arm64/kvm/handle_exit.c
+++ b/arch/arm64/kvm/handle_exit.c
@@ -147,6 +147,13 @@ static int kvm_handle_unknown_ec(struct kvm_vcpu *vcpu, struct kvm_run *run)
 	return 1;
 }
 
+static int handle_sve(struct kvm_vcpu *vcpu, struct kvm_run *run)
+{
+	/* Until SVE is supported for guests: */
+	kvm_inject_undefined(vcpu);
+	return 1;
+}
+
 static exit_handle_fn arm_exit_handlers[] = {
 	[0 ... ESR_ELx_EC_MAX]	= kvm_handle_unknown_ec,
 	[ESR_ELx_EC_WFx]	= kvm_handle_wfx,
@@ -160,6 +167,7 @@ static exit_handle_fn arm_exit_handlers[] = {
 	[ESR_ELx_EC_HVC64]	= handle_hvc,
 	[ESR_ELx_EC_SMC64]	= handle_smc,
 	[ESR_ELx_EC_SYS64]	= kvm_handle_sys_reg,
+	[ESR_ELx_EC_SVE]	= handle_sve,
 	[ESR_ELx_EC_IABT_LOW]	= kvm_handle_guest_abort,
 	[ESR_ELx_EC_DABT_LOW]	= kvm_handle_guest_abort,
 	[ESR_ELx_EC_SOFTSTP_LOW]= kvm_handle_guest_debug,
-- 
2.1.4

[PATCH 22/27] arm64/sve: KVM: Treat guest SVE use as undefined instruction execution

[Dave Martin]

When trapping forbidden attempts by a guest to use SVE, we want the
guest to see a trap consistent with SVE not being implemented.

This patch injects an undefined instruction exception into the
guest in response to such an exception.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/kvm/handle_exit.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c
index 17d8a16..e3e42d0 100644
--- a/arch/arm64/kvm/handle_exit.c
+++ b/arch/arm64/kvm/handle_exit.c
@@ -147,6 +147,13 @@ static int kvm_handle_unknown_ec(struct kvm_vcpu *vcpu, struct kvm_run *run)
 	return 1;
 }
 
+static int handle_sve(struct kvm_vcpu *vcpu, struct kvm_run *run)
+{
+	/* Until SVE is supported for guests: */
+	kvm_inject_undefined(vcpu);
+	return 1;
+}
+
 static exit_handle_fn arm_exit_handlers[] = {
 	[0 ... ESR_ELx_EC_MAX]	= kvm_handle_unknown_ec,
 	[ESR_ELx_EC_WFx]	= kvm_handle_wfx,
@@ -160,6 +167,7 @@ static exit_handle_fn arm_exit_handlers[] = {
 	[ESR_ELx_EC_HVC64]	= handle_hvc,
 	[ESR_ELx_EC_SMC64]	= handle_smc,
 	[ESR_ELx_EC_SYS64]	= kvm_handle_sys_reg,
+	[ESR_ELx_EC_SVE]	= handle_sve,
 	[ESR_ELx_EC_IABT_LOW]	= kvm_handle_guest_abort,
 	[ESR_ELx_EC_DABT_LOW]	= kvm_handle_guest_abort,
 	[ESR_ELx_EC_SOFTSTP_LOW]= kvm_handle_guest_debug,
-- 
2.1.4

[PATCH 23/27] arm64/sve: KVM: Hide SVE from CPU features exposed to guests

[Dave Martin]

KVM guests cannot currently use SVE, because SVE is always
configured to trap to EL2.

However, a guest that sees SVE reported as present in
ID_AA64PFR0_EL1 may legitimately expect that SVE works and try to
use it.  Instead of working, the guest will receive an injected
undef exception, which may cause the guest to oops or go into a
spin.

To avoid misleading the guest into believing that SVE will work,
this patch masks out the SVE field from ID_AA64PFR0_EL1 when a
guest attempts to read this register.  No support is explicitly
added for ID_AA64ZFR0_EL1 either, so that is still emulated as
reading as zero, which is consistent with SVE not being
implemented.

This is a temporary measure, and will be removed in a later series
when full KVM support for SVE is implemented.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/kvm/sys_regs.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 6583dd7..9e8c54e 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -897,8 +897,20 @@ static u64 read_id_reg(struct sys_reg_desc const *r, bool raz)
 {
 	u32 id = sys_reg((u32)r->Op0, (u32)r->Op1,
 			 (u32)r->CRn, (u32)r->CRm, (u32)r->Op2);
+	u64 val = raz ? 0 : read_sanitised_ftr_reg(id);
 
-	return raz ? 0 : read_sanitised_ftr_reg(id);
+	if (id == SYS_ID_AA64PFR0_EL1) {
+		static bool printed;
+
+		if ((val & (0xfUL << ID_AA64PFR0_SVE_SHIFT)) && !printed) {
+			kvm_info("SVE unsupported for guests, suppressing\n");
+			printed = true;
+		}
+
+		val &= ~(0xfUL << ID_AA64PFR0_SVE_SHIFT);
+	}
+
+	return val;
 }
 
 /* cpufeature ID register access trap handlers */
-- 
2.1.4

[PATCH 24/27] arm64/sve: Detect SVE and activate runtime support

[Dave Martin]

This patch enables detection of hardware SVE support via the
cpufeatures framework, and reports its presence to the kernel and
userspace via the new ARM64_SVE cpucap and HWCAP_SVE hwcap
respectively.

Userspace can also detect SVE using ID_AA64PFR0_EL1, using the
cpufeatures MRS emulation.

When running on hardware that supports SVE, this enables runtime
kernel support for SVE, and allows user tasks to execute SVE
instructions and make of the of the SVE-specific user/kernel
interface extensions implemented by this series.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/include/asm/cpucaps.h    |  3 ++-
 arch/arm64/include/asm/cpufeature.h |  3 ++-
 arch/arm64/include/uapi/asm/hwcap.h |  1 +
 arch/arm64/kernel/cpufeature.c      | 16 ++++++++++++++++
 arch/arm64/kernel/cpuinfo.c         |  1 +
 5 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index 8d2272c..0b4ce6a 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -39,7 +39,8 @@
 #define ARM64_WORKAROUND_QCOM_FALKOR_E1003	18
 #define ARM64_WORKAROUND_858921			19
 #define ARM64_WORKAROUND_CAVIUM_30115		20
+#define ARM64_SVE				21
 
-#define ARM64_NCAPS				21
+#define ARM64_NCAPS				22
 
 #endif /* __ASM_CPUCAPS_H */
diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index 05eec27..27182bb 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -272,7 +272,8 @@ static inline bool system_uses_ttbr0_pan(void)
 
 static inline bool system_supports_sve(void)
 {
-	return false;
+	return IS_ENABLED(CONFIG_ARM64_SVE) &&
+		cpus_have_const_cap(ARM64_SVE);
 }
 
 /*
diff --git a/arch/arm64/include/uapi/asm/hwcap.h b/arch/arm64/include/uapi/asm/hwcap.h
index 4e187ce..7d2ec07 100644
--- a/arch/arm64/include/uapi/asm/hwcap.h
+++ b/arch/arm64/include/uapi/asm/hwcap.h
@@ -35,5 +35,6 @@
 #define HWCAP_JSCVT		(1 << 13)
 #define HWCAP_FCMA		(1 << 14)
 #define HWCAP_LRCPC		(1 << 15)
+#define HWCAP_SVE		(1 << 16)
 
 #endif /* _UAPI__ASM_HWCAP_H */
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index f322c90..390cb73 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -125,6 +125,7 @@ static const struct arm64_ftr_bits ftr_id_aa64isar1[] = {
 };
 
 static const struct arm64_ftr_bits ftr_id_aa64pfr0[] = {
+	ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_SVE_SHIFT, 4, 0),
 	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_EXACT, ID_AA64PFR0_GIC_SHIFT, 4, 0),
 	S_ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_ASIMD_SHIFT, 4, ID_AA64PFR0_ASIMD_NI),
 	S_ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_FP_SHIFT, 4, ID_AA64PFR0_FP_NI),
@@ -913,6 +914,18 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.min_field_value = 0,
 		.matches = has_no_fpsimd,
 	},
+#ifdef CONFIG_ARM64_SVE
+	{
+		.desc = "Scalable Vector Extension",
+		.capability = ARM64_SVE,
+		.def_scope = SCOPE_SYSTEM,
+		.sys_reg = SYS_ID_AA64PFR0_EL1,
+		.sign = FTR_UNSIGNED,
+		.field_pos = ID_AA64PFR0_SVE_SHIFT,
+		.min_field_value = ID_AA64PFR0_SVE,
+		.matches = has_cpuid_feature,
+	},
+#endif /* CONFIG_ARM64_SVE */
 	{},
 };
 
@@ -944,6 +957,9 @@ static const struct arm64_cpu_capabilities arm64_elf_hwcaps[] = {
 	HWCAP_CAP(SYS_ID_AA64ISAR1_EL1, ID_AA64ISAR1_JSCVT_SHIFT, FTR_UNSIGNED, 1, CAP_HWCAP, HWCAP_JSCVT),
 	HWCAP_CAP(SYS_ID_AA64ISAR1_EL1, ID_AA64ISAR1_FCMA_SHIFT, FTR_UNSIGNED, 1, CAP_HWCAP, HWCAP_FCMA),
 	HWCAP_CAP(SYS_ID_AA64ISAR1_EL1, ID_AA64ISAR1_LRCPC_SHIFT, FTR_UNSIGNED, 1, CAP_HWCAP, HWCAP_LRCPC),
+#ifdef CONFIG_ARM64_SVE
+	HWCAP_CAP(SYS_ID_AA64PFR0_EL1, ID_AA64PFR0_SVE_SHIFT, FTR_UNSIGNED, ID_AA64PFR0_SVE, CAP_HWCAP, HWCAP_SVE),
+#endif
 	{},
 };
 
diff --git a/arch/arm64/kernel/cpuinfo.c b/arch/arm64/kernel/cpuinfo.c
index 0e087d7..943aad0 100644
--- a/arch/arm64/kernel/cpuinfo.c
+++ b/arch/arm64/kernel/cpuinfo.c
@@ -69,6 +69,7 @@ static const char *const hwcap_str[] = {
 	"jscvt",
 	"fcma",
 	"lrcpc",
+	"sve",
 	NULL
 };
 
-- 
2.1.4

[PATCH 25/27] arm64/sve: Add documentation

[Dave Martin]

This patch adds basic documentation of the user/kernel interface
provided by the for SVE.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 Documentation/arm64/sve.txt | 454 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 454 insertions(+)
 create mode 100644 Documentation/arm64/sve.txt

diff --git a/Documentation/arm64/sve.txt b/Documentation/arm64/sve.txt
new file mode 100644
index 0000000..df25abf0
--- /dev/null
+++ b/Documentation/arm64/sve.txt
@@ -0,0 +1,454 @@
+            Scalable Vector Extension support for AArch64 Linux
+            ===================================================
+
+Author: Dave Martin <Dave.Martin@arm.com>
+Date:   4 August 2017
+
+This document outlines briefly the interface provided to userspace by Linux in
+order to support use of the ARM Scalable Vector Extension (SVE).
+
+This is an outline of the most important features and issues only and not
+intended to be exhaustive.
+
+This document does not aim to describe the SVE architecture or programmer's
+model.  To aid understanding, a minimal description of relevant programmer's
+model features for SVE is included in Appendix A.
+
+
+1.  General
+-----------
+
+* SVE registers Z0..Z31, P0..P15 and FFR and the current vector length VL, are
+  tracked per-thread.
+
+* The presence of SVE is reported to userspace via HWCAP_SVE in the aux vector
+  AT_HWCAP entry.  SVE is reported in /proc/cpuinfo as "sve".
+
+
+2.  Vector length terminology
+-----------------------------
+
+The size of an SVE vector (Z) register is referred to as the "vector length".
+
+To avoid confusion about the units used to express vector length, the kernel
+adopts the following conventions:
+
+* Vector length (VL) = size of a Z-register in bytes
+
+* Vector quadwords (VQ) = size of a Z-register in units of 128 bits
+
+(So, VL = 16 * VQ.)
+
+The VQ convention is used where the underlying granularity is important, such
+as in data structure definitions.  In most other situations, the VL convention
+is used.  This is consistent with the meaning of the "VL" pseudo-register in
+the SVE instruction set architecture.
+
+
+3.  System call behaviour
+-------------------------
+
+* On syscall, V0..V31 are preserved (as without SVE).  Thus, bits [127:0] of
+  Z0..Z31 are preserved.  All other bits of Z0..Z31, and all of P0..P15 and FFR
+  become unspecified on return from a syscall.
+
+* The SVE registers are not used to pass arguments to or receive results from
+  any syscall.
+
+* In practice the affected registers/bits will be preserved or will be replaced
+  with zeros on return from a syscall, but userspace should not make
+  assumptions about this.  The kernel behaviour may vary on a case-by-case
+  basis.
+
+
+4.  Signal handling
+-------------------
+
+* A new signal frame record sve_context encodes the SVE registers on signal
+  delivery. [1]
+
+* This record is supplementary to fpsimd_context.  The FPSR and FPCR registers
+  are only present in fpsimd_context.  For convenience, the content of V0..V31
+  is duplicated between sve_context and fpsimd_context.
+
+* The signal frame record for SVE always contains basic metadata, in particular
+  the thread's vector length (in sve_context.vl).
+
+* The SVE registers may or may not be included in the record, depending on
+  whether the registers are live for the thread.  The registers are present if
+  and only if:
+  sve_context.head.size >= SVE_SIG_CONTEXT_SIZE(sve_vq_from_vl(sve_context.vl)).
+
+* If the registers are present, the remainder of the record has a vl-dependent
+  size and layout.  Macros SIG_SVE_* are defined [1] to facilitate access to
+  the members.
+
+* If the SVE context is too big to fit in sigcontext.__reserved[], then extra
+  space is allocated on the stack, an extra_context record is written in
+  __reserved[] referencing this space.  sve_context is then written in the
+  extra space.  Refer to [1] for further details about this mechanism.
+
+
+5.  Signal return
+-----------------
+
+When returning from a signal handler:
+
+* If there is no sve_context record in the signal frame, or if the record is
+  present but contains no register data as desribed in the previous section,
+  then the SVE registers/bits become non-live and take unspecified values.
+
+* If sve_context is present in the signal frame and contains full register
+  data, the SVE registers become live and are populated with the specified
+  data.  However, for backward compatibility reasons, bits [127:0] of Z0..Z31
+  are always restored from the corresponding members of fpsimd_context.vregs[]
+  and not from sve_context.  The remaining bits are restored from sve_context.
+
+* Inclusion of fpsimd_context in the signal frame remains mandatory,
+  irrespective of whether sve_context is present or not.
+
+* The vector length cannot be changed via signal return.  If sve_context.vl in
+  the signal frame does not match the current vector length, the signal return
+  attempt is treated as illegal, resulting in a forced SIGSEGV.
+
+
+6.  prctl extensions
+--------------------
+
+Some new prctl() calls are added to allow programs to manage the SVE vector
+length:
+
+prctl(PR_SVE_SET_VL, unsigned long arg)
+
+    Sets the vector length of the calling thread and related flags, where
+    arg == vl | flags.
+
+    vl is the desired vector length, where sve_vl_valid(vl) must be true.
+
+    flags:
+
+	PR_SVE_SET_VL_INHERIT
+
+	    Inherit the current vector length across execve().  Otherwise, the
+	    vector length is reset to the system default at execve().  (See
+	    Section 9.)
+
+	PR_SVE_SET_VL_ONEXEC
+
+	    Defer the requested vector length change until the next execve().
+	    This allows launching of a new program with a different vector
+	    length, while avoiding runtime side effects in the caller.
+
+	    This also overrides the effect of PR_SVE_SET_VL_INHERIT for the
+	    first execve().
+
+	    Without PR_SVE_SET_VL_ONEXEC, any outstanding deferred vector
+	    length change is cancelled.
+
+    Return value: a nonnegative on success, or a negative value on error:
+	EINVAL: SVE not supported, invalid vector length requested, or
+	    invalid flags.
+
+    On success, the calling thread's vector length is changed to the largest
+    value supported by the system that is less than or equal to vl.
+    If vl == SVE_VL_MAX, the calling thread's vector length is changed to the
+    largest value supported by the system.
+
+    The returned value describes the resulting configuration, encoded as for
+    PR_SVE_GET_VL.
+
+    Changing the vector length causes all of P0..P15, FFR and all bits of
+    Z0..V31 except for Z0 bits [127:0] .. Z31 bits [127:0] to become
+    unspecified.  Calling PR_SVE_SET_VL with vl equal to the thread's current
+    vector length does not constitute a change to the vector length for this
+    purpose.
+
+
+prctl(PR_SVE_GET_VL)
+
+    Gets the vector length of the calling thread.
+
+    The following flag may be OR-ed into the result:
+
+	PR_SVE_SET_VL_INHERIT
+
+	    Vector length will be inherited across execve().
+
+    There is no way to determine whether there is an outstanding deferred
+    vector length change (which would only normally be the case between a
+    fork() or vfork() and the corresponding execve() in typical use).
+
+    To extract the vector length from the result, and it with
+    PR_SVE_VL_LEN_MASK.
+
+    Return value: a nonnegative value on success, or a negative value on error:
+	EINVAL: SVE not supported.
+
+
+7.  ptrace extensions
+---------------------
+
+* A new regset NT_ARM_SVE is defined for use with PTRACE_GETREGSET and
+  PTRACE_SETREGSET.
+
+  Refer to [2] for definitions.
+
+The regset data starts with struct user_sve_header, containing:
+
+    size
+
+	Size of the complete regset, in bytes.
+	This depends on vl and possibly on other things in the future.
+
+	If a call to PTRACE_GETREGSET requests less data than the value of
+	size, the caller can allocate a larger buffer and retry in order to
+	read the complete regset.
+
+    max_size
+
+	Maximum size in bytes that the regset can grow to for the target
+	thread.  The regset won't grow bigger than this even if the target
+	thread changes its vector length etc.
+
+    vl
+
+	Target thread's current vector length, in bytes.
+
+    max_vl
+
+	Maximum possible vector length for the target thread.
+
+    flags
+
+	either
+
+	    SVE_PT_REGS_FPSIMD
+
+		SVE registers are not live (GETREGSET) or are to be made
+		non-live (SETREGSET).
+
+		The payload is of type struct user_fpsimd_state, with the same
+		meaning as for NT_PRFPREG, starting at offset
+		SVE_PT_FPSIMD_OFFSET from the start of user_sve_header.
+
+		Extra data might be appended in the future: the size of the
+		payload should be obtained using SVE_PT_FPSIMD_SIZE(vq, flags).
+
+		vq should be obtained using sve_vq_from_vl(vl).
+
+		or
+
+	    SVE_PT_REGS_SVE
+
+		SVE registers are live (GETREGSET) or are to be made live
+		(SETREGSET).
+
+		The payload contains the SVE register data, starting at offset
+		SVE_PT_SVE_OFFSET from the start of user_sve_header, and with
+		size SVE_PT_SVE_SIZE(vq, flags);
+
+	... OR-ed with zero or more of the following flags, which have the same
+	meaning and behaviour as the corresponding PR_SET_VL_* flags:
+
+	    SVE_PT_VL_INHERIT
+
+	    SVE_PT_VL_ONEXEC (SETREGSET only).
+
+* The effects of changing the vector length and/or flags are equivalent to
+  those documented for PR_SVE_SET_VL.
+
+* In the SVE_PT_REGS_SVE case, the size and layout of the payload depends on
+  the header fields.  The SVE_PT_SVE_*() macros are provided to facilitate
+  access to the members.
+
+* In either case, for SETREGSET it is permissible to omit the payload, in which
+  case only the vector length and flags are changed (along with any
+  consequences of those changes).
+
+  The effect of writing a partial, incomplete payload is unspecified.
+
+
+8.  ELF coredump extensions
+---------------------------
+
+* A NT_ARM_SVE note will be added to each coredump for each thread of the
+  dumped process.  The contents will be equivalent to the data that would have
+  been read if a PTRACE_GETREGSET of NT_ARM_SVE were executed for each thread
+  when the coredump was generated.
+
+  Each of these notes may be padded up to a larger size: if so, the surplus
+  data should be ignored when parsing the coredump.
+
+
+9.  System runtime configuration
+--------------------------------
+
+* To mitigate the ABI impact of expansion of the signal frame, a policy
+  mechanism is provided for administrators, distro maintainers and developers
+  to set the default vector length for userspace processes:
+
+/proc/cpu/sve_default_vector_length
+
+    Writing the text representation of an integer to this file sets the system
+    default vector length to the specified value, unless the value is greater
+    than the maximum vector length supported by the system in which case the
+    default vector length is set to that maximum.
+
+    The result can be determined by reopening the file and reading its
+    contents.
+
+    At boot, the default vector length is initially set to 64 or the maximum
+    supported vector length, whichever is smaller.  This determines the initial
+    vector length of the init process (PID 1).
+
+    Reading this file returns the current system default vector length.
+
+* At every execve() call, the new vector length of the new process is set to
+  the system default vector length, unless
+
+    * PR_SVE_SET_VL_INHERIT (or equivalently SVE_PT_VL_INHERIT) is set for the
+      calling thread, or
+
+    * a deferred vector length change is pending, established via the
+      PR_SVE_SET_VL_ONEXEC flag (or SVE_PT_VL_ONEXEC).
+
+* Modifying the system default vector length does not affect the vector length
+  of any existing process or thread that does not make an execve() call.
+
+
+Appendix A.  SVE programmer's model (informative)
+=================================================
+
+This section provides a minimal description of the additions made by SVE to the
+ARMv8-A programmer's model that are relevant to this document.
+
+Note: This section is for information only and not intended to be complete or
+to replace any architectural specification.
+
+A.1.  Registers
+---------------
+
+In A64 state, SVE adds the following:
+
+* 32 8VL-bit vector registers Z0..Z31
+  For each Zn, Zn bits [127:0] alias the ARMv8-A vector register Vn.
+
+  A register write using a Vn register name zeros all bits of the corresponding
+  Zn except for bits [127:0].
+
+* 16 VL-bit predicate registers P0..P15
+
+* 1 VL-bit special-purpose predicate register FFR (the "first-fault register")
+
+* a VL "pseudo-register" that determines the size of each vector register
+
+  The SVE instruction set architecture provides no way to write VL directly.
+  Instead, it can be modified only by EL1 and above, by writing appropriate
+  system registers.
+
+* The value of VL can be configured at runtime by EL1 and above:
+  16 <= VL <= VLmax, where VL must be a multiple of 16.
+
+* The maximum vector length is determined by the hardware:
+  16 <= VLmax <= 256.
+
+  (The SVE architecture specifies 256, but permits future architecture
+  revisions to raise this limit.)
+
+* FPSR and FPCR are retained from ARMv8-A, and interact with SVE floating-point
+  operations in a similar way to the way in which they interact with ARMv8
+  floating-point operations.
+
+         8VL-1                       128               0  bit index
+        +----          ////            -----------------+
+     Z0 |                               :       V0      |
+      :                                          :
+     Z7 |                               :       V7      |
+     Z8 |                               :     * V8      |
+      :                                       :  :
+    Z15 |                               :     *V15      |
+    Z16 |                               :      V16      |
+      :                                          :
+    Z31 |                               :      V31      |
+        +----          ////            -----------------+
+                                                 31    0
+         VL-1                  0                +-------+
+        +----       ////      --+          FPSR |       |
+     P0 |                       |               +-------+
+      : |                       |         *FPCR |       |
+    P15 |                       |               +-------+
+        +----       ////      --+
+    FFR |                       |               +-----+
+        +----       ////      --+            VL |     |
+                                                +-----+
+
+(*) callee-save:
+    This only applies to bits [63:0] of Z-/V-registers.
+    FPCR contains callee-save and caller-save bits.  See [3] for details.
+
+
+A.2.  Procedure call standard
+-----------------------------
+
+The ARMv8-A base procedure call standard is extended as follows with respect to
+the additional SVE register state:
+
+* All SVE register bits that are not shared with FP/SIMD are caller-save.
+
+* Z8 bits [63:0] .. Z15 bits [63:0] are callee-save.
+
+  This follows from the way these bits are mapped to V8..V15, which are caller-
+  save in the base procedure call standard.
+
+
+Appendix B.  ARMv8-A FP/SIMD programmer's model
+===============================================
+
+Note: This section is for information only and not intended to be complete or
+to replace any architectural specification.
+
+Refer to [3] for for more information.
+
+ARMv8-A defines the following floating-point / SIMD register state:
+
+* 32 128-bit vector registers V0..V31
+* 2 32-bit status/control registers FPSR, FPCR
+
+         127           0  bit index
+        +---------------+
+     V0 |               |
+      : :               :
+     V7 |               |
+   * V8 |               |
+   :  : :               :
+   *V15 |               |
+    V16 |               |
+      : :               :
+    V31 |               |
+        +---------------+
+
+                 31    0
+                +-------+
+           FPSR |       |
+                +-------+
+          *FPCR |       |
+                +-------+
+
+(*) callee-save:
+    This only applies to bits [63:0] of V-registers.
+    FPCR contains a mixture of callee-save and caller-save bits.
+
+
+References
+==========
+
+[1] arch/arm64/include/uapi/asm/sigcontext.h
+    AArch64 Linux signal ABI definitions
+
+[2] arch/arm64/include/uapi/asm/ptrace.h
+    AArch64 Linux ptrace ABI definitions
+
+[3] ARM IHI0055C
+    http://infocenter.arm.com/help/topic/com.arm.doc.ihi0055c/IHI0055C_beta_aapcs64.pdf
+    http://infocenter.arm.com/help/topic/com.arm.doc.subset.swdev.abi/index.html
+    Procedure Call Standard for the ARM 64-bit Architecture (AArch64)
-- 
2.1.4

[RFC PATCH 26/27] arm64: signal: Report signal frame size to userspace via auxv

[Dave Martin]

Stateful CPU architecture extensions may require the signal frame
to grow to a size that exceeds the arch's MINSIGSTKSZ #define.
However, changing this #define is an ABI break.

To allow userspace the option of determining the signal frame size
in a more forwards-compatible way, this patch adds a new auxv entry
tagged with AT_MINSIGSTKSZ, which provides the maximum signal frame
size that the process can observe during its lifetime.

If AT_MINSIGSTKSZ is absent from the aux vector, the caller can
assume that the MINSIGSTKSZ #define is sufficient.  This allows for
a consistent interface with older kernels that do not provide
AT_MINSIGSTKSZ.

The idea is that libc could expose this via sysconf() or some
similar mechanism.

There is deliberately no AT_SIGSTKSZ.  The kernel knows nothing
about userspace's own stack overheads and should not pretend to
know.

For arm64:

The primary motivation for this interface is the Scalable Vector
Extension, which can require at least 4KB or so of extra space
in the signal frame for the largest hardware implementations.

To determine the correct value, a "Christmas tree" mode (via the
add_all argument) is added to setup_sigframe_layout(), to simulate
addition of all possible records to the signal frame at maximum
possible size.

If this procedure goes wrong somehow, resulting in a stupidly large
frame layout and hence failure of sigframe_alloc() to allocate a
record to the frame, then this is indicative of a kernel bug: the
kernel's internal SIGFRAME_MAXSZ is supposed to sanity-check
against generting frames that we consider _impossibly_ large.  In
this case, SIGSTKSZ is returned as a "reasonable guess that is at
least bigger than MINSIGSTKSZ" and we WARN().

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/include/asm/elf.h         |  5 +++++
 arch/arm64/include/asm/processor.h   |  3 +++
 arch/arm64/include/uapi/asm/auxvec.h |  3 ++-
 arch/arm64/kernel/signal.c           | 38 +++++++++++++++++++++++++++++++-----
 4 files changed, 43 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/include/asm/elf.h b/arch/arm64/include/asm/elf.h
index acae781..7cee254 100644
--- a/arch/arm64/include/asm/elf.h
+++ b/arch/arm64/include/asm/elf.h
@@ -24,6 +24,10 @@
 #include <asm/ptrace.h>
 #include <asm/user.h>
 
+#ifndef __ASSEMBLY__
+#include <asm/processor.h> /* for get_minsigstksz(), used by ARCH_DLINFO */
+#endif
+
 /*
  * AArch64 static relocation types.
  */
@@ -149,6 +153,7 @@ typedef struct user_fpsimd_state elf_fpregset_t;
 do {									\
 	NEW_AUX_ENT(AT_SYSINFO_EHDR,					\
 		    (elf_addr_t)current->mm->context.vdso);		\
+	NEW_AUX_ENT(AT_MINSIGSTKSZ, get_minsigstksz());			\
 } while (0)
 
 #define ARCH_HAS_SETUP_ADDITIONAL_PAGES
diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
index 70b03b4..121d011 100644
--- a/arch/arm64/include/asm/processor.h
+++ b/arch/arm64/include/asm/processor.h
@@ -197,6 +197,9 @@ static inline void spin_lock_prefetch(const void *ptr)
 int cpu_enable_pan(void *__unused);
 int cpu_enable_cache_maint_trap(void *__unused);
 
+/* User signal frame size discovery: */
+int get_minsigstksz(void);
+
 /* Userspace interface for PR_SVE_{SET,GET}_VL prctl()s: */
 #define SVE_SET_VL(arg)	sve_set_current_vl(arg)
 #define SVE_GET_VL()	sve_get_current_vl()
diff --git a/arch/arm64/include/uapi/asm/auxvec.h b/arch/arm64/include/uapi/asm/auxvec.h
index 4cf0c17..1d45b28 100644
--- a/arch/arm64/include/uapi/asm/auxvec.h
+++ b/arch/arm64/include/uapi/asm/auxvec.h
@@ -18,7 +18,8 @@
 
 /* vDSO location */
 #define AT_SYSINFO_EHDR	33
+#define AT_MINSIGSTKSZ	34	/* stack needed for signal delivery */
 
-#define AT_VECTOR_SIZE_ARCH 1 /* entries in ARCH_DLINFO */
+#define AT_VECTOR_SIZE_ARCH 2 /* entries in ARCH_DLINFO */
 
 #endif
diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c
index 2694143..221bcf2 100644
--- a/arch/arm64/kernel/signal.c
+++ b/arch/arm64/kernel/signal.c
@@ -560,8 +560,15 @@ asmlinkage long sys_rt_sigreturn(struct pt_regs *regs)
 	return 0;
 }
 
-/* Determine the layout of optional records in the signal frame */
-static int setup_sigframe_layout(struct rt_sigframe_user_layout *user)
+/*
+ * Determine the layout of optional records in the signal frame
+ *
+ * add_all: if true, lays out the biggest possible signal frame for
+ *	this task; otherwise, generates a layout for the current state
+ *	of the task.
+ */
+static int setup_sigframe_layout(struct rt_sigframe_user_layout *user,
+				 bool add_all)
 {
 	int err;
 
@@ -571,7 +578,7 @@ static int setup_sigframe_layout(struct rt_sigframe_user_layout *user)
 		return err;
 
 	/* fault information, if valid */
-	if (current->thread.fault_code) {
+	if (add_all || current->thread.fault_code) {
 		err = sigframe_alloc(user, &user->esr_offset,
 				     sizeof(struct esr_context));
 		if (err)
@@ -595,7 +602,6 @@ static int setup_sigframe_layout(struct rt_sigframe_user_layout *user)
 	return sigframe_alloc_end(user);
 }
 
-
 static int setup_sigframe(struct rt_sigframe_user_layout *user,
 			  struct pt_regs *regs, sigset_t *set)
 {
@@ -693,7 +699,7 @@ static int get_sigframe(struct rt_sigframe_user_layout *user,
 	int err;
 
 	init_user_layout(user);
-	err = setup_sigframe_layout(user);
+	err = setup_sigframe_layout(user, false);
 	if (err)
 		return err;
 
@@ -927,3 +933,25 @@ asmlinkage void do_notify_resume(struct pt_regs *regs,
 		thread_flags = READ_ONCE(current_thread_info()->flags);
 	} while (thread_flags & _TIF_WORK_MASK);
 }
+
+/*
+ * Determine the stack space required for guaranteed signal devliery.
+ * This function is used to populate AT_MINSIGSTKSZ at process startup.
+ */
+int get_minsigstksz(void)
+{
+	struct rt_sigframe_user_layout user;
+	int err;
+
+	init_user_layout(&user);
+	err = setup_sigframe_layout(&user, true);
+
+	if (err) {
+		WARN_ON(1);
+
+		return SIGSTKSZ;
+	} else
+		return sigframe_size(&user) +
+			round_up(sizeof(struct frame_record), 16) +
+			16; /* max alignment padding */
+}
-- 
2.1.4

[RFC PATCH 27/27] arm64/sve: signal: Include SVE when computing AT_MINSIGSTKSZ

[Dave Martin]

The SVE context block in the signal frame needs to be considered
too when computing the maximum possible signal frame size.

Because the size of this block depends on the vector length, this
patch computes the size based not on the thread's current vector
length but instead on the maximum possible vector length: this
determines the maximum size of SVE context block that can be
observed in any signal frame for the lifetime of the process.

Signed-off-by: Dave Martin <Dave.Martin@arm.com>
---
 arch/arm64/kernel/signal.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c
index 221bcf2..cb01353 100644
--- a/arch/arm64/kernel/signal.c
+++ b/arch/arm64/kernel/signal.c
@@ -586,11 +586,18 @@ static int setup_sigframe_layout(struct rt_sigframe_user_layout *user,
 	}
 
 	if (system_supports_sve()) {
-		unsigned int vq = 0;
+		unsigned int vq;
 
-		if (test_thread_flag(TIF_SVE)) {
-			BUG_ON(!sve_vl_valid(current->thread.sve_vl));
-			vq = sve_vq_from_vl(current->thread.sve_vl);
+		if (!add_all && !test_thread_flag(TIF_SVE))
+			vq = 0;
+		else {
+			unsigned int vl = sve_max_vl;
+
+			if (!add_all)
+				vl = current->thread.sve_vl;
+
+			BUG_ON(!sve_vl_valid(vl));
+			vq = sve_vq_from_vl(vl);
 		}
 
 		err = sigframe_alloc(user, &user->sve_offset,
-- 
2.1.4

Re: [PATCH 21/27] arm64/sve: KVM: Prevent guests from using SVE

[Marc Zyngier]

On 09/08/17 13:05, Dave Martin wrote:
> Until KVM has full SVE support, guests must not be allowed to
> execute SVE instructions.
> 
> This patch enables the necessary traps, and also ensures that the
> traps are disabled again on exit from the guest so that the host
> can still use SVE if it wants to.
> 
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/include/asm/kvm_arm.h | 3 ++-
>  arch/arm64/kvm/hyp/switch.c      | 6 +++---
>  2 files changed, 5 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
> index dbf0537..8a19651 100644
> --- a/arch/arm64/include/asm/kvm_arm.h
> +++ b/arch/arm64/include/asm/kvm_arm.h
> @@ -186,7 +186,7 @@
>  #define CPTR_EL2_TTA	(1 << 20)
>  #define CPTR_EL2_TFP	(1 << CPTR_EL2_TFP_SHIFT)
>  #define CPTR_EL2_TZ	(1 << 8)
> -#define CPTR_EL2_DEFAULT	0x000033ff
> +#define CPTR_EL2_DEFAULT	(0x000033ff & ~CPTR_EL2_TZ)

I must say I'm not overly fond of this construct. I'd rather introduce a
RES1 field that matches the v8.2 description, instead of this ugly
constant and something that clears it.

>  
>  /* Hyp Debug Configuration Register bits */
>  #define MDCR_EL2_TPMS		(1 << 14)
> @@ -237,5 +237,6 @@
>  
>  #define CPACR_EL1_FPEN		(3 << 20)
>  #define CPACR_EL1_TTA		(1 << 28)
> +#define CPACR_EL1_DEFAULT	(CPACR_EL1_FPEN | CPACR_EL1_ZEN_EL1EN)
>  
>  #endif /* __ARM64_KVM_ARM_H__ */
> diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> index 35a90b8..951f3eb 100644
> --- a/arch/arm64/kvm/hyp/switch.c
> +++ b/arch/arm64/kvm/hyp/switch.c
> @@ -48,7 +48,7 @@ static void __hyp_text __activate_traps_vhe(void)
>  
>  	val = read_sysreg(cpacr_el1);
>  	val |= CPACR_EL1_TTA;
> -	val &= ~CPACR_EL1_FPEN;
> +	val &= ~(CPACR_EL1_FPEN | CPACR_EL1_ZEN);
>  	write_sysreg(val, cpacr_el1);
>  
>  	write_sysreg(__kvm_hyp_vector, vbar_el1);
> @@ -59,7 +59,7 @@ static void __hyp_text __activate_traps_nvhe(void)
>  	u64 val;
>  
>  	val = CPTR_EL2_DEFAULT;
> -	val |= CPTR_EL2_TTA | CPTR_EL2_TFP;
> +	val |= CPTR_EL2_TTA | CPTR_EL2_TFP | CPTR_EL2_TZ;
>  	write_sysreg(val, cptr_el2);
>  }
>  
> @@ -117,7 +117,7 @@ static void __hyp_text __deactivate_traps_vhe(void)
>  
>  	write_sysreg(mdcr_el2, mdcr_el2);
>  	write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);
> -	write_sysreg(CPACR_EL1_FPEN, cpacr_el1);
> +	write_sysreg(CPACR_EL1_DEFAULT, cpacr_el1);
>  	write_sysreg(vectors, vbar_el1);
>  }
>  
> 

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 21/27] arm64/sve: KVM: Prevent guests from using SVE

[Marc Zyngier]

On 09/08/17 13:05, Dave Martin wrote:
> Until KVM has full SVE support, guests must not be allowed to
> execute SVE instructions.
> 
> This patch enables the necessary traps, and also ensures that the
> traps are disabled again on exit from the guest so that the host
> can still use SVE if it wants to.
> 
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/include/asm/kvm_arm.h | 3 ++-
>  arch/arm64/kvm/hyp/switch.c      | 6 +++---
>  2 files changed, 5 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
> index dbf0537..8a19651 100644
> --- a/arch/arm64/include/asm/kvm_arm.h
> +++ b/arch/arm64/include/asm/kvm_arm.h
> @@ -186,7 +186,7 @@
>  #define CPTR_EL2_TTA	(1 << 20)
>  #define CPTR_EL2_TFP	(1 << CPTR_EL2_TFP_SHIFT)
>  #define CPTR_EL2_TZ	(1 << 8)
> -#define CPTR_EL2_DEFAULT	0x000033ff
> +#define CPTR_EL2_DEFAULT	(0x000033ff & ~CPTR_EL2_TZ)

I must say I'm not overly fond of this construct. I'd rather introduce a
RES1 field that matches the v8.2 description, instead of this ugly
constant and something that clears it.

>  
>  /* Hyp Debug Configuration Register bits */
>  #define MDCR_EL2_TPMS		(1 << 14)
> @@ -237,5 +237,6 @@
>  
>  #define CPACR_EL1_FPEN		(3 << 20)
>  #define CPACR_EL1_TTA		(1 << 28)
> +#define CPACR_EL1_DEFAULT	(CPACR_EL1_FPEN | CPACR_EL1_ZEN_EL1EN)
>  
>  #endif /* __ARM64_KVM_ARM_H__ */
> diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> index 35a90b8..951f3eb 100644
> --- a/arch/arm64/kvm/hyp/switch.c
> +++ b/arch/arm64/kvm/hyp/switch.c
> @@ -48,7 +48,7 @@ static void __hyp_text __activate_traps_vhe(void)
>  
>  	val = read_sysreg(cpacr_el1);
>  	val |= CPACR_EL1_TTA;
> -	val &= ~CPACR_EL1_FPEN;
> +	val &= ~(CPACR_EL1_FPEN | CPACR_EL1_ZEN);
>  	write_sysreg(val, cpacr_el1);
>  
>  	write_sysreg(__kvm_hyp_vector, vbar_el1);
> @@ -59,7 +59,7 @@ static void __hyp_text __activate_traps_nvhe(void)
>  	u64 val;
>  
>  	val = CPTR_EL2_DEFAULT;
> -	val |= CPTR_EL2_TTA | CPTR_EL2_TFP;
> +	val |= CPTR_EL2_TTA | CPTR_EL2_TFP | CPTR_EL2_TZ;
>  	write_sysreg(val, cptr_el2);
>  }
>  
> @@ -117,7 +117,7 @@ static void __hyp_text __deactivate_traps_vhe(void)
>  
>  	write_sysreg(mdcr_el2, mdcr_el2);
>  	write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);
> -	write_sysreg(CPACR_EL1_FPEN, cpacr_el1);
> +	write_sysreg(CPACR_EL1_DEFAULT, cpacr_el1);
>  	write_sysreg(vectors, vbar_el1);
>  }
>  
> 

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 23/27] arm64/sve: KVM: Hide SVE from CPU features exposed to guests

[Marc Zyngier]

On 09/08/17 13:05, Dave Martin wrote:
> KVM guests cannot currently use SVE, because SVE is always
> configured to trap to EL2.
> 
> However, a guest that sees SVE reported as present in
> ID_AA64PFR0_EL1 may legitimately expect that SVE works and try to
> use it.  Instead of working, the guest will receive an injected
> undef exception, which may cause the guest to oops or go into a
> spin.
> 
> To avoid misleading the guest into believing that SVE will work,
> this patch masks out the SVE field from ID_AA64PFR0_EL1 when a
> guest attempts to read this register.  No support is explicitly
> added for ID_AA64ZFR0_EL1 either, so that is still emulated as
> reading as zero, which is consistent with SVE not being
> implemented.
> 
> This is a temporary measure, and will be removed in a later series
> when full KVM support for SVE is implemented.
> 
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/kvm/sys_regs.c | 14 +++++++++++++-
>  1 file changed, 13 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
> index 6583dd7..9e8c54e 100644
> --- a/arch/arm64/kvm/sys_regs.c
> +++ b/arch/arm64/kvm/sys_regs.c
> @@ -897,8 +897,20 @@ static u64 read_id_reg(struct sys_reg_desc const *r, bool raz)
>  {
>  	u32 id = sys_reg((u32)r->Op0, (u32)r->Op1,
>  			 (u32)r->CRn, (u32)r->CRm, (u32)r->Op2);
> +	u64 val = raz ? 0 : read_sanitised_ftr_reg(id);
>  
> -	return raz ? 0 : read_sanitised_ftr_reg(id);
> +	if (id == SYS_ID_AA64PFR0_EL1) {
> +		static bool printed;
> +
> +		if ((val & (0xfUL << ID_AA64PFR0_SVE_SHIFT)) && !printed) {
> +			kvm_info("SVE unsupported for guests, suppressing\n");
> +			printed = true;
> +		}

Ideally, this should be a vcpu_unimpl_once(). But:
- it doesn't exist
- vcpu_unimpl looks hopelessly x86 specific

How about turning it into a pr_err_once() instead?

> +
> +		val &= ~(0xfUL << ID_AA64PFR0_SVE_SHIFT);
> +	}
> +
> +	return val;
>  }
>  
>  /* cpufeature ID register access trap handlers */
> 

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 05/27] arm64: fpsimd: Simplify uses of {set,clear}_ti_thread_flag()

[Ard Biesheuvel]

On 9 August 2017 at 13:05, Dave Martin <Dave.Martin@arm.com> wrote:
> The existing FPSIMD context switch code contains a couple of
> instances of {set,clear}_ti_thread(task_thread_info(task)).  Since
> there are thread flag manipulators that operate directly on
> task_struct, this verbosity isn't strictly needed.
>
> For consistency, this patch simplifies the affected calls.  This
> should have no impact on behaviour.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>

Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>

> ---
>  arch/arm64/kernel/fpsimd.c | 6 ++----
>  1 file changed, 2 insertions(+), 4 deletions(-)
>
> diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> index 138fcfa..9c1f268e 100644
> --- a/arch/arm64/kernel/fpsimd.c
> +++ b/arch/arm64/kernel/fpsimd.c
> @@ -159,11 +159,9 @@ void fpsimd_thread_switch(struct task_struct *next)
>
>                 if (__this_cpu_read(fpsimd_last_state) == st
>                     && st->cpu == smp_processor_id())
> -                       clear_ti_thread_flag(task_thread_info(next),
> -                                            TIF_FOREIGN_FPSTATE);
> +                       clear_tsk_thread_flag(next, TIF_FOREIGN_FPSTATE);
>                 else
> -                       set_ti_thread_flag(task_thread_info(next),
> -                                          TIF_FOREIGN_FPSTATE);
> +                       set_tsk_thread_flag(next, TIF_FOREIGN_FPSTATE);
>         }
>  }
>
> --
> 2.1.4
>

Re: [PATCH 11/27] arm64/sve: Core task context handling

[Ard Biesheuvel]

Hi Dave,

On 9 August 2017 at 13:05, Dave Martin <Dave.Martin@arm.com> wrote:
> This patch adds the core support for switching and managing the SVE
> architectural state of user tasks.
>
> Calls to the existing FPSIMD low-level save/restore functions are
> factored out as new functions task_fpsimd_{save,load}(), since SVE
> now dynamically may or may not need to be handled at these points
> depending on the kernel configuration, hardware features discovered
> at boot, and the runtime state of the task.  To make these
> decisions as fast as possible, const cpucaps are used where
> feasible, via the system_supports_sve() helper.
>
> The SVE registers are only tracked for threads that have explicitly
> used SVE, indicated by the new thread flag TIF_SVE.  Otherwise, the
> FPSIMD view of the architectural state is stored in
> thread.fpsimd_state as usual.
>
> When in use, the SVE registers are not stored directly in
> thread_struct due to their potentially large and variable size.
> Because the task_struct slab allocator must be configured very
> early during kernel boot, it is also tricky to configure it
> correctly to match the maximum vector length provided by the
> hardware, since this depends on examining secondary CPUs as well as
> the primary.  Instead, a pointer sve_state in thread_struct points
> to a dynamically allocated buffer containing the SVE register data,
> and code is added to allocate, duplicate and free this buffer at
> appropriate times.
>
> TIF_SVE is set when taking an SVE access trap from userspace, if
> suitable hardware support has been detected.  This enables SVE for
> the thread: a subsequent return to userspace will disable the trap
> accordingly.  If such a trap is taken without sufficient hardware
> support, SIGILL is sent to the thread instead as if an undefined
> instruction had been executed: this may happen if userspace tries
> to use SVE in a system where not all CPUs support it for example.
>
> The kernel may clear TIF_SVE and disable SVE for the thread
> whenever an explicit syscall is made by userspace, though this is
> considered an optimisation opportunity rather than a deterministic
> guarantee: the kernel may not do this on every syscall, but it is
> permitted to do so.  For backwards compatibility reasons and
> conformance with the spirit of the base AArch64 procedure call
> standard, the subset of the SVE register state that aliases the
> FPSIMD registers is still preserved across a syscall even if this
> happens.
>
> TIF_SVE is also cleared, and SVE disabled, on exec: this is an
> obvious slow path and a hint that we are running a new binary that
> may not use SVE.
>
> Code is added to sync data between thread.fpsimd_state and
> thread.sve_state whenever enabling/disabling SVE, in a manner
> consistent with the SVE architectural programmer's model.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/include/asm/fpsimd.h      |  19 +++
>  arch/arm64/include/asm/processor.h   |   2 +
>  arch/arm64/include/asm/thread_info.h |   1 +
>  arch/arm64/include/asm/traps.h       |   2 +
>  arch/arm64/kernel/entry.S            |  14 +-
>  arch/arm64/kernel/fpsimd.c           | 241 ++++++++++++++++++++++++++++++++++-
>  arch/arm64/kernel/process.c          |   6 +-
>  arch/arm64/kernel/traps.c            |   4 +-
>  8 files changed, 279 insertions(+), 10 deletions(-)
>
> diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
> index 026a7c7..72090a1 100644
> --- a/arch/arm64/include/asm/fpsimd.h
> +++ b/arch/arm64/include/asm/fpsimd.h
> @@ -20,6 +20,8 @@
>
>  #ifndef __ASSEMBLY__
>
> +#include <linux/stddef.h>
> +
>  /*
>   * FP/SIMD storage area has:
>   *  - FPSR and FPCR
> @@ -72,6 +74,23 @@ extern void sve_load_state(void const *state, u32 const *pfpsr,
>                            unsigned long vq_minus_1);
>  extern unsigned int sve_get_vl(void);
>
> +#ifdef CONFIG_ARM64_SVE
> +
> +extern size_t sve_state_size(struct task_struct const *task);
> +
> +extern void sve_alloc(struct task_struct *task);
> +extern void fpsimd_release_thread(struct task_struct *task);
> +extern void fpsimd_dup_sve(struct task_struct *dst,
> +                          struct task_struct const *src);
> +
> +#else /* ! CONFIG_ARM64_SVE */
> +
> +static void __maybe_unused sve_alloc(struct task_struct *task) { }
> +static void __maybe_unused fpsimd_release_thread(struct task_struct *task) { }
> +static void __maybe_unused fpsimd_dup_sve(struct task_struct *dst,
> +                                         struct task_struct const *src) { }
> +#endif /* ! CONFIG_ARM64_SVE */
> +
>  /* For use by EFI runtime services calls only */
>  extern void __efi_fpsimd_begin(void);
>  extern void __efi_fpsimd_end(void);
> diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
> index b7334f1..969feed 100644
> --- a/arch/arm64/include/asm/processor.h
> +++ b/arch/arm64/include/asm/processor.h
> @@ -85,6 +85,8 @@ struct thread_struct {
>         unsigned long           tp2_value;
>  #endif
>         struct fpsimd_state     fpsimd_state;
> +       void                    *sve_state;     /* SVE registers, if any */
> +       u16                     sve_vl;         /* SVE vector length */
>         unsigned long           fault_address;  /* fault info */
>         unsigned long           fault_code;     /* ESR_EL1 value */
>         struct debug_info       debug;          /* debugging */
> diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h
> index 46c3b93..1a4b30b 100644
> --- a/arch/arm64/include/asm/thread_info.h
> +++ b/arch/arm64/include/asm/thread_info.h
> @@ -96,6 +96,7 @@ struct thread_info {
>  #define TIF_RESTORE_SIGMASK    20
>  #define TIF_SINGLESTEP         21
>  #define TIF_32BIT              22      /* 32bit process */
> +#define TIF_SVE                        23      /* Scalable Vector Extension in use */
>
>  #define _TIF_SIGPENDING                (1 << TIF_SIGPENDING)
>  #define _TIF_NEED_RESCHED      (1 << TIF_NEED_RESCHED)
> diff --git a/arch/arm64/include/asm/traps.h b/arch/arm64/include/asm/traps.h
> index 02e9035..f058c07 100644
> --- a/arch/arm64/include/asm/traps.h
> +++ b/arch/arm64/include/asm/traps.h
> @@ -34,6 +34,8 @@ struct undef_hook {
>
>  void register_undef_hook(struct undef_hook *hook);
>  void unregister_undef_hook(struct undef_hook *hook);
> +void force_signal_inject(int signal, int code, struct pt_regs *regs,
> +                        unsigned long address);
>
>  void arm64_notify_segfault(struct pt_regs *regs, unsigned long addr);
>
> diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
> index cace76d..c33069c 100644
> --- a/arch/arm64/kernel/entry.S
> +++ b/arch/arm64/kernel/entry.S
> @@ -524,6 +524,8 @@ el0_sync:
>         b.eq    el0_ia
>         cmp     x24, #ESR_ELx_EC_FP_ASIMD       // FP/ASIMD access
>         b.eq    el0_fpsimd_acc
> +       cmp     x24, #ESR_ELx_EC_SVE            // SVE access
> +       b.eq    el0_sve_acc
>         cmp     x24, #ESR_ELx_EC_FP_EXC64       // FP/ASIMD exception
>         b.eq    el0_fpsimd_exc
>         cmp     x24, #ESR_ELx_EC_SYS64          // configurable trap
> @@ -622,9 +624,19 @@ el0_fpsimd_acc:
>         mov     x1, sp
>         bl      do_fpsimd_acc
>         b       ret_to_user
> +el0_sve_acc:
> +       /*
> +        * Scalable Vector Extension access
> +        */
> +       enable_dbg
> +       ct_user_exit
> +       mov     x0, x25
> +       mov     x1, sp
> +       bl      do_sve_acc
> +       b       ret_to_user
>  el0_fpsimd_exc:
>         /*
> -        * Floating Point or Advanced SIMD exception
> +        * Floating Point, Advanced SIMD or SVE exception
>          */
>         enable_dbg
>         ct_user_exit
> diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> index 9c1f268e..37dd1b2 100644
> --- a/arch/arm64/kernel/fpsimd.c
> +++ b/arch/arm64/kernel/fpsimd.c
> @@ -24,12 +24,17 @@
>  #include <linux/init.h>
>  #include <linux/percpu.h>
>  #include <linux/preempt.h>
> +#include <linux/ptrace.h>
>  #include <linux/sched/signal.h>
>  #include <linux/signal.h>
> +#include <linux/slab.h>
>
>  #include <asm/fpsimd.h>
>  #include <asm/cputype.h>
>  #include <asm/simd.h>
> +#include <asm/sigcontext.h>
> +#include <asm/sysreg.h>
> +#include <asm/traps.h>
>
>  #define FPEXC_IOF      (1 << 0)
>  #define FPEXC_DZF      (1 << 1)
> @@ -38,6 +43,10 @@
>  #define FPEXC_IXF      (1 << 4)
>  #define FPEXC_IDF      (1 << 7)
>
> +/* Forward declarations for local functions used by both SVE and FPSIMD */
> +static void task_fpsimd_load(void);
> +static void task_fpsimd_save(void);
> +

We usually try to avoid forward declarations for functions with static
linkage. Is it possible to reorder them and get rid of this?

>  /*
>   * In order to reduce the number of times the FPSIMD state is needlessly saved
>   * and restored, we need to keep track of two things:
> @@ -99,6 +108,160 @@
>   */
>  static DEFINE_PER_CPU(struct fpsimd_state *, fpsimd_last_state);
>
> +static void sve_free(struct task_struct *task)
> +{
> +       kfree(task->thread.sve_state);
> +       task->thread.sve_state = NULL;
> +}
> +
> +/* Offset of FFR in the SVE register dump */
> +static size_t sve_ffr_offset(int vl)
> +{
> +       BUG_ON(!sve_vl_valid(vl));

BUG_ON() is a pretty heavy hammer, so we should not use it unless the
kernel state is so corrupted that there is no way to carry on. I have
a feeling this may not be the case for some of the occurrences in this
patch.

> +       return SVE_SIG_FFR_OFFSET(sve_vq_from_vl(vl)) - SVE_SIG_REGS_OFFSET;
> +}
> +
> +static void *sve_pffr(struct task_struct *task)
> +{
> +       unsigned int vl = task->thread.sve_vl;
> +
> +       BUG_ON(!sve_vl_valid(vl) || !task->thread.sve_state);
> +       return (char *)task->thread.sve_state + sve_ffr_offset(vl);
> +}
> +
> +static u64 sve_cpacr_trap_on(u64 cpacr)
> +{
> +       return cpacr & ~(u64)CPACR_EL1_ZEN_EL0EN;
> +}
> +
> +static u64 sve_cpacr_trap_off(u64 cpacr)
> +{
> +       return cpacr | CPACR_EL1_ZEN_EL0EN;
> +}
> +
> +static void change_cpacr(u64 old, u64 new)
> +{
> +       if (old != new)
> +               write_sysreg(new, CPACR_EL1);
> +}
> +
> +#ifdef CONFIG_ARM64_SVE
> +
> +#define ZREG(sve_state, vq, n) ((char *)(sve_state) +          \
> +       (SVE_SIG_ZREG_OFFSET(vq, n) - SVE_SIG_REGS_OFFSET))
> +
> +static void sve_to_fpsimd(struct task_struct *task)
> +{
> +       unsigned int vl = task->thread.sve_vl;
> +       unsigned int vq;
> +       void const *sst = task->thread.sve_state;
> +       struct fpsimd_state *fst = &task->thread.fpsimd_state;
> +       unsigned int i;
> +
> +       if (!system_supports_sve())
> +               return;
> +
> +       BUG_ON(!sve_vl_valid(vl));
> +       vq = sve_vq_from_vl(vl);
> +
> +       for (i = 0; i < 32; ++i)
> +               memcpy(&fst->vregs[i], ZREG(sst, vq, i),
> +                      sizeof(fst->vregs[i]));
> +}
> +
> +static void fpsimd_to_sve(struct task_struct *task)
> +{
> +       unsigned int vl = task->thread.sve_vl;
> +       unsigned int vq;
> +       void *sst = task->thread.sve_state;
> +       struct fpsimd_state const *fst = &task->thread.fpsimd_state;
> +       unsigned int i;
> +
> +       if (!system_supports_sve())
> +               return;
> +
> +       BUG_ON(!sve_vl_valid(vl));
> +       vq = sve_vq_from_vl(vl);
> +
> +       for (i = 0; i < 32; ++i)
> +               memcpy(ZREG(sst, vq, i), &fst->vregs[i],
> +                      sizeof(fst->vregs[i]));
> +}
> +
> +size_t sve_state_size(struct task_struct const *task)
> +{
> +       unsigned int vl = task->thread.sve_vl;
> +
> +       BUG_ON(!sve_vl_valid(vl));
> +       return SVE_SIG_REGS_SIZE(sve_vq_from_vl(vl));
> +}
> +
> +void sve_alloc(struct task_struct *task)
> +{
> +       if (task->thread.sve_state) {
> +               memset(task->thread.sve_state, 0, sve_state_size(current));
> +               return;
> +       }
> +
> +       /* This is a small allocation (maximum ~8KB) and Should Not Fail. */
> +       task->thread.sve_state =
> +               kzalloc(sve_state_size(task), GFP_KERNEL);
> +
> +       /*
> +        * If future SVE revisions can have larger vectors though,
> +        * this may cease to be true:
> +        */
> +       BUG_ON(!task->thread.sve_state);
> +}
> +
> +/*
> + * Handle SVE state across fork():
> + *
> + * dst and src must not end up with aliases of the same sve_state.
> + * Because a task cannot fork except in a syscall, we can discard SVE
> + * state for dst here: reallocation will be deferred until dst tries
> + * to use SVE.
> + */
> +void fpsimd_dup_sve(struct task_struct *dst, struct task_struct const *src)
> +{
> +       BUG_ON(!in_syscall(task_pt_regs(dst)));
> +
> +       if (test_and_clear_tsk_thread_flag(dst, TIF_SVE)) {
> +               sve_to_fpsimd(dst);
> +               dst->thread.sve_state = NULL;
> +       }
> +}
> +
> +void fpsimd_release_thread(struct task_struct *dead_task)
> +{
> +       sve_free(dead_task);
> +}
> +
> +#endif /* CONFIG_ARM64_SVE */
> +
> +/*
> + * Trapped SVE access
> + */
> +void do_sve_acc(unsigned int esr, struct pt_regs *regs)
> +{
> +       BUG_ON(is_compat_task());
> +
> +       /* Even if we chose not to use SVE, the hardware could still trap: */
> +       if (unlikely(!system_supports_sve())) {
> +               force_signal_inject(SIGILL, ILL_ILLOPC, regs, 0);
> +               return;
> +       }
> +
> +       task_fpsimd_save();
> +
> +       sve_alloc(current);
> +       fpsimd_to_sve(current);
> +       if (test_and_set_thread_flag(TIF_SVE))
> +               WARN_ON(1); /* SVE access shouldn't have trapped */
> +
> +       task_fpsimd_load();
> +}
> +
>  /*
>   * Trapped FP/ASIMD access.
>   */
> @@ -135,6 +298,55 @@ void do_fpsimd_exc(unsigned int esr, struct pt_regs *regs)
>         send_sig_info(SIGFPE, &info, current);
>  }
>
> +static void task_fpsimd_load(void)
> +{
> +       if (system_supports_sve() && test_thread_flag(TIF_SVE)) {
> +               unsigned int vl = current->thread.sve_vl;
> +
> +               BUG_ON(!sve_vl_valid(vl));
> +               sve_load_state(sve_pffr(current),
> +                              &current->thread.fpsimd_state.fpsr,
> +                              sve_vq_from_vl(vl) - 1);
> +       } else
> +               fpsimd_load_state(&current->thread.fpsimd_state);
> +

Please use braces consistently on all branches of an if ()

> +       if (system_supports_sve()) {
> +               u64 cpacr = read_sysreg(CPACR_EL1);
> +               u64 new_cpacr;
> +
> +               /* Toggle SVE trapping for userspace if needed */
> +               if (test_thread_flag(TIF_SVE))
> +                       new_cpacr = sve_cpacr_trap_off(cpacr);
> +               else
> +                       new_cpacr = sve_cpacr_trap_on(cpacr);
> +
> +               change_cpacr(cpacr, new_cpacr);

I understand you want to avoid setting CPACR to the same value, but
this does look a bit clunky IMO. Wouldn't it be much better to have a
generic accessor with a mask and a value that encapsulates this?

> +               /* Serialised by exception return to user */
> +       }
> +}
> +
> +static void task_fpsimd_save(void)
> +{
> +       if (system_supports_sve() &&
> +           in_syscall(current_pt_regs()) &&
> +           test_thread_flag(TIF_SVE)) {
> +               u64 cpacr = read_sysreg(CPACR_EL1);
> +
> +               clear_thread_flag(TIF_SVE);
> +
> +               /* Trap if the task tries to use SVE again: */
> +               change_cpacr(cpacr, sve_cpacr_trap_on(cpacr));
> +       }
> +
> +       if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) {
> +               if (system_supports_sve() && test_thread_flag(TIF_SVE))
> +                       sve_save_state(sve_pffr(current),
> +                                      &current->thread.fpsimd_state.fpsr);
> +               else
> +                       fpsimd_save_state(&current->thread.fpsimd_state);
> +       }
> +}
> +
>  void fpsimd_thread_switch(struct task_struct *next)
>  {
>         if (!system_supports_fpsimd())
> @@ -144,8 +356,8 @@ void fpsimd_thread_switch(struct task_struct *next)
>          * the registers is in fact the most recent userland FPSIMD state of
>          * 'current'.
>          */
> -       if (current->mm && !test_thread_flag(TIF_FOREIGN_FPSTATE))
> -               fpsimd_save_state(&current->thread.fpsimd_state);
> +       if (current->mm)
> +               task_fpsimd_save();
>
>         if (next->mm) {
>                 /*
> @@ -172,8 +384,25 @@ void fpsimd_flush_thread(void)
>
>         local_bh_disable();
>
> -       memset(&current->thread.fpsimd_state, 0, sizeof(struct fpsimd_state));
>         fpsimd_flush_task_state(current);
> +
> +       memset(&current->thread.fpsimd_state, 0, sizeof(struct fpsimd_state));
> +

Any reason in particular this needs to be reordered?

> +       if (system_supports_sve()) {
> +               clear_thread_flag(TIF_SVE);
> +               sve_free(current);
> +
> +               /*
> +                * User tasks must have a valid vector length set, but tasks
> +                * forked early (e.g., init) may not initially have one.
> +                * By now, we will know what the hardware supports, so
> +                * sve_default_vl should be valid, and thus the above
> +                * assignment should ensure a valid VL for the task.
> +                * If not, something went badly wrong.
> +                */
> +               BUG_ON(!sve_vl_valid(current->thread.sve_vl));
> +       }
> +
>         set_thread_flag(TIF_FOREIGN_FPSTATE);
>
>         local_bh_enable();
> @@ -211,7 +440,7 @@ void fpsimd_restore_current_state(void)
>         if (test_and_clear_thread_flag(TIF_FOREIGN_FPSTATE)) {
>                 struct fpsimd_state *st = &current->thread.fpsimd_state;
>
> -               fpsimd_load_state(st);
> +               task_fpsimd_load();
>                 __this_cpu_write(fpsimd_last_state, st);
>                 st->cpu = smp_processor_id();
>         }
> @@ -375,8 +604,8 @@ static int fpsimd_cpu_pm_notifier(struct notifier_block *self,
>  {
>         switch (cmd) {
>         case CPU_PM_ENTER:
> -               if (current->mm && !test_thread_flag(TIF_FOREIGN_FPSTATE))
> -                       fpsimd_save_state(&current->thread.fpsimd_state);
> +               if (current->mm)
> +                       task_fpsimd_save();
>                 this_cpu_write(fpsimd_last_state, NULL);
>                 break;
>         case CPU_PM_EXIT:
> diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
> index 659ae80..e51cb1f 100644
> --- a/arch/arm64/kernel/process.c
> +++ b/arch/arm64/kernel/process.c
> @@ -239,13 +239,17 @@ void flush_thread(void)
>
>  void release_thread(struct task_struct *dead_task)
>  {
> +       fpsimd_release_thread(dead_task);
>  }
>
>  int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
>  {
>         if (current->mm)
>                 fpsimd_preserve_current_state();
> -       *dst = *src;
> +       memcpy(dst, src, arch_task_struct_size);
> +
> +       fpsimd_dup_sve(dst, src);
> +

Is this used for anything except fork()? If not, do we really need to
duplicate the SVE state?

>         return 0;
>  }
>
> diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
> index 8964795..286064e 100644
> --- a/arch/arm64/kernel/traps.c
> +++ b/arch/arm64/kernel/traps.c
> @@ -379,8 +379,8 @@ static int call_undef_hook(struct pt_regs *regs)
>         return fn ? fn(regs, instr) : 1;
>  }
>
> -static void force_signal_inject(int signal, int code, struct pt_regs *regs,
> -                               unsigned long address)
> +void force_signal_inject(int signal, int code, struct pt_regs *regs,
> +                        unsigned long address)
>  {
>         siginfo_t info;
>         void __user *pc = (void __user *)instruction_pointer(regs);
> --
> 2.1.4
>

Re: [PATCH 16/27] arm64/sve: Preserve SVE registers around kernel-mode NEON use

[Ard Biesheuvel]

On 9 August 2017 at 13:05, Dave Martin <Dave.Martin@arm.com> wrote:
> Kernel-mode NEON will corrupt the SVE vector registers, due to the
> way they alias the FPSIMD vector registers in the hardware.
>
> This patch ensures that any live SVE register content for the task
> is saved by kernel_neon_begin().  The data will be restored in the
> usual way on return to userspace.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/kernel/fpsimd.c | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> index 955c873..b7fb836 100644
> --- a/arch/arm64/kernel/fpsimd.c
> +++ b/arch/arm64/kernel/fpsimd.c
> @@ -758,8 +758,10 @@ void kernel_neon_begin(void)
>         __this_cpu_write(kernel_neon_busy, true);
>
>         /* Save unsaved task fpsimd state, if any: */
> -       if (current->mm && !test_and_set_thread_flag(TIF_FOREIGN_FPSTATE))
> -               fpsimd_save_state(&current->thread.fpsimd_state);
> +       if (current->mm) {
> +               task_fpsimd_save();
> +               set_thread_flag(TIF_FOREIGN_FPSTATE);
> +       }
>
>         /* Invalidate any task state remaining in the fpsimd regs: */
>         __this_cpu_write(fpsimd_last_state, NULL);

Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>

Re: [PATCH 16/27] arm64/sve: Preserve SVE registers around kernel-mode NEON use

[Ard Biesheuvel]

On 9 August 2017 at 13:05, Dave Martin <Dave.Martin@arm.com> wrote:
> Kernel-mode NEON will corrupt the SVE vector registers, due to the
> way they alias the FPSIMD vector registers in the hardware.
>
> This patch ensures that any live SVE register content for the task
> is saved by kernel_neon_begin().  The data will be restored in the
> usual way on return to userspace.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/kernel/fpsimd.c | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> index 955c873..b7fb836 100644
> --- a/arch/arm64/kernel/fpsimd.c
> +++ b/arch/arm64/kernel/fpsimd.c
> @@ -758,8 +758,10 @@ void kernel_neon_begin(void)
>         __this_cpu_write(kernel_neon_busy, true);
>
>         /* Save unsaved task fpsimd state, if any: */
> -       if (current->mm && !test_and_set_thread_flag(TIF_FOREIGN_FPSTATE))
> -               fpsimd_save_state(&current->thread.fpsimd_state);
> +       if (current->mm) {
> +               task_fpsimd_save();
> +               set_thread_flag(TIF_FOREIGN_FPSTATE);
> +       }
>
>         /* Invalidate any task state remaining in the fpsimd regs: */
>         __this_cpu_write(fpsimd_last_state, NULL);

Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>

Re: [PATCH 17/27] arm64/sve: Preserve SVE registers around EFI runtime service calls

[Ard Biesheuvel]

On 9 August 2017 at 13:05, Dave Martin <Dave.Martin@arm.com> wrote:
> The EFI runtime services ABI allows EFI to make free use of the
> FPSIMD registers during EFI runtime service calls, subject to the
> callee-save requirements of the AArch64 procedure call standard.
>
> However, the SVE architecture allows upper bits of the SVE vector
> registers to be zeroed as a side-effect of FPSIMD V-register
> writes.  This means that the SVE vector registers must be saved in
> their entirety in order to avoid data loss: non-SVE-aware EFI
> implementations cannot restore them correctly.
>
> The non-IRQ case is already handled gracefully by
> kernel_neon_begin().  For the IRQ case, this patch allocates a
> suitable per-CPU stash buffer for the full SVE register state and
> uses it to preserve the affected registers around EFI calls.  It is
> currently unclear how the EFI runtime services ABI will be
> clarified with respect to SVE, so it safest to assume that the
> predicate registers and FFR must be saved and restored too.
>
> No attempt is made to restore the restore the vector length after
> a call, for now.  It is deemed rather insane for EFI to change it,
> and contemporary EFI implementations certainly won't.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/kernel/fpsimd.c | 53 ++++++++++++++++++++++++++++++++++++++++++----
>  1 file changed, 49 insertions(+), 4 deletions(-)
>
> diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> index b7fb836..c727b47 100644
> --- a/arch/arm64/kernel/fpsimd.c
> +++ b/arch/arm64/kernel/fpsimd.c
> @@ -120,12 +120,14 @@ int sve_max_vl = -1;
>  /* Set of available vector lengths, as vq_to_bit(vq): */
>  static DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX);
>  static bool sve_vq_map_finalised;
> +static void __percpu *efi_sve_state;
>
>  #else /* ! CONFIG_ARM64_SVE */
>
>  /* Dummy declaration for code that will be optimised out: */
>  extern DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX);
>  extern bool sve_vq_map_finalised;
> +extern void __percpu *efi_sve_state;
>
>  #endif /* ! CONFIG_ARM64_SVE */
>
> @@ -416,6 +418,23 @@ int sve_verify_vq_map(void)
>         return ret;
>  }
>
> +static void __init sve_kernel_mode_neon_setup(void)
> +{
> +       if (!IS_ENABLED(CONFIG_KERNEL_MODE_NEON))
> +               return;
> +
> +       /*
> +        * alloc_percpu() warns and prints a backtrace if this goes wrong.
> +        * This is evidence of a crippled system and we are returning void,
> +        * so no attempt is made to handle this situation here.
> +        */
> +       BUG_ON(!sve_vl_valid(sve_max_vl));
> +       efi_sve_state = __alloc_percpu(
> +               SVE_SIG_REGS_SIZE(sve_vq_from_vl(sve_max_vl)), 16);
> +       if (!efi_sve_state)
> +               panic("Cannot allocate percpu memory for EFI SVE save/restore");


Do we really need to panic here?

> +}
> +
>  void __init sve_setup(void)
>  {
>         u64 zcr;
> @@ -455,6 +474,8 @@ void __init sve_setup(void)
>                 sve_max_vl);
>         pr_info("SVE: default vector length %u bytes per vector\n",
>                 sve_default_vl);
> +
> +       sve_kernel_mode_neon_setup();
>  }
>
>  void fpsimd_release_thread(struct task_struct *dead_task)
> @@ -797,6 +818,7 @@ EXPORT_SYMBOL(kernel_neon_end);
>
>  DEFINE_PER_CPU(struct fpsimd_state, efi_fpsimd_state);
>  DEFINE_PER_CPU(bool, efi_fpsimd_state_used);
> +DEFINE_PER_CPU(bool, efi_sve_state_used);
>

Could this be static?

>  /*
>   * EFI runtime services support functions
> @@ -825,7 +847,20 @@ void __efi_fpsimd_begin(void)
>         if (may_use_simd())
>                 kernel_neon_begin();
>         else {
> -               fpsimd_save_state(this_cpu_ptr(&efi_fpsimd_state));
> +               /*
> +                * If !efi_sve_state, SVE can't be in use yet and doesn't need
> +                * preserving:
> +                */
> +               if (system_supports_sve() && likely(efi_sve_state)) {
> +                       char *sve_state = this_cpu_ptr(efi_sve_state);
> +
> +                       __this_cpu_write(efi_sve_state_used, true);
> +
> +                       sve_save_state(sve_state + sve_ffr_offset(sve_max_vl),
> +                                      &this_cpu_ptr(&efi_fpsimd_state)->fpsr);
> +               } else
> +                       fpsimd_save_state(this_cpu_ptr(&efi_fpsimd_state));
> +

Consistent braces please

>                 __this_cpu_write(efi_fpsimd_state_used, true);
>         }
>  }
> @@ -838,10 +873,20 @@ void __efi_fpsimd_end(void)
>         if (!system_supports_fpsimd())
>                 return;
>
> -       if (__this_cpu_xchg(efi_fpsimd_state_used, false))
> -               fpsimd_load_state(this_cpu_ptr(&efi_fpsimd_state));
> -       else
> +       if (!__this_cpu_xchg(efi_fpsimd_state_used, false))
>                 kernel_neon_end();
> +       else
> +               if (system_supports_sve() &&
> +                   likely(__this_cpu_read(efi_sve_state_used))) {
> +                       char const *sve_state = this_cpu_ptr(efi_sve_state);
> +
> +                       sve_load_state(sve_state + sve_ffr_offset(sve_max_vl),
> +                                      &this_cpu_ptr(&efi_fpsimd_state)->fpsr,
> +                                      sve_vq_from_vl(sve_get_vl()) - 1);
> +
> +                       __this_cpu_write(efi_sve_state_used, false);
> +               } else
> +                       fpsimd_load_state(this_cpu_ptr(&efi_fpsimd_state));

Please use braces for non-trivial if/else conditions

>  }
>
>  #endif /* CONFIG_KERNEL_MODE_NEON */
> --
> 2.1.4
>

With those fixed

Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>

Re: [PATCH 17/27] arm64/sve: Preserve SVE registers around EFI runtime service calls

[Dave Martin]

On Tue, Aug 15, 2017 at 06:44:45PM +0100, Ard Biesheuvel wrote:
> On 9 August 2017 at 13:05, Dave Martin <Dave.Martin@arm.com> wrote:
> > The EFI runtime services ABI allows EFI to make free use of the
> > FPSIMD registers during EFI runtime service calls, subject to the
> > callee-save requirements of the AArch64 procedure call standard.
> >
> > However, the SVE architecture allows upper bits of the SVE vector
> > registers to be zeroed as a side-effect of FPSIMD V-register
> > writes.  This means that the SVE vector registers must be saved in
> > their entirety in order to avoid data loss: non-SVE-aware EFI
> > implementations cannot restore them correctly.
> >
> > The non-IRQ case is already handled gracefully by
> > kernel_neon_begin().  For the IRQ case, this patch allocates a
> > suitable per-CPU stash buffer for the full SVE register state and
> > uses it to preserve the affected registers around EFI calls.  It is
> > currently unclear how the EFI runtime services ABI will be
> > clarified with respect to SVE, so it safest to assume that the
> > predicate registers and FFR must be saved and restored too.
> >
> > No attempt is made to restore the restore the vector length after
> > a call, for now.  It is deemed rather insane for EFI to change it,
> > and contemporary EFI implementations certainly won't.
> >
> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> > ---
> >  arch/arm64/kernel/fpsimd.c | 53 ++++++++++++++++++++++++++++++++++++++++++----
> >  1 file changed, 49 insertions(+), 4 deletions(-)
> >
> > diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c

[...]

> > +static void __init sve_kernel_mode_neon_setup(void)
> > +{
> > +       if (!IS_ENABLED(CONFIG_KERNEL_MODE_NEON))
> > +               return;
> > +
> > +       /*
> > +        * alloc_percpu() warns and prints a backtrace if this goes wrong.
> > +        * This is evidence of a crippled system and we are returning void,
> > +        * so no attempt is made to handle this situation here.
> > +        */
> > +       BUG_ON(!sve_vl_valid(sve_max_vl));
> > +       efi_sve_state = __alloc_percpu(
> > +               SVE_SIG_REGS_SIZE(sve_vq_from_vl(sve_max_vl)), 16);
> > +       if (!efi_sve_state)
> > +               panic("Cannot allocate percpu memory for EFI SVE save/restore");
> 
> 
> Do we really need to panic here?

Debatable.  I'm a bit unfomfortable just leaving the kernel to bleed to
death of its own accord.  OTOH, if this allocation fails, the kernel is
unlikely to survive long enough to do any real damage.

Unfortunately we're in a call chain that leads to a void function
called from core code, so returning an error is impossible.  I'm very
unconvinved that applying percpu offset on NULL will result in
something that is "NULL enough" to guarantee a fault.  Perhaps this is
supposed to be a guarantee of the percpu design, but I've seen no
statement about this anywhere.

A similar quandary also effects
drivers/firmware/efi/arm-init.c:efi_init().  In the end I followed its
lead, but I don't know which answer is correct.


[...]

> >  void fpsimd_release_thread(struct task_struct *dead_task)
> > @@ -797,6 +818,7 @@ EXPORT_SYMBOL(kernel_neon_end);
> >
> >  DEFINE_PER_CPU(struct fpsimd_state, efi_fpsimd_state);
> >  DEFINE_PER_CPU(bool, efi_fpsimd_state_used);
> > +DEFINE_PER_CPU(bool, efi_sve_state_used);
> >
> 
> Could this be static?

Hmm, looks like it.  I just followed the pattern I'd already set, but
it looks like efi_fpsimd_state and efi_fpsimd_state_used ought to be
static too.

I'll fix those, though it'll be a separate patch since Catalin already
took the EFI fpsimd stuff.

> >  /*
> >   * EFI runtime services support functions
> > @@ -825,7 +847,20 @@ void __efi_fpsimd_begin(void)
> >         if (may_use_simd())
> >                 kernel_neon_begin();
> >         else {
> > -               fpsimd_save_state(this_cpu_ptr(&efi_fpsimd_state));
> > +               /*
> > +                * If !efi_sve_state, SVE can't be in use yet and doesn't need
> > +                * preserving:
> > +                */
> > +               if (system_supports_sve() && likely(efi_sve_state)) {
> > +                       char *sve_state = this_cpu_ptr(efi_sve_state);
> > +
> > +                       __this_cpu_write(efi_sve_state_used, true);
> > +
> > +                       sve_save_state(sve_state + sve_ffr_offset(sve_max_vl),
> > +                                      &this_cpu_ptr(&efi_fpsimd_state)->fpsr);
> > +               } else
> > +                       fpsimd_save_state(this_cpu_ptr(&efi_fpsimd_state));
> > +
> 
> Consistent braces please

Argh, I always seem to have a blind spot about this one.

(I do disagree with this rule, but that's another story...)

Will fix, and I'll double check whether any others instances of this
slipped through.

> >                 __this_cpu_write(efi_fpsimd_state_used, true);
> >         }
> >  }
> > @@ -838,10 +873,20 @@ void __efi_fpsimd_end(void)
> >         if (!system_supports_fpsimd())
> >                 return;
> >
> > -       if (__this_cpu_xchg(efi_fpsimd_state_used, false))
> > -               fpsimd_load_state(this_cpu_ptr(&efi_fpsimd_state));
> > -       else
> > +       if (!__this_cpu_xchg(efi_fpsimd_state_used, false))
> >                 kernel_neon_end();
> > +       else
> > +               if (system_supports_sve() &&
> > +                   likely(__this_cpu_read(efi_sve_state_used))) {
> > +                       char const *sve_state = this_cpu_ptr(efi_sve_state);
> > +
> > +                       sve_load_state(sve_state + sve_ffr_offset(sve_max_vl),
> > +                                      &this_cpu_ptr(&efi_fpsimd_state)->fpsr,
> > +                                      sve_vq_from_vl(sve_get_vl()) - 1);
> > +
> > +                       __this_cpu_write(efi_sve_state_used, false);
> > +               } else
> > +                       fpsimd_load_state(this_cpu_ptr(&efi_fpsimd_state));
> 
> Please use braces for non-trivial if/else conditions

Do you mean add braces around the else clause?  It looks like I should
add those for consistency.

Otherwise, I'm not sure what you're getting at here.  I don't think the
nature of the _condition_ has any bearing on this (?)

> 
> >  }
> >
> >  #endif /* CONFIG_KERNEL_MODE_NEON */
> > --
> > 2.1.4
> >
> 
> With those fixed
> 
> Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>

Thanks, I'll hold off on applying that until we've concluded on the
above.

Cheers
---Dave

Re: [PATCH 11/27] arm64/sve: Core task context handling

[Dave Martin]

On Tue, Aug 15, 2017 at 06:31:05PM +0100, Ard Biesheuvel wrote:
> Hi Dave,
> 
> On 9 August 2017 at 13:05, Dave Martin <Dave.Martin@arm.com> wrote:
> > This patch adds the core support for switching and managing the SVE
> > architectural state of user tasks.
> >
> > Calls to the existing FPSIMD low-level save/restore functions are
> > factored out as new functions task_fpsimd_{save,load}(), since SVE
> > now dynamically may or may not need to be handled at these points
> > depending on the kernel configuration, hardware features discovered
> > at boot, and the runtime state of the task.  To make these
> > decisions as fast as possible, const cpucaps are used where
> > feasible, via the system_supports_sve() helper.
> >
> > The SVE registers are only tracked for threads that have explicitly
> > used SVE, indicated by the new thread flag TIF_SVE.  Otherwise, the
> > FPSIMD view of the architectural state is stored in
> > thread.fpsimd_state as usual.
> >
> > When in use, the SVE registers are not stored directly in
> > thread_struct due to their potentially large and variable size.
> > Because the task_struct slab allocator must be configured very
> > early during kernel boot, it is also tricky to configure it
> > correctly to match the maximum vector length provided by the
> > hardware, since this depends on examining secondary CPUs as well as
> > the primary.  Instead, a pointer sve_state in thread_struct points
> > to a dynamically allocated buffer containing the SVE register data,
> > and code is added to allocate, duplicate and free this buffer at
> > appropriate times.
> >
> > TIF_SVE is set when taking an SVE access trap from userspace, if
> > suitable hardware support has been detected.  This enables SVE for
> > the thread: a subsequent return to userspace will disable the trap
> > accordingly.  If such a trap is taken without sufficient hardware
> > support, SIGILL is sent to the thread instead as if an undefined
> > instruction had been executed: this may happen if userspace tries
> > to use SVE in a system where not all CPUs support it for example.
> >
> > The kernel may clear TIF_SVE and disable SVE for the thread
> > whenever an explicit syscall is made by userspace, though this is
> > considered an optimisation opportunity rather than a deterministic
> > guarantee: the kernel may not do this on every syscall, but it is
> > permitted to do so.  For backwards compatibility reasons and
> > conformance with the spirit of the base AArch64 procedure call
> > standard, the subset of the SVE register state that aliases the
> > FPSIMD registers is still preserved across a syscall even if this
> > happens.
> >
> > TIF_SVE is also cleared, and SVE disabled, on exec: this is an
> > obvious slow path and a hint that we are running a new binary that
> > may not use SVE.
> >
> > Code is added to sync data between thread.fpsimd_state and
> > thread.sve_state whenever enabling/disabling SVE, in a manner
> > consistent with the SVE architectural programmer's model.
> >
> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> > ---
> >  arch/arm64/include/asm/fpsimd.h      |  19 +++
> >  arch/arm64/include/asm/processor.h   |   2 +
> >  arch/arm64/include/asm/thread_info.h |   1 +
> >  arch/arm64/include/asm/traps.h       |   2 +
> >  arch/arm64/kernel/entry.S            |  14 +-
> >  arch/arm64/kernel/fpsimd.c           | 241 ++++++++++++++++++++++++++++++++++-
> >  arch/arm64/kernel/process.c          |   6 +-
> >  arch/arm64/kernel/traps.c            |   4 +-
> >  8 files changed, 279 insertions(+), 10 deletions(-)
> >
> > diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
> > index 026a7c7..72090a1 100644
> > --- a/arch/arm64/include/asm/fpsimd.h
> > +++ b/arch/arm64/include/asm/fpsimd.h
> > @@ -20,6 +20,8 @@
> >
> >  #ifndef __ASSEMBLY__
> >
> > +#include <linux/stddef.h>
> > +
> >  /*
> >   * FP/SIMD storage area has:
> >   *  - FPSR and FPCR
> > @@ -72,6 +74,23 @@ extern void sve_load_state(void const *state, u32 const *pfpsr,
> >                            unsigned long vq_minus_1);
> >  extern unsigned int sve_get_vl(void);
> >
> > +#ifdef CONFIG_ARM64_SVE
> > +
> > +extern size_t sve_state_size(struct task_struct const *task);
> > +
> > +extern void sve_alloc(struct task_struct *task);
> > +extern void fpsimd_release_thread(struct task_struct *task);
> > +extern void fpsimd_dup_sve(struct task_struct *dst,
> > +                          struct task_struct const *src);
> > +
> > +#else /* ! CONFIG_ARM64_SVE */
> > +
> > +static void __maybe_unused sve_alloc(struct task_struct *task) { }
> > +static void __maybe_unused fpsimd_release_thread(struct task_struct *task) { }
> > +static void __maybe_unused fpsimd_dup_sve(struct task_struct *dst,
> > +                                         struct task_struct const *src) { }
> > +#endif /* ! CONFIG_ARM64_SVE */
> > +
> >  /* For use by EFI runtime services calls only */
> >  extern void __efi_fpsimd_begin(void);
> >  extern void __efi_fpsimd_end(void);
> > diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
> > index b7334f1..969feed 100644
> > --- a/arch/arm64/include/asm/processor.h
> > +++ b/arch/arm64/include/asm/processor.h
> > @@ -85,6 +85,8 @@ struct thread_struct {
> >         unsigned long           tp2_value;
> >  #endif
> >         struct fpsimd_state     fpsimd_state;
> > +       void                    *sve_state;     /* SVE registers, if any */
> > +       u16                     sve_vl;         /* SVE vector length */
> >         unsigned long           fault_address;  /* fault info */
> >         unsigned long           fault_code;     /* ESR_EL1 value */
> >         struct debug_info       debug;          /* debugging */
> > diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h
> > index 46c3b93..1a4b30b 100644
> > --- a/arch/arm64/include/asm/thread_info.h
> > +++ b/arch/arm64/include/asm/thread_info.h
> > @@ -96,6 +96,7 @@ struct thread_info {
> >  #define TIF_RESTORE_SIGMASK    20
> >  #define TIF_SINGLESTEP         21
> >  #define TIF_32BIT              22      /* 32bit process */
> > +#define TIF_SVE                        23      /* Scalable Vector Extension in use */
> >
> >  #define _TIF_SIGPENDING                (1 << TIF_SIGPENDING)
> >  #define _TIF_NEED_RESCHED      (1 << TIF_NEED_RESCHED)
> > diff --git a/arch/arm64/include/asm/traps.h b/arch/arm64/include/asm/traps.h
> > index 02e9035..f058c07 100644
> > --- a/arch/arm64/include/asm/traps.h
> > +++ b/arch/arm64/include/asm/traps.h
> > @@ -34,6 +34,8 @@ struct undef_hook {
> >
> >  void register_undef_hook(struct undef_hook *hook);
> >  void unregister_undef_hook(struct undef_hook *hook);
> > +void force_signal_inject(int signal, int code, struct pt_regs *regs,
> > +                        unsigned long address);
> >
> >  void arm64_notify_segfault(struct pt_regs *regs, unsigned long addr);
> >
> > diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
> > index cace76d..c33069c 100644
> > --- a/arch/arm64/kernel/entry.S
> > +++ b/arch/arm64/kernel/entry.S
> > @@ -524,6 +524,8 @@ el0_sync:
> >         b.eq    el0_ia
> >         cmp     x24, #ESR_ELx_EC_FP_ASIMD       // FP/ASIMD access
> >         b.eq    el0_fpsimd_acc
> > +       cmp     x24, #ESR_ELx_EC_SVE            // SVE access
> > +       b.eq    el0_sve_acc
> >         cmp     x24, #ESR_ELx_EC_FP_EXC64       // FP/ASIMD exception
> >         b.eq    el0_fpsimd_exc
> >         cmp     x24, #ESR_ELx_EC_SYS64          // configurable trap
> > @@ -622,9 +624,19 @@ el0_fpsimd_acc:
> >         mov     x1, sp
> >         bl      do_fpsimd_acc
> >         b       ret_to_user
> > +el0_sve_acc:
> > +       /*
> > +        * Scalable Vector Extension access
> > +        */
> > +       enable_dbg
> > +       ct_user_exit
> > +       mov     x0, x25
> > +       mov     x1, sp
> > +       bl      do_sve_acc
> > +       b       ret_to_user
> >  el0_fpsimd_exc:
> >         /*
> > -        * Floating Point or Advanced SIMD exception
> > +        * Floating Point, Advanced SIMD or SVE exception
> >          */
> >         enable_dbg
> >         ct_user_exit
> > diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> > index 9c1f268e..37dd1b2 100644
> > --- a/arch/arm64/kernel/fpsimd.c
> > +++ b/arch/arm64/kernel/fpsimd.c
> > @@ -24,12 +24,17 @@
> >  #include <linux/init.h>
> >  #include <linux/percpu.h>
> >  #include <linux/preempt.h>
> > +#include <linux/ptrace.h>
> >  #include <linux/sched/signal.h>
> >  #include <linux/signal.h>
> > +#include <linux/slab.h>
> >
> >  #include <asm/fpsimd.h>
> >  #include <asm/cputype.h>
> >  #include <asm/simd.h>
> > +#include <asm/sigcontext.h>
> > +#include <asm/sysreg.h>
> > +#include <asm/traps.h>
> >
> >  #define FPEXC_IOF      (1 << 0)
> >  #define FPEXC_DZF      (1 << 1)
> > @@ -38,6 +43,10 @@
> >  #define FPEXC_IXF      (1 << 4)
> >  #define FPEXC_IDF      (1 << 7)
> >
> > +/* Forward declarations for local functions used by both SVE and FPSIMD */
> > +static void task_fpsimd_load(void);
> > +static void task_fpsimd_save(void);
> > +
> 
> We usually try to avoid forward declarations for functions with static
> linkage. Is it possible to reorder them and get rid of this?

So do I :)  There was some sort of historical reason for this, to do
with some forward and backward dependencies and trying to corral
affected code into a single #ifdef rather than having multiple #ifdefs.
These two declarations were the most stubborn, and I gave up on them.

Now that the ifdeffery is mostly gone it looks like these _can_ probably
be got rid of, so I'll try to do that.

> 
> >  /*
> >   * In order to reduce the number of times the FPSIMD state is needlessly saved
> >   * and restored, we need to keep track of two things:
> > @@ -99,6 +108,160 @@
> >   */
> >  static DEFINE_PER_CPU(struct fpsimd_state *, fpsimd_last_state);
> >
> > +static void sve_free(struct task_struct *task)
> > +{
> > +       kfree(task->thread.sve_state);
> > +       task->thread.sve_state = NULL;
> > +}
> > +
> > +/* Offset of FFR in the SVE register dump */
> > +static size_t sve_ffr_offset(int vl)
> > +{
> > +       BUG_ON(!sve_vl_valid(vl));
> 
> BUG_ON() is a pretty heavy hammer, so we should not use it unless the
> kernel state is so corrupted that there is no way to carry on. I have
> a feeling this may not be the case for some of the occurrences in this
> patch.

These were useful during development, but you're right, some of these
are probably superfluous.

Bugs aside, there should be no way for an invalid value to get into
thread.sve_vl, sve_max_vl etc., with the one exception that the init
task and kernel tasks forked from it have thread.sve_vl = 0 (which
doesn't matter because SVE state is never loaded, saved or otherwise
manipulated for these tasks due to never entering userspace).  (The
logic that attempts to ensure that these values are only set validly is
in patches 14 (thread.sve_vl), 15 (sve_max_vl), and 12 and 20
(sve_default_vl).)

I think if we try to enter userspace with an invalid vl that may be
worth a BUG(), but elsewhere we can probably just use SVE_MIN_VL
instead of "invalid" values.

In particular, poisoning sve_max_vl and sve_default_vl with -1 is
probably not needed now the development phase is over.  These could be
initially SVE_MIN_VL without ill effects.

I'll take a look at this.

> 
> > +       return SVE_SIG_FFR_OFFSET(sve_vq_from_vl(vl)) - SVE_SIG_REGS_OFFSET;
> > +}
> > +
> > +static void *sve_pffr(struct task_struct *task)
> > +{
> > +       unsigned int vl = task->thread.sve_vl;
> > +
> > +       BUG_ON(!sve_vl_valid(vl) || !task->thread.sve_state);
> > +       return (char *)task->thread.sve_state + sve_ffr_offset(vl);
> > +}
> > +
> > +static u64 sve_cpacr_trap_on(u64 cpacr)
> > +{
> > +       return cpacr & ~(u64)CPACR_EL1_ZEN_EL0EN;
> > +}
> > +
> > +static u64 sve_cpacr_trap_off(u64 cpacr)
> > +{
> > +       return cpacr | CPACR_EL1_ZEN_EL0EN;
> > +}
> > +
> > +static void change_cpacr(u64 old, u64 new)
> > +{
> > +       if (old != new)
> > +               write_sysreg(new, CPACR_EL1);
> > +}
> > +
> > +#ifdef CONFIG_ARM64_SVE
> > +
> > +#define ZREG(sve_state, vq, n) ((char *)(sve_state) +          \
> > +       (SVE_SIG_ZREG_OFFSET(vq, n) - SVE_SIG_REGS_OFFSET))
> > +
> > +static void sve_to_fpsimd(struct task_struct *task)
> > +{
> > +       unsigned int vl = task->thread.sve_vl;
> > +       unsigned int vq;
> > +       void const *sst = task->thread.sve_state;
> > +       struct fpsimd_state *fst = &task->thread.fpsimd_state;
> > +       unsigned int i;
> > +
> > +       if (!system_supports_sve())
> > +               return;
> > +
> > +       BUG_ON(!sve_vl_valid(vl));
> > +       vq = sve_vq_from_vl(vl);
> > +
> > +       for (i = 0; i < 32; ++i)
> > +               memcpy(&fst->vregs[i], ZREG(sst, vq, i),
> > +                      sizeof(fst->vregs[i]));
> > +}
> > +
> > +static void fpsimd_to_sve(struct task_struct *task)
> > +{
> > +       unsigned int vl = task->thread.sve_vl;
> > +       unsigned int vq;
> > +       void *sst = task->thread.sve_state;
> > +       struct fpsimd_state const *fst = &task->thread.fpsimd_state;
> > +       unsigned int i;
> > +
> > +       if (!system_supports_sve())
> > +               return;
> > +
> > +       BUG_ON(!sve_vl_valid(vl));
> > +       vq = sve_vq_from_vl(vl);
> > +
> > +       for (i = 0; i < 32; ++i)
> > +               memcpy(ZREG(sst, vq, i), &fst->vregs[i],
> > +                      sizeof(fst->vregs[i]));
> > +}
> > +
> > +size_t sve_state_size(struct task_struct const *task)
> > +{
> > +       unsigned int vl = task->thread.sve_vl;
> > +
> > +       BUG_ON(!sve_vl_valid(vl));
> > +       return SVE_SIG_REGS_SIZE(sve_vq_from_vl(vl));
> > +}
> > +
> > +void sve_alloc(struct task_struct *task)
> > +{
> > +       if (task->thread.sve_state) {
> > +               memset(task->thread.sve_state, 0, sve_state_size(current));
> > +               return;
> > +       }
> > +
> > +       /* This is a small allocation (maximum ~8KB) and Should Not Fail. */
> > +       task->thread.sve_state =
> > +               kzalloc(sve_state_size(task), GFP_KERNEL);
> > +
> > +       /*
> > +        * If future SVE revisions can have larger vectors though,
> > +        * this may cease to be true:
> > +        */
> > +       BUG_ON(!task->thread.sve_state);
> > +}
> > +
> > +/*
> > + * Handle SVE state across fork():
> > + *
> > + * dst and src must not end up with aliases of the same sve_state.
> > + * Because a task cannot fork except in a syscall, we can discard SVE
> > + * state for dst here: reallocation will be deferred until dst tries
> > + * to use SVE.
> > + */
> > +void fpsimd_dup_sve(struct task_struct *dst, struct task_struct const *src)
> > +{
> > +       BUG_ON(!in_syscall(task_pt_regs(dst)));
> > +
> > +       if (test_and_clear_tsk_thread_flag(dst, TIF_SVE)) {
> > +               sve_to_fpsimd(dst);
> > +               dst->thread.sve_state = NULL;

Looking at this in reponse to your comment on arch_dup_task_struct(), I
think there's a bug here: if the forking task has TIF_SVE clear, then a
non-NULL sve_state pointer will remain cloned from the parent task.
The assignment of NULL needs to move outside the if.

This bug would pop up with programs that use SVE and then go
multithreaded, or programs that use SVE and then fork+exec.  In the
former case we'd end up with aliased state; in the latter, the state
would get freed by the exec while the parent task still owns it.

> > +       }
> > +}
> > +
> > +void fpsimd_release_thread(struct task_struct *dead_task)
> > +{
> > +       sve_free(dead_task);
> > +}
> > +
> > +#endif /* CONFIG_ARM64_SVE */
> > +
> > +/*
> > + * Trapped SVE access
> > + */
> > +void do_sve_acc(unsigned int esr, struct pt_regs *regs)
> > +{
> > +       BUG_ON(is_compat_task());
> > +
> > +       /* Even if we chose not to use SVE, the hardware could still trap: */
> > +       if (unlikely(!system_supports_sve())) {
> > +               force_signal_inject(SIGILL, ILL_ILLOPC, regs, 0);
> > +               return;
> > +       }
> > +
> > +       task_fpsimd_save();
> > +
> > +       sve_alloc(current);
> > +       fpsimd_to_sve(current);
> > +       if (test_and_set_thread_flag(TIF_SVE))
> > +               WARN_ON(1); /* SVE access shouldn't have trapped */
> > +
> > +       task_fpsimd_load();
> > +}
> > +
> >  /*
> >   * Trapped FP/ASIMD access.
> >   */
> > @@ -135,6 +298,55 @@ void do_fpsimd_exc(unsigned int esr, struct pt_regs *regs)
> >         send_sig_info(SIGFPE, &info, current);
> >  }
> >
> > +static void task_fpsimd_load(void)
> > +{
> > +       if (system_supports_sve() && test_thread_flag(TIF_SVE)) {
> > +               unsigned int vl = current->thread.sve_vl;
> > +
> > +               BUG_ON(!sve_vl_valid(vl));
> > +               sve_load_state(sve_pffr(current),
> > +                              &current->thread.fpsimd_state.fpsr,
> > +                              sve_vq_from_vl(vl) - 1);
> > +       } else
> > +               fpsimd_load_state(&current->thread.fpsimd_state);
> > +
> 
> Please use braces consistently on all branches of an if ()

Arg, will fix.

> > +       if (system_supports_sve()) {
> > +               u64 cpacr = read_sysreg(CPACR_EL1);
> > +               u64 new_cpacr;
> > +
> > +               /* Toggle SVE trapping for userspace if needed */
> > +               if (test_thread_flag(TIF_SVE))
> > +                       new_cpacr = sve_cpacr_trap_off(cpacr);
> > +               else
> > +                       new_cpacr = sve_cpacr_trap_on(cpacr);
> > +
> > +               change_cpacr(cpacr, new_cpacr);
> 
> I understand you want to avoid setting CPACR to the same value, but
> this does look a bit clunky IMO. Wouldn't it be much better to have a
> generic accessor with a mask and a value that encapsulates this?

Hmmm, this was a replacement for some more obscure logic that was there
previously and that Mark Rutland suggested cleaning up.

Because of the way this is used, I think it would be reasonable to have

	static void change_cpacr(u64 new_bits, u64 mask)

which elides the cpacr write if no change would result.  Then the _off()
and _on() could be folded into the call sites as 0 and
CPACR_EL1_ZEN_EL0EN respectively.

But since this function would be used for nothing other than toggling
CPACR_EL1_ZEN_EL0EN this looks pointlessly general.

Alternatively we could have two functions

	static void sve_el0_trap_on(void);
	static void sve_el0_trap_off(void);

which contain the CPACR manipulation so the caller doesn't have to spell
it out.

Any thoughts?

> 
> > +               /* Serialised by exception return to user */
> > +       }
> > +}
> > +
> > +static void task_fpsimd_save(void)
> > +{
> > +       if (system_supports_sve() &&
> > +           in_syscall(current_pt_regs()) &&
> > +           test_thread_flag(TIF_SVE)) {
> > +               u64 cpacr = read_sysreg(CPACR_EL1);
> > +
> > +               clear_thread_flag(TIF_SVE);
> > +
> > +               /* Trap if the task tries to use SVE again: */
> > +               change_cpacr(cpacr, sve_cpacr_trap_on(cpacr));
> > +       }
> > +
> > +       if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) {
> > +               if (system_supports_sve() && test_thread_flag(TIF_SVE))
> > +                       sve_save_state(sve_pffr(current),
> > +                                      &current->thread.fpsimd_state.fpsr);
> > +               else
> > +                       fpsimd_save_state(&current->thread.fpsimd_state);
> > +       }
> > +}
> > +
> >  void fpsimd_thread_switch(struct task_struct *next)
> >  {
> >         if (!system_supports_fpsimd())
> > @@ -144,8 +356,8 @@ void fpsimd_thread_switch(struct task_struct *next)
> >          * the registers is in fact the most recent userland FPSIMD state of
> >          * 'current'.
> >          */
> > -       if (current->mm && !test_thread_flag(TIF_FOREIGN_FPSTATE))
> > -               fpsimd_save_state(&current->thread.fpsimd_state);
> > +       if (current->mm)
> > +               task_fpsimd_save();
> >
> >         if (next->mm) {
> >                 /*
> > @@ -172,8 +384,25 @@ void fpsimd_flush_thread(void)
> >
> >         local_bh_disable();
> >
> > -       memset(&current->thread.fpsimd_state, 0, sizeof(struct fpsimd_state));
> >         fpsimd_flush_task_state(current);
> > +
> > +       memset(&current->thread.fpsimd_state, 0, sizeof(struct fpsimd_state));
> > +
> 
> Any reason in particular this needs to be reordered?

This seems to have been left behind from some overlapping fixes.

The recent commit cb84d11e ("arm64: neon: Remove support for nested or
hardirq kernel-mode NEON") adds the local_bh_disable(), which prevents
this code from racing with fpsimd_thread_switch().

Previously there was a bug, which commit 674c242c ("arm64: flush
FP/SIMD state correctly after execve()") tries to fix by adding the
fpsimd_flush_task_state().  However, I think this is still buggy:
context switch can occur during the memset and replace the zeros with
data from the registers again (which is what the patch was intended to
prevent).  It's important in this case that the
fpsimd_flush_task_state() is first, so that a racing context switch
doesn't try to save anything.

I think I reordered these calls earlier in development while debugging
some race problems, but this is not really an SVE fix.  The kernel-mode
NEON simplifications make this irrelevant for mainline since preemption
is disabled anyway, but a separate fix is probably needed for stable,
either reordering those calls or adding preempt_disable().

For this patch, I think the reordering can be dropped.

Does this logic sound correct to you?

[...]

> >  int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
> >  {
> >         if (current->mm)
> >                 fpsimd_preserve_current_state();
> > -       *dst = *src;
> > +       memcpy(dst, src, arch_task_struct_size);

Looking at this, we can go back to the assignment here.  The memcpy()
is from the days when I reserved extra space off the end of task_struct
to store the SVE state, and hadn't thought about whether the SVE state
really needed to be copied across fork.

> > +
> > +       fpsimd_dup_sve(dst, src);
> > +
> 
> Is this used for anything except fork()? If not, do we really need to

No, I don't think so.

> duplicate the SVE state?

No, but fpsimd_dup_sve() is perhaps unfortunately named because it

doesn't do that.  It dups the fpsimd subset of the state (required by
the pre-existing syscall ABI) which lives in sve_state if TIF_SVE is
currently set.  It should also nuke the sve_state pointer in dst, but
this currently looks broken (see my comments on fpsimd_dup_sve above).

Any thoughts from your side?

[...]

Thanks for the review
---Dave

Re: [PATCH 21/27] arm64/sve: KVM: Prevent guests from using SVE

[Dave Martin]

On Tue, Aug 15, 2017 at 05:33:15PM +0100, Marc Zyngier wrote:
> On 09/08/17 13:05, Dave Martin wrote:
> > Until KVM has full SVE support, guests must not be allowed to
> > execute SVE instructions.
> > 
> > This patch enables the necessary traps, and also ensures that the
> > traps are disabled again on exit from the guest so that the host
> > can still use SVE if it wants to.
> > 
> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> > ---
> >  arch/arm64/include/asm/kvm_arm.h | 3 ++-
> >  arch/arm64/kvm/hyp/switch.c      | 6 +++---
> >  2 files changed, 5 insertions(+), 4 deletions(-)
> > 
> > diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
> > index dbf0537..8a19651 100644
> > --- a/arch/arm64/include/asm/kvm_arm.h
> > +++ b/arch/arm64/include/asm/kvm_arm.h
> > @@ -186,7 +186,7 @@
> >  #define CPTR_EL2_TTA	(1 << 20)
> >  #define CPTR_EL2_TFP	(1 << CPTR_EL2_TFP_SHIFT)
> >  #define CPTR_EL2_TZ	(1 << 8)
> > -#define CPTR_EL2_DEFAULT	0x000033ff
> > +#define CPTR_EL2_DEFAULT	(0x000033ff & ~CPTR_EL2_TZ)
> 
> I must say I'm not overly fond of this construct. I'd rather introduce a
> RES1 field that matches the v8.2 description, instead of this ugly
> constant and something that clears it.

Sorry, I don't get your meaning here.  v8.2 neither immediately predates
or postdates SVE.  What are you propsing?

I guess we could just write 0x000032ff now that the only meaning the
architecture can ever assign to bit 8 is known.

[...]

Cheers
---Dave

Re: [PATCH 23/27] arm64/sve: KVM: Hide SVE from CPU features exposed to guests

[Dave Martin]

On Tue, Aug 15, 2017 at 05:37:55PM +0100, Marc Zyngier wrote:
> On 09/08/17 13:05, Dave Martin wrote:
> > KVM guests cannot currently use SVE, because SVE is always
> > configured to trap to EL2.
> > 
> > However, a guest that sees SVE reported as present in
> > ID_AA64PFR0_EL1 may legitimately expect that SVE works and try to
> > use it.  Instead of working, the guest will receive an injected
> > undef exception, which may cause the guest to oops or go into a
> > spin.
> > 
> > To avoid misleading the guest into believing that SVE will work,
> > this patch masks out the SVE field from ID_AA64PFR0_EL1 when a
> > guest attempts to read this register.  No support is explicitly
> > added for ID_AA64ZFR0_EL1 either, so that is still emulated as
> > reading as zero, which is consistent with SVE not being
> > implemented.
> > 
> > This is a temporary measure, and will be removed in a later series
> > when full KVM support for SVE is implemented.
> > 
> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> > ---
> >  arch/arm64/kvm/sys_regs.c | 14 +++++++++++++-
> >  1 file changed, 13 insertions(+), 1 deletion(-)
> > 
> > diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
> > index 6583dd7..9e8c54e 100644
> > --- a/arch/arm64/kvm/sys_regs.c
> > +++ b/arch/arm64/kvm/sys_regs.c
> > @@ -897,8 +897,20 @@ static u64 read_id_reg(struct sys_reg_desc const *r, bool raz)
> >  {
> >  	u32 id = sys_reg((u32)r->Op0, (u32)r->Op1,
> >  			 (u32)r->CRn, (u32)r->CRm, (u32)r->Op2);
> > +	u64 val = raz ? 0 : read_sanitised_ftr_reg(id);
> >  
> > -	return raz ? 0 : read_sanitised_ftr_reg(id);
> > +	if (id == SYS_ID_AA64PFR0_EL1) {
> > +		static bool printed;
> > +
> > +		if ((val & (0xfUL << ID_AA64PFR0_SVE_SHIFT)) && !printed) {
> > +			kvm_info("SVE unsupported for guests, suppressing\n");
> > +			printed = true;
> > +		}
> 
> Ideally, this should be a vcpu_unimpl_once(). But:
> - it doesn't exist
> - vcpu_unimpl looks hopelessly x86 specific

Yeah, I looked for an appropriate function and didn't find one ... and
writing one just for this seemed overkill.

> How about turning it into a pr_err_once() instead?

Can do, though should it be an err?

No error has occurred here, rather I want people who discover that their
guest mysteriously doesn't see SVE gets a clue about why.

[...]

Cheers
---Dave

Re: [PATCH 02/27] arm64: KVM: Hide unsupported AArch64 CPU features from guests

[Marc Zyngier]

On 09/08/17 13:05, Dave Martin wrote:
> Currently, a guest kernel sees the true CPU feature registers
> (ID_*_EL1) when it reads them using MRS instructions.  This means
> that the guest will observe features that are present in the
> hardware but the host doesn't understand or doesn't provide support
> for.  A guest may legimitately try to use such a feature as per the
> architecture, but use of the feature may trap instead of working
> normally, triggering undef injection into the guest.
> 
> This is not a problem for the host, but the guest may go wrong when
> running on newer hardware than the host knows about.
> 
> This patch hides from guest VMs any AArch64-specific CPU features
> that the host doesn't support, by exposing to the guest the
> sanitised versions of the registers computed by the cpufeatures
> framework, instead of the true hardware registers.  To achieve
> this, HCR_EL2.TID3 is now set for AArch64 guests, and emulation
> code is added to KVM to report the sanitised versions of the
> affected registers in response to MRS and register reads from
> userspace.
> 
> The affected registers are removed from invariant_sys_regs[] (since
> the invariant_sys_regs handling is no longer quite correct for
> them) and added to sys_reg_desgs[], with appropriate access(),
> get_user() and set_user() methods.  No runtime vcpu storage is
> allocated for the registers: instead, they are read on demand from
> the cpufeatures framework.  This may need modification in the
> future if there is a need for userspace to customise the features
> visible to the guest.
> 
> Attempts by userspace to write the registers are handled similarly
> to the current invariant_sys_regs handling: writes are permitted,
> but only if they don't attempt to change the value.  This is
> sufficient to support VM snapshot/restore from userspace.
> 
> Because of the additional registers, restoring a VM on an older
> kernel may not work unless userspace knows how to handle the extra
> VM registers exposed to the KVM user ABI by this patch.
> 
> Under the principle of least damage, this patch makes no attempt to
> handle any of the other registers currently in
> invariant_sys_regs[], or to emulate registers for AArch32: however,
> these could be handled in a similar way in future, as necessary.
> 
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/kvm/hyp/switch.c |   6 ++
>  arch/arm64/kvm/sys_regs.c   | 224 +++++++++++++++++++++++++++++++++++---------
>  2 files changed, 185 insertions(+), 45 deletions(-)
> 
> diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
> index 945e79c..35a90b8 100644
> --- a/arch/arm64/kvm/hyp/switch.c
> +++ b/arch/arm64/kvm/hyp/switch.c
> @@ -81,11 +81,17 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu)
>  	 * it will cause an exception.
>  	 */
>  	val = vcpu->arch.hcr_el2;
> +
>  	if (!(val & HCR_RW) && system_supports_fpsimd()) {
>  		write_sysreg(1 << 30, fpexc32_el2);
>  		isb();
>  	}
> +
> +	if (val & HCR_RW) /* for AArch64 only: */
> +		val |= HCR_TID3; /* TID3: trap feature register accesses */
> +
>  	write_sysreg(val, hcr_el2);
> +
>  	/* Trap on AArch32 cp15 c15 accesses (EL1 or EL0) */
>  	write_sysreg(1 << 15, hstr_el2);
>  	/*
> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
> index 2e070d3..6583dd7 100644
> --- a/arch/arm64/kvm/sys_regs.c
> +++ b/arch/arm64/kvm/sys_regs.c
> @@ -892,6 +892,135 @@ static bool access_cntp_cval(struct kvm_vcpu *vcpu,
>  	return true;
>  }
>  
> +/* Read a sanitised cpufeature ID register by sys_reg_desc */
> +static u64 read_id_reg(struct sys_reg_desc const *r, bool raz)
> +{
> +	u32 id = sys_reg((u32)r->Op0, (u32)r->Op1,
> +			 (u32)r->CRn, (u32)r->CRm, (u32)r->Op2);
> +
> +	return raz ? 0 : read_sanitised_ftr_reg(id);
> +}
> +
> +/* cpufeature ID register access trap handlers */
> +
> +static bool __access_id_reg(struct kvm_vcpu *vcpu,
> +			    struct sys_reg_params *p,
> +			    const struct sys_reg_desc const *r,
> +			    bool raz)
> +{
> +	if (p->is_write) {
> +		kvm_inject_undefined(vcpu);
> +		return false;
> +	}

I don't think this is supposed to happen (should have UNDEF-ed at EL1).
You can call write_to_read_only() in that case, which will spit out a
warning and inject the exception.

> +
> +	p->regval = read_id_reg(r, raz);
> +	return true;
> +}
> +
> +static bool access_id_reg(struct kvm_vcpu *vcpu,
> +			  struct sys_reg_params *p,
> +			  const struct sys_reg_desc *r)
> +{
> +	return __access_id_reg(vcpu, p, r, false);
> +}
> +
> +static bool access_raz_id_reg(struct kvm_vcpu *vcpu,
> +			      struct sys_reg_params *p,
> +			      const struct sys_reg_desc *r)
> +{
> +	return __access_id_reg(vcpu, p, r, true);
> +}
> +
> +static int reg_from_user(u64 *val, const void __user *uaddr, u64 id);
> +static int reg_to_user(void __user *uaddr, const u64 *val, u64 id);
> +static u64 sys_reg_to_index(const struct sys_reg_desc *reg);
> +
> +/*
> + * cpufeature ID register user accessors
> + *
> + * For now, these registers are immutable for userspace, so no values
> + * are stored, and for set_id_reg() we don't allow the effective value
> + * to be changed.
> + */
> +static int __get_id_reg(const struct sys_reg_desc *rd, void __user *uaddr,
> +			bool raz)
> +{
> +	const u64 id = sys_reg_to_index(rd);
> +	const u64 val = read_id_reg(rd, raz);
> +
> +	BUG_ON(KVM_REG_SIZE(id) != sizeof(val));
> +	return reg_to_user(uaddr, &val, id);
> +}
> +
> +static int __set_id_reg(const struct sys_reg_desc *rd, void __user *uaddr,
> +			bool raz)
> +{
> +	const u64 id = sys_reg_to_index(rd);
> +	int err;
> +	u64 val;
> +
> +	BUG_ON(KVM_REG_SIZE(id) != sizeof(val));
> +	err = reg_from_user(&val, uaddr, id);
> +	if (err)
> +		return err;
> +
> +	/* This is what we mean by invariant: you can't change it. */
> +	if (val != read_id_reg(rd, raz))
> +		return -EINVAL;
> +
> +	return 0;
> +}
> +
> +static int get_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
> +		      const struct kvm_one_reg *reg, void __user *uaddr)
> +{
> +	return __get_id_reg(rd, uaddr, false);
> +}
> +
> +static int set_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
> +		      const struct kvm_one_reg *reg, void __user *uaddr)
> +{
> +	return __set_id_reg(rd, uaddr, false);
> +}
> +
> +static int get_raz_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
> +			  const struct kvm_one_reg *reg, void __user *uaddr)
> +{
> +	return __get_id_reg(rd, uaddr, true);
> +}
> +
> +static int set_raz_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
> +			  const struct kvm_one_reg *reg, void __user *uaddr)
> +{
> +	return __set_id_reg(rd, uaddr, true);
> +}
> +
> +/* sys_reg_desc initialiser for cpufeature ID register name_EL1 */
> +#define _ID(name) {			\
> +	SYS_DESC(SYS_##name##_EL1),	\
> +	.access	= access_id_reg,	\
> +	.get_user = get_id_reg,		\
> +	.set_user = set_id_reg,		\
> +}
> +
> +/*
> + * sys_reg_desc initialiser for cpufeature ID register ID_name_EL1
> + * (So we can get 4 regs to 1 line.)
> + */
> +#define ID(name) _ID(ID_##name)
> +
> +/*
> + * sys_reg_desc initialiser for unknown (RAZ) cpufeature ID register
> + * Op0=3, Op1=0, CRn=0, CRm=crm, Op2=op2
> + * (1 <= crm < 8, 0 <= Op2 < 8).
> + */
> +#define _ID_RAZ(crm, op2) {				\
> +	Op0(3), Op1(0), CRn(0), CRm(crm), Op2(op2),	\
> +	.access = access_raz_id_reg,			\
> +	.get_user = get_raz_id_reg,			\
> +	.set_user = set_raz_id_reg,			\
> +}
> +
>  /*
>   * Architected system registers.
>   * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2
> @@ -944,6 +1073,32 @@ static const struct sys_reg_desc sys_reg_descs[] = {
>  	{ SYS_DESC(SYS_DBGVCR32_EL2), NULL, reset_val, DBGVCR32_EL2, 0 },
>  
>  	{ SYS_DESC(SYS_MPIDR_EL1), NULL, reset_mpidr, MPIDR_EL1 },
> +
> +	/*
> +	 * All non-RAZ feature registers listed here must also be
> +	 * present in arm64_ftr_regs[].
> +	 */
> +
> +	/* AArch64 mappings of the AArch32 ID registers */
> +	/* ID_AFR0_EL1 not exposed to guests for now */
> +	ID(PFR0),	ID(PFR1),	ID(DFR0),	_ID_RAZ(1,3),
> +	ID(MMFR0),	ID(MMFR1),	ID(MMFR2),	ID(MMFR3),
> +	ID(ISAR0),	ID(ISAR1),	ID(ISAR2),	ID(ISAR3),
> +	ID(ISAR4),	ID(ISAR5),	ID(MMFR4),	_ID_RAZ(2,7),
> +	_ID(MVFR0),	_ID(MVFR1),	_ID(MVFR2),	_ID_RAZ(3,3),
> +	_ID_RAZ(3,4),	_ID_RAZ(3,5),	_ID_RAZ(3,6),	_ID_RAZ(3,7),

#bikeshed:

OK, this is giving me a headache. Too many variants with similar names.
ID and _ID
I'm also slightly perplexed with the amalgamation of RAZ because the
register is not defined yet in the architecture, and RAZ because we
don't expose it (like ID_AFR0_EL1). Yes, there is a number of comments
to document that, but the code should aim to be be self-documenting. How
about IDRAZ() for those we want to "hide", and IDRSV for encodings that
are not allocated yet? It would look like this:

	IDREG(ID_PFR0),		IDREG(ID_PFR1),		IDREG(ID_DFR0),
	IDRAZ(ID_AFR0),		IDREG(ID_MMFR0),	IDREG(ID_MMFR1),
	IDREG(ID_MMFR2),	IDREG(ID_MMFR3),	IDREG(ID_ISAR0),
	IDREG(ID_ISAR1),	IDREG(ID_ISAR2),	IDREG(ID_ISAR3),
	IDREG(ID_ISAR4),	IDREG(ID_ISAR5),	IDREG(ID_MMFR4),
	IDRSV(2,7),		IDREG(MVFR0),		IDREG(MVFR1),
	IDREG(MVFR2),		IDRSV(3,3),		IDRSV(3,4),	
	IDRSV(3,5),		IDRSV(3,6),		IDRSV(3,7),

Yes, only 3 a line. Lines are cheap. And yes, they also have similar
names, but I said #bikeshed.

> +
> +	/* AArch64 ID registers */
> +	ID(AA64PFR0),	ID(AA64PFR1),	_ID_RAZ(4,2),	_ID_RAZ(4,3),
> +	_ID_RAZ(4,4),	_ID_RAZ(4,5),	_ID_RAZ(4,6),	_ID_RAZ(4,7),
> +	ID(AA64DFR0),	ID(AA64DFR1),	_ID_RAZ(5,2),	_ID_RAZ(5,3),
> +	/* ID_AA64AFR0_EL1 and ID_AA64AFR0_EL1 not exposed to guests for now */
> +	_ID_RAZ(5,4),	_ID_RAZ(5,5),	_ID_RAZ(5,6),	_ID_RAZ(5,7),
> +	ID(AA64ISAR0),	ID(AA64ISAR1),	_ID_RAZ(6,2),	_ID_RAZ(6,3),
> +	_ID_RAZ(6,4),	_ID_RAZ(6,5),	_ID_RAZ(6,6),	_ID_RAZ(6,7),
> +	ID(AA64MMFR0),	ID(AA64MMFR1),	ID(AA64MMFR2),	_ID_RAZ(7,3),
> +	_ID_RAZ(7,4),	_ID_RAZ(7,5),	_ID_RAZ(7,6),	_ID_RAZ(7,7),
> +
>  	{ SYS_DESC(SYS_SCTLR_EL1), access_vm_reg, reset_val, SCTLR_EL1, 0x00C50078 },
>  	{ SYS_DESC(SYS_CPACR_EL1), NULL, reset_val, CPACR_EL1, 0 },
>  	{ SYS_DESC(SYS_TTBR0_EL1), access_vm_reg, reset_unknown, TTBR0_EL1 },
> @@ -1790,8 +1945,8 @@ static const struct sys_reg_desc *index_to_sys_reg_desc(struct kvm_vcpu *vcpu,
>  	if (!r)
>  		r = find_reg(&params, sys_reg_descs, ARRAY_SIZE(sys_reg_descs));
>  
> -	/* Not saved in the sys_reg array? */
> -	if (r && !r->reg)
> +	/* Not saved in the sys_reg array and not otherwise accessible? */
> +	if (r && !(r->reg || r->get_user))
>  		r = NULL;
>  
>  	return r;
> @@ -1815,20 +1970,6 @@ static const struct sys_reg_desc *index_to_sys_reg_desc(struct kvm_vcpu *vcpu,
>  FUNCTION_INVARIANT(midr_el1)
>  FUNCTION_INVARIANT(ctr_el0)
>  FUNCTION_INVARIANT(revidr_el1)
> -FUNCTION_INVARIANT(id_pfr0_el1)
> -FUNCTION_INVARIANT(id_pfr1_el1)
> -FUNCTION_INVARIANT(id_dfr0_el1)
> -FUNCTION_INVARIANT(id_afr0_el1)
> -FUNCTION_INVARIANT(id_mmfr0_el1)
> -FUNCTION_INVARIANT(id_mmfr1_el1)
> -FUNCTION_INVARIANT(id_mmfr2_el1)
> -FUNCTION_INVARIANT(id_mmfr3_el1)
> -FUNCTION_INVARIANT(id_isar0_el1)
> -FUNCTION_INVARIANT(id_isar1_el1)
> -FUNCTION_INVARIANT(id_isar2_el1)
> -FUNCTION_INVARIANT(id_isar3_el1)
> -FUNCTION_INVARIANT(id_isar4_el1)
> -FUNCTION_INVARIANT(id_isar5_el1)
>  FUNCTION_INVARIANT(clidr_el1)
>  FUNCTION_INVARIANT(aidr_el1)
>  
> @@ -1836,20 +1977,6 @@ FUNCTION_INVARIANT(aidr_el1)
>  static struct sys_reg_desc invariant_sys_regs[] = {
>  	{ SYS_DESC(SYS_MIDR_EL1), NULL, get_midr_el1 },
>  	{ SYS_DESC(SYS_REVIDR_EL1), NULL, get_revidr_el1 },
> -	{ SYS_DESC(SYS_ID_PFR0_EL1), NULL, get_id_pfr0_el1 },
> -	{ SYS_DESC(SYS_ID_PFR1_EL1), NULL, get_id_pfr1_el1 },
> -	{ SYS_DESC(SYS_ID_DFR0_EL1), NULL, get_id_dfr0_el1 },
> -	{ SYS_DESC(SYS_ID_AFR0_EL1), NULL, get_id_afr0_el1 },
> -	{ SYS_DESC(SYS_ID_MMFR0_EL1), NULL, get_id_mmfr0_el1 },
> -	{ SYS_DESC(SYS_ID_MMFR1_EL1), NULL, get_id_mmfr1_el1 },
> -	{ SYS_DESC(SYS_ID_MMFR2_EL1), NULL, get_id_mmfr2_el1 },
> -	{ SYS_DESC(SYS_ID_MMFR3_EL1), NULL, get_id_mmfr3_el1 },
> -	{ SYS_DESC(SYS_ID_ISAR0_EL1), NULL, get_id_isar0_el1 },
> -	{ SYS_DESC(SYS_ID_ISAR1_EL1), NULL, get_id_isar1_el1 },
> -	{ SYS_DESC(SYS_ID_ISAR2_EL1), NULL, get_id_isar2_el1 },
> -	{ SYS_DESC(SYS_ID_ISAR3_EL1), NULL, get_id_isar3_el1 },
> -	{ SYS_DESC(SYS_ID_ISAR4_EL1), NULL, get_id_isar4_el1 },
> -	{ SYS_DESC(SYS_ID_ISAR5_EL1), NULL, get_id_isar5_el1 },
>  	{ SYS_DESC(SYS_CLIDR_EL1), NULL, get_clidr_el1 },
>  	{ SYS_DESC(SYS_AIDR_EL1), NULL, get_aidr_el1 },
>  	{ SYS_DESC(SYS_CTR_EL0), NULL, get_ctr_el0 },
> @@ -2079,12 +2206,31 @@ static bool copy_reg_to_user(const struct sys_reg_desc *reg, u64 __user **uind)
>  	return true;
>  }
>  
> +static int walk_one_sys_reg(const struct sys_reg_desc *rd,
> +			    u64 __user **uind,
> +			    unsigned int *total)
> +{
> +	/*
> +	 * Ignore registers we trap but don't save,
> +	 * and for which no custom user accessor is provided.
> +	 */
> +	if (!(rd->reg || rd->get_user))
> +		return 0;
> +
> +	if (!copy_reg_to_user(rd, uind))
> +		return -EFAULT;
> +
> +	(*total)++;
> +	return 0;
> +}
> +
>  /* Assumed ordered tables, see kvm_sys_reg_table_init. */
>  static int walk_sys_regs(struct kvm_vcpu *vcpu, u64 __user *uind)
>  {
>  	const struct sys_reg_desc *i1, *i2, *end1, *end2;
>  	unsigned int total = 0;
>  	size_t num;
> +	int err;
>  
>  	/* We check for duplicates here, to allow arch-specific overrides. */
>  	i1 = get_target_table(vcpu->arch.target, true, &num);
> @@ -2098,21 +2244,9 @@ static int walk_sys_regs(struct kvm_vcpu *vcpu, u64 __user *uind)
>  	while (i1 || i2) {
>  		int cmp = cmp_sys_reg(i1, i2);
>  		/* target-specific overrides generic entry. */
> -		if (cmp <= 0) {
> -			/* Ignore registers we trap but don't save. */
> -			if (i1->reg) {
> -				if (!copy_reg_to_user(i1, &uind))
> -					return -EFAULT;
> -				total++;
> -			}
> -		} else {
> -			/* Ignore registers we trap but don't save. */
> -			if (i2->reg) {
> -				if (!copy_reg_to_user(i2, &uind))
> -					return -EFAULT;
> -				total++;
> -			}
> -		}
> +		err = walk_one_sys_reg(cmp <= 0 ? i1 : i2, &uind, &total);

Please move this ternary operator out of the function parameters, as
that code is hairy enough. Or use the new function twice within the "if"
statement.

> +		if (err)
> +			return err;
>  
>  		if (cmp <= 0 && ++i1 == end1)
>  			i1 = NULL;
> 

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 23/27] arm64/sve: KVM: Hide SVE from CPU features exposed to guests

[Marc Zyngier]

On 16/08/17 11:54, Dave Martin wrote:
> On Tue, Aug 15, 2017 at 05:37:55PM +0100, Marc Zyngier wrote:
>> On 09/08/17 13:05, Dave Martin wrote:
>>> KVM guests cannot currently use SVE, because SVE is always
>>> configured to trap to EL2.
>>>
>>> However, a guest that sees SVE reported as present in
>>> ID_AA64PFR0_EL1 may legitimately expect that SVE works and try to
>>> use it.  Instead of working, the guest will receive an injected
>>> undef exception, which may cause the guest to oops or go into a
>>> spin.
>>>
>>> To avoid misleading the guest into believing that SVE will work,
>>> this patch masks out the SVE field from ID_AA64PFR0_EL1 when a
>>> guest attempts to read this register.  No support is explicitly
>>> added for ID_AA64ZFR0_EL1 either, so that is still emulated as
>>> reading as zero, which is consistent with SVE not being
>>> implemented.
>>>
>>> This is a temporary measure, and will be removed in a later series
>>> when full KVM support for SVE is implemented.
>>>
>>> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
>>> ---
>>>  arch/arm64/kvm/sys_regs.c | 14 +++++++++++++-
>>>  1 file changed, 13 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
>>> index 6583dd7..9e8c54e 100644
>>> --- a/arch/arm64/kvm/sys_regs.c
>>> +++ b/arch/arm64/kvm/sys_regs.c
>>> @@ -897,8 +897,20 @@ static u64 read_id_reg(struct sys_reg_desc const *r, bool raz)
>>>  {
>>>  	u32 id = sys_reg((u32)r->Op0, (u32)r->Op1,
>>>  			 (u32)r->CRn, (u32)r->CRm, (u32)r->Op2);
>>> +	u64 val = raz ? 0 : read_sanitised_ftr_reg(id);
>>>  
>>> -	return raz ? 0 : read_sanitised_ftr_reg(id);
>>> +	if (id == SYS_ID_AA64PFR0_EL1) {
>>> +		static bool printed;
>>> +
>>> +		if ((val & (0xfUL << ID_AA64PFR0_SVE_SHIFT)) && !printed) {
>>> +			kvm_info("SVE unsupported for guests, suppressing\n");
>>> +			printed = true;
>>> +		}
>>
>> Ideally, this should be a vcpu_unimpl_once(). But:
>> - it doesn't exist
>> - vcpu_unimpl looks hopelessly x86 specific
> 
> Yeah, I looked for an appropriate function and didn't find one ... and
> writing one just for this seemed overkill.
> 
>> How about turning it into a pr_err_once() instead?
> 
> Can do, though should it be an err?
> 
> No error has occurred here, rather I want people who discover that their
> guest mysteriously doesn't see SVE gets a clue about why.
An "err" is a good way to make it appear on the console. If you want to
make it noticed, that's the way.

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 21/27] arm64/sve: KVM: Prevent guests from using SVE

[Marc Zyngier]

On 16/08/17 11:50, Dave Martin wrote:
> On Tue, Aug 15, 2017 at 05:33:15PM +0100, Marc Zyngier wrote:
>> On 09/08/17 13:05, Dave Martin wrote:
>>> Until KVM has full SVE support, guests must not be allowed to
>>> execute SVE instructions.
>>>
>>> This patch enables the necessary traps, and also ensures that the
>>> traps are disabled again on exit from the guest so that the host
>>> can still use SVE if it wants to.
>>>
>>> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
>>> ---
>>>  arch/arm64/include/asm/kvm_arm.h | 3 ++-
>>>  arch/arm64/kvm/hyp/switch.c      | 6 +++---
>>>  2 files changed, 5 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
>>> index dbf0537..8a19651 100644
>>> --- a/arch/arm64/include/asm/kvm_arm.h
>>> +++ b/arch/arm64/include/asm/kvm_arm.h
>>> @@ -186,7 +186,7 @@
>>>  #define CPTR_EL2_TTA	(1 << 20)
>>>  #define CPTR_EL2_TFP	(1 << CPTR_EL2_TFP_SHIFT)
>>>  #define CPTR_EL2_TZ	(1 << 8)
>>> -#define CPTR_EL2_DEFAULT	0x000033ff
>>> +#define CPTR_EL2_DEFAULT	(0x000033ff & ~CPTR_EL2_TZ)
>>
>> I must say I'm not overly fond of this construct. I'd rather introduce a
>> RES1 field that matches the v8.2 description, instead of this ugly
>> constant and something that clears it.
> 
> Sorry, I don't get your meaning here.  v8.2 neither immediately predates
> or postdates SVE.  

The ARMv8 ARM (DDI406B_a, D7.2.19) says otherwise. This bit is only
defined as having a possibility of being 0 on an v8.2 implementation if
SVE is implemented.

> What are you propsing?

What I'm proposing is:

#define CPTR_EL2_RES1		0x32ff
#define CPTR_EL2_DEFAULT	CPTR_EL2_RES1

and none of that pointless constant clearing.

> I guess we could just write 0x000032ff now that the only meaning the
> architecture can ever assign to bit 8 is known.
Exactly.

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 23/27] arm64/sve: KVM: Hide SVE from CPU features exposed to guests

[Dave Martin]

On Wed, Aug 16, 2017 at 12:10:53PM +0100, Marc Zyngier wrote:
> On 16/08/17 11:54, Dave Martin wrote:
> > On Tue, Aug 15, 2017 at 05:37:55PM +0100, Marc Zyngier wrote:
> >> On 09/08/17 13:05, Dave Martin wrote:

[...]

> >>> +	if (id == SYS_ID_AA64PFR0_EL1) {
> >>> +		static bool printed;
> >>> +
> >>> +		if ((val & (0xfUL << ID_AA64PFR0_SVE_SHIFT)) && !printed) {
> >>> +			kvm_info("SVE unsupported for guests, suppressing\n");
> >>> +			printed = true;
> >>> +		}
> >>
> >> Ideally, this should be a vcpu_unimpl_once(). But:
> >> - it doesn't exist
> >> - vcpu_unimpl looks hopelessly x86 specific
> > 
> > Yeah, I looked for an appropriate function and didn't find one ... and
> > writing one just for this seemed overkill.
> > 
> >> How about turning it into a pr_err_once() instead?
> > 
> > Can do, though should it be an err?
> > 
> > No error has occurred here, rather I want people who discover that their
> > guest mysteriously doesn't see SVE gets a clue about why.
> An "err" is a good way to make it appear on the console. If you want to
> make it noticed, that's the way.

OK, I was concerned that overuse of err might be frowned upon.

If this is the preferred way to signal such things, I'm happy to that.

Cheers
---Dave

Re: [PATCH 21/27] arm64/sve: KVM: Prevent guests from using SVE

[Marc Zyngier]

On 16/08/17 12:20, Marc Zyngier wrote:
> On 16/08/17 11:50, Dave Martin wrote:
>> On Tue, Aug 15, 2017 at 05:33:15PM +0100, Marc Zyngier wrote:
>>> On 09/08/17 13:05, Dave Martin wrote:
>>>> Until KVM has full SVE support, guests must not be allowed to
>>>> execute SVE instructions.
>>>>
>>>> This patch enables the necessary traps, and also ensures that the
>>>> traps are disabled again on exit from the guest so that the host
>>>> can still use SVE if it wants to.
>>>>
>>>> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
>>>> ---
>>>>  arch/arm64/include/asm/kvm_arm.h | 3 ++-
>>>>  arch/arm64/kvm/hyp/switch.c      | 6 +++---
>>>>  2 files changed, 5 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
>>>> index dbf0537..8a19651 100644
>>>> --- a/arch/arm64/include/asm/kvm_arm.h
>>>> +++ b/arch/arm64/include/asm/kvm_arm.h
>>>> @@ -186,7 +186,7 @@
>>>>  #define CPTR_EL2_TTA	(1 << 20)
>>>>  #define CPTR_EL2_TFP	(1 << CPTR_EL2_TFP_SHIFT)
>>>>  #define CPTR_EL2_TZ	(1 << 8)
>>>> -#define CPTR_EL2_DEFAULT	0x000033ff
>>>> +#define CPTR_EL2_DEFAULT	(0x000033ff & ~CPTR_EL2_TZ)
>>>
>>> I must say I'm not overly fond of this construct. I'd rather introduce a
>>> RES1 field that matches the v8.2 description, instead of this ugly
>>> constant and something that clears it.
>>
>> Sorry, I don't get your meaning here.  v8.2 neither immediately predates
>> or postdates SVE.  
> 
> The ARMv8 ARM (DDI406B_a, D7.2.19) says otherwise. This bit is only

Of course, the days of the ARMv7 ARM are still haunting me. This should
read DDI487B_a.

Apologies for the confusion.

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 21/27] arm64/sve: KVM: Prevent guests from using SVE

[Dave Martin]

On Wed, Aug 16, 2017 at 12:20:41PM +0100, Marc Zyngier wrote:
> On 16/08/17 11:50, Dave Martin wrote:
> > On Tue, Aug 15, 2017 at 05:33:15PM +0100, Marc Zyngier wrote:
> >> On 09/08/17 13:05, Dave Martin wrote:
> >>> Until KVM has full SVE support, guests must not be allowed to
> >>> execute SVE instructions.
> >>>
> >>> This patch enables the necessary traps, and also ensures that the
> >>> traps are disabled again on exit from the guest so that the host
> >>> can still use SVE if it wants to.
> >>>
> >>> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> >>> ---
> >>>  arch/arm64/include/asm/kvm_arm.h | 3 ++-
> >>>  arch/arm64/kvm/hyp/switch.c      | 6 +++---
> >>>  2 files changed, 5 insertions(+), 4 deletions(-)
> >>>
> >>> diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
> >>> index dbf0537..8a19651 100644
> >>> --- a/arch/arm64/include/asm/kvm_arm.h
> >>> +++ b/arch/arm64/include/asm/kvm_arm.h
> >>> @@ -186,7 +186,7 @@
> >>>  #define CPTR_EL2_TTA	(1 << 20)
> >>>  #define CPTR_EL2_TFP	(1 << CPTR_EL2_TFP_SHIFT)
> >>>  #define CPTR_EL2_TZ	(1 << 8)
> >>> -#define CPTR_EL2_DEFAULT	0x000033ff
> >>> +#define CPTR_EL2_DEFAULT	(0x000033ff & ~CPTR_EL2_TZ)
> >>
> >> I must say I'm not overly fond of this construct. I'd rather introduce a
> >> RES1 field that matches the v8.2 description, instead of this ugly
> >> constant and something that clears it.
> > 
> > Sorry, I don't get your meaning here.  v8.2 neither immediately predates
> > or postdates SVE.  
> 
> The ARMv8 ARM (DDI406B_a, D7.2.19) says otherwise. This bit is only
> defined as having a possibility of being 0 on an v8.2 implementation if
> SVE is implemented.

Right.  I was confused by the v8.2 reference, since if this weren't true
for v8.0 of the architecture, we couldn't simply change a compile time
constant here.  In fact, there's a compatible retroactive change to all
arch versions >= v8.0.

> > What are you propsing?
> 
> What I'm proposing is:
> 
> #define CPTR_EL2_RES1		0x32ff
> #define CPTR_EL2_DEFAULT	CPTR_EL2_RES1
> 
> and none of that pointless constant clearing.
> 
> > I guess we could just write 0x000032ff now that the only meaning the
> > architecture can ever assign to bit 8 is known.
> Exactly.

OK, good -- I'll change that.

I was trying to avoid magic numberness, but it's a bit futile when
talking about RESx bits.

Cheers
---Dave

Re: [PATCH 15/27] arm64/sve: Probe SVE capabilities and usable vector lengths

[Suzuki K Poulose]

On 09/08/17 13:05, Dave Martin wrote:
> [This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]
>
> This patch uses the cpufeatures framework to determine common SVE
> capabilities and vector lengths, and configures the runtime SVE
> support code appropriately.
>
> ZCR_ELx is not really a feature register, but it is convenient to
> use it as a template for recording the maximum vector length
> supported by a CPU, using the LEN field.  This field is similar to
> a feature field in that it is a contiguous bitfield for which we
> want to determine the minimum system-wide value.  This patch adds
> ZCR as a pseudo-register in cpuinfo/cpufeatures, with appropriate
> custom code to populate it.  Finding the minimum supported value of
> the LEN field is left to the cpufeatures framework in the usual
> way.
>
> The meaning of ID_AA64ZFR0_EL1 is not architecturally defined yet,
> so for now we just require it to be zero.
>
> Note that much of this code is dormant and SVE still won't be used
> yet, since system_supports_sve() remains hardwired to false.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>

Dave,

The cpufeature bits look good to me, with one minor comment.


> diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> index bce95de..955c873 100644
> --- a/arch/arm64/kernel/fpsimd.c
> +++ b/arch/arm64/kernel/fpsimd.c

...

> +void __init sve_setup(void)
> +{
> +       u64 zcr;
> +       unsigned int max_vl;
> +
> +       if (!system_supports_sve())
> +               return;
> +
> +       /*
> +        * The architecture mandates 128-bit vectors be supported, and
> +        * the code assumes elsewhere that sve_vq_map is non-empty:
> +        */
> +       BUG_ON(!test_bit(vq_to_bit(1), sve_vq_map));
> +
> +       sve_vq_map_finalised = true;

We have something local in cpufeature.c, sys_caps_initialised. May be we could
reuse it here ? With or without that change, FWIW.

Acked-by: Suzuki K Poulose <suzuki.poulose@arm.com>

Re: [PATCH 24/27] arm64/sve: Detect SVE and activate runtime support

[Suzuki K Poulose]

On 09/08/17 13:05, Dave Martin wrote:
> [This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]
>
> This patch enables detection of hardware SVE support via the
> cpufeatures framework, and reports its presence to the kernel and
> userspace via the new ARM64_SVE cpucap and HWCAP_SVE hwcap
> respectively.
>
> Userspace can also detect SVE using ID_AA64PFR0_EL1, using the
> cpufeatures MRS emulation.
>
> When running on hardware that supports SVE, this enables runtime
> kernel support for SVE, and allows user tasks to execute SVE
> instructions and make of the of the SVE-specific user/kernel
> interface extensions implemented by this series.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>


> ---
>  arch/arm64/include/asm/cpucaps.h    |  3 ++-
>  arch/arm64/include/asm/cpufeature.h |  3 ++-
>  arch/arm64/include/uapi/asm/hwcap.h |  1 +
>  arch/arm64/kernel/cpufeature.c      | 16 ++++++++++++++++
>  arch/arm64/kernel/cpuinfo.c         |  1 +
>  5 files changed, 22 insertions(+), 2 deletions(-)
>

...

>
> @@ -944,6 +957,9 @@ static const struct arm64_cpu_capabilities arm64_elf_hwcaps[] = {
>         HWCAP_CAP(SYS_ID_AA64ISAR1_EL1, ID_AA64ISAR1_JSCVT_SHIFT, FTR_UNSIGNED, 1, CAP_HWCAP, HWCAP_JSCVT),
>         HWCAP_CAP(SYS_ID_AA64ISAR1_EL1, ID_AA64ISAR1_FCMA_SHIFT, FTR_UNSIGNED, 1, CAP_HWCAP, HWCAP_FCMA),
>         HWCAP_CAP(SYS_ID_AA64ISAR1_EL1, ID_AA64ISAR1_LRCPC_SHIFT, FTR_UNSIGNED, 1, CAP_HWCAP, HWCAP_LRCPC),
> +#ifdef CONFIG_ARM64_SVE
> +       HWCAP_CAP(SYS_ID_AA64PFR0_EL1, ID_AA64PFR0_SVE_SHIFT, FTR_UNSIGNED, ID_AA64PFR0_SVE, CAP_HWCAP, HWCAP_SVE),
> +#endif
>         {},
>  };
>
> diff --git a/arch/arm64/kernel/cpuinfo.c b/arch/arm64/kernel/cpuinfo.c
> index 0e087d7..943aad0 100644
> --- a/arch/arm64/kernel/cpuinfo.c
> +++ b/arch/arm64/kernel/cpuinfo.c
> @@ -69,6 +69,7 @@ static const char *const hwcap_str[] = {
>         "jscvt",
>         "fcma",
>         "lrcpc",
> +       "sve",
>         NULL
>  };
>


Dave,

Could you please update the Documentation for the exposed CPU feature
registers to include the SVE field ? Rest looks good to me. With that
change,

Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>

Re: [PATCH 02/27] arm64: KVM: Hide unsupported AArch64 CPU features from guests

[Dave Martin]

On Wed, Aug 16, 2017 at 12:10:38PM +0100, Marc Zyngier wrote:
> On 09/08/17 13:05, Dave Martin wrote:
> > Currently, a guest kernel sees the true CPU feature registers
> > (ID_*_EL1) when it reads them using MRS instructions.  This means
> > that the guest will observe features that are present in the
> > hardware but the host doesn't understand or doesn't provide support
> > for.  A guest may legimitately try to use such a feature as per the
> > architecture, but use of the feature may trap instead of working
> > normally, triggering undef injection into the guest.
> > 
> > This is not a problem for the host, but the guest may go wrong when
> > running on newer hardware than the host knows about.
> > 
> > This patch hides from guest VMs any AArch64-specific CPU features
> > that the host doesn't support, by exposing to the guest the
> > sanitised versions of the registers computed by the cpufeatures
> > framework, instead of the true hardware registers.  To achieve
> > this, HCR_EL2.TID3 is now set for AArch64 guests, and emulation
> > code is added to KVM to report the sanitised versions of the
> > affected registers in response to MRS and register reads from
> > userspace.
> > 
> > The affected registers are removed from invariant_sys_regs[] (since
> > the invariant_sys_regs handling is no longer quite correct for
> > them) and added to sys_reg_desgs[], with appropriate access(),
> > get_user() and set_user() methods.  No runtime vcpu storage is
> > allocated for the registers: instead, they are read on demand from
> > the cpufeatures framework.  This may need modification in the
> > future if there is a need for userspace to customise the features
> > visible to the guest.
> > 
> > Attempts by userspace to write the registers are handled similarly
> > to the current invariant_sys_regs handling: writes are permitted,
> > but only if they don't attempt to change the value.  This is
> > sufficient to support VM snapshot/restore from userspace.
> > 
> > Because of the additional registers, restoring a VM on an older
> > kernel may not work unless userspace knows how to handle the extra
> > VM registers exposed to the KVM user ABI by this patch.
> > 
> > Under the principle of least damage, this patch makes no attempt to
> > handle any of the other registers currently in
> > invariant_sys_regs[], or to emulate registers for AArch32: however,
> > these could be handled in a similar way in future, as necessary.
> > 
> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> > ---
> >  arch/arm64/kvm/hyp/switch.c |   6 ++
> >  arch/arm64/kvm/sys_regs.c   | 224 +++++++++++++++++++++++++++++++++++---------
> >  2 files changed, 185 insertions(+), 45 deletions(-)
> > 

[...]

> > diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
> > index 2e070d3..6583dd7 100644
> > --- a/arch/arm64/kvm/sys_regs.c
> > +++ b/arch/arm64/kvm/sys_regs.c
> > @@ -892,6 +892,135 @@ static bool access_cntp_cval(struct kvm_vcpu *vcpu,
> >  	return true;
> >  }
> >  
> > +/* Read a sanitised cpufeature ID register by sys_reg_desc */
> > +static u64 read_id_reg(struct sys_reg_desc const *r, bool raz)
> > +{
> > +	u32 id = sys_reg((u32)r->Op0, (u32)r->Op1,
> > +			 (u32)r->CRn, (u32)r->CRm, (u32)r->Op2);
> > +
> > +	return raz ? 0 : read_sanitised_ftr_reg(id);
> > +}
> > +
> > +/* cpufeature ID register access trap handlers */
> > +
> > +static bool __access_id_reg(struct kvm_vcpu *vcpu,
> > +			    struct sys_reg_params *p,
> > +			    const struct sys_reg_desc const *r,
> > +			    bool raz)
> > +{
> > +	if (p->is_write) {
> > +		kvm_inject_undefined(vcpu);
> > +		return false;
> > +	}
> 
> I don't think this is supposed to happen (should have UNDEF-ed at EL1).
> You can call write_to_read_only() in that case, which will spit out a
> warning and inject the exception.

I'll check this -- sounds about right.

If is should never happen, should I just delete that code or BUG()?  I
notice a BUG_ON() for a similar situation in access_vm_reg() for example.

Or do we not quite trust hardware not to get this wrong?
(It feels like the kind of thing that could slip through validation
and/or would be considered not worth a respin, but it seems wrong to
work around a theoretical hardware bug before it's confirmed to exist,
unless we think for some reason that it's really likely.)

> > +
> > +	p->regval = read_id_reg(r, raz);
> > +	return true;
> > +}

[...]

> > @@ -944,6 +1073,32 @@ static const struct sys_reg_desc sys_reg_descs[] = {
> >  	{ SYS_DESC(SYS_DBGVCR32_EL2), NULL, reset_val, DBGVCR32_EL2, 0 },
> >  
> >  	{ SYS_DESC(SYS_MPIDR_EL1), NULL, reset_mpidr, MPIDR_EL1 },
> > +
> > +	/*
> > +	 * All non-RAZ feature registers listed here must also be
> > +	 * present in arm64_ftr_regs[].
> > +	 */
> > +
> > +	/* AArch64 mappings of the AArch32 ID registers */
> > +	/* ID_AFR0_EL1 not exposed to guests for now */
> > +	ID(PFR0),	ID(PFR1),	ID(DFR0),	_ID_RAZ(1,3),
> > +	ID(MMFR0),	ID(MMFR1),	ID(MMFR2),	ID(MMFR3),
> > +	ID(ISAR0),	ID(ISAR1),	ID(ISAR2),	ID(ISAR3),
> > +	ID(ISAR4),	ID(ISAR5),	ID(MMFR4),	_ID_RAZ(2,7),
> > +	_ID(MVFR0),	_ID(MVFR1),	_ID(MVFR2),	_ID_RAZ(3,3),
> > +	_ID_RAZ(3,4),	_ID_RAZ(3,5),	_ID_RAZ(3,6),	_ID_RAZ(3,7),
> 
> #bikeshed:
> 
> OK, this is giving me a headache. Too many variants with similar names.
> ID and _ID
> I'm also slightly perplexed with the amalgamation of RAZ because the
> register is not defined yet in the architecture, and RAZ because we
> don't expose it (like ID_AFR0_EL1). Yes, there is a number of comments

This "raz" overloading already seems present in other places, such as the
cpufeatures code.  (Which is not necessarily a good reason for adding
more of it...)

> to document that, but the code should aim to be be self-documenting. How
> about IDRAZ() for those we want to "hide", and IDRSV for encodings that
> are not allocated yet? It would look like this:
> 
> 	IDREG(ID_PFR0),		IDREG(ID_PFR1),		IDREG(ID_DFR0),
> 	IDRAZ(ID_AFR0),		IDREG(ID_MMFR0),	IDREG(ID_MMFR1),
> 	IDREG(ID_MMFR2),	IDREG(ID_MMFR3),	IDREG(ID_ISAR0),
> 	IDREG(ID_ISAR1),	IDREG(ID_ISAR2),	IDREG(ID_ISAR3),
> 	IDREG(ID_ISAR4),	IDREG(ID_ISAR5),	IDREG(ID_MMFR4),
> 	IDRSV(2,7),		IDREG(MVFR0),		IDREG(MVFR1),
> 	IDREG(MVFR2),		IDRSV(3,3),		IDRSV(3,4),	
> 	IDRSV(3,5),		IDRSV(3,6),		IDRSV(3,7),
> 
> Yes, only 3 a line. Lines are cheap. And yes, they also have similar
> names, but I said #bikeshed.

So, point taken, but the main reason for making this a table was to make
it easy to see by eye how the entries map to the encoding while hacking
this up, which helped me to make sure no entries were missed or in the
wrong place etc.

With 3 entries per line that visual map is lost, and with 2 entries per
line it's debatable whether it's worth having multiple entries per line
at all.

So now that the table exists maybe we should just have one entry per
line like everything else -- it really depends on which option you think
is best for ongoing maintenance.


Having one per line allows much less cryptic names, allowing the
temptingly short but ambiguous "RAZ" to be avoided:

	ID_SANITISED(ID_ISAR5),
	ID_RAZ_FOR_GUEST(ID_AFR0),
	ID_UNALLOCATED(crm, op2)

With a whole line and different lengths, it's easier to pick out
the different cases by eye, so they don't all look like IDRXX (and are a
more tasteful colour perhaps).

Blank lines and/or comments can split the list into sensible blocks for
readability if needed.

If you're happy with naming along those broad lines then I'm happy to
see what it looks like.

> > +
> > +	/* AArch64 ID registers */
> > +	ID(AA64PFR0),	ID(AA64PFR1),	_ID_RAZ(4,2),	_ID_RAZ(4,3),
> > +	_ID_RAZ(4,4),	_ID_RAZ(4,5),	_ID_RAZ(4,6),	_ID_RAZ(4,7),
> > +	ID(AA64DFR0),	ID(AA64DFR1),	_ID_RAZ(5,2),	_ID_RAZ(5,3),
> > +	/* ID_AA64AFR0_EL1 and ID_AA64AFR0_EL1 not exposed to guests for now */

There are no sysreg definitions for IA_AA64AFR{0,1}_EL1 yet.

If we want to macroise those rather than just commenting, I guess
they'll need adding in sysreg.h.  I'd prefer not to imply these are
"unallocated" or similar when the architecture does define them.

Can I take it there's no problem with zombie entries in sysreg.h so long
as they're at least referenced somewhere?  (Arguably they wouldn't be
zombies then, but hopefully you see what I mean.)

[...]

> > +static int walk_one_sys_reg(const struct sys_reg_desc *rd,
> > +			    u64 __user **uind,
> > +			    unsigned int *total)
> > +{
> > +	/*
> > +	 * Ignore registers we trap but don't save,
> > +	 * and for which no custom user accessor is provided.
> > +	 */
> > +	if (!(rd->reg || rd->get_user))
> > +		return 0;
> > +
> > +	if (!copy_reg_to_user(rd, uind))
> > +		return -EFAULT;
> > +
> > +	(*total)++;
> > +	return 0;
> > +}
> > +
> >  /* Assumed ordered tables, see kvm_sys_reg_table_init. */
> >  static int walk_sys_regs(struct kvm_vcpu *vcpu, u64 __user *uind)
> >  {
> >  	const struct sys_reg_desc *i1, *i2, *end1, *end2;
> >  	unsigned int total = 0;
> >  	size_t num;
> > +	int err;
> >  
> >  	/* We check for duplicates here, to allow arch-specific overrides. */
> >  	i1 = get_target_table(vcpu->arch.target, true, &num);
> > @@ -2098,21 +2244,9 @@ static int walk_sys_regs(struct kvm_vcpu *vcpu, u64 __user *uind)
> >  	while (i1 || i2) {
> >  		int cmp = cmp_sys_reg(i1, i2);
> >  		/* target-specific overrides generic entry. */
> > -		if (cmp <= 0) {
> > -			/* Ignore registers we trap but don't save. */
> > -			if (i1->reg) {
> > -				if (!copy_reg_to_user(i1, &uind))
> > -					return -EFAULT;
> > -				total++;
> > -			}
> > -		} else {
> > -			/* Ignore registers we trap but don't save. */
> > -			if (i2->reg) {
> > -				if (!copy_reg_to_user(i2, &uind))
> > -					return -EFAULT;
> > -				total++;
> > -			}
> > -		}
> > +		err = walk_one_sys_reg(cmp <= 0 ? i1 : i2, &uind, &total);
> 
> Please move this ternary operator out of the function parameters, as
> that code is hairy enough. Or use the new function twice within the "if"
> statement.

Can do.  Making this a one-liner doesn't buy us an awful lot.

Cheers
---Dave

Re: [PATCH 02/27] arm64: KVM: Hide unsupported AArch64 CPU features from guests

[Marc Zyngier]

On 16/08/17 21:32, Dave Martin wrote:
> On Wed, Aug 16, 2017 at 12:10:38PM +0100, Marc Zyngier wrote:
>> On 09/08/17 13:05, Dave Martin wrote:
>>> Currently, a guest kernel sees the true CPU feature registers
>>> (ID_*_EL1) when it reads them using MRS instructions.  This means
>>> that the guest will observe features that are present in the
>>> hardware but the host doesn't understand or doesn't provide support
>>> for.  A guest may legimitately try to use such a feature as per the
>>> architecture, but use of the feature may trap instead of working
>>> normally, triggering undef injection into the guest.
>>>
>>> This is not a problem for the host, but the guest may go wrong when
>>> running on newer hardware than the host knows about.
>>>
>>> This patch hides from guest VMs any AArch64-specific CPU features
>>> that the host doesn't support, by exposing to the guest the
>>> sanitised versions of the registers computed by the cpufeatures
>>> framework, instead of the true hardware registers.  To achieve
>>> this, HCR_EL2.TID3 is now set for AArch64 guests, and emulation
>>> code is added to KVM to report the sanitised versions of the
>>> affected registers in response to MRS and register reads from
>>> userspace.
>>>
>>> The affected registers are removed from invariant_sys_regs[] (since
>>> the invariant_sys_regs handling is no longer quite correct for
>>> them) and added to sys_reg_desgs[], with appropriate access(),
>>> get_user() and set_user() methods.  No runtime vcpu storage is
>>> allocated for the registers: instead, they are read on demand from
>>> the cpufeatures framework.  This may need modification in the
>>> future if there is a need for userspace to customise the features
>>> visible to the guest.
>>>
>>> Attempts by userspace to write the registers are handled similarly
>>> to the current invariant_sys_regs handling: writes are permitted,
>>> but only if they don't attempt to change the value.  This is
>>> sufficient to support VM snapshot/restore from userspace.
>>>
>>> Because of the additional registers, restoring a VM on an older
>>> kernel may not work unless userspace knows how to handle the extra
>>> VM registers exposed to the KVM user ABI by this patch.
>>>
>>> Under the principle of least damage, this patch makes no attempt to
>>> handle any of the other registers currently in
>>> invariant_sys_regs[], or to emulate registers for AArch32: however,
>>> these could be handled in a similar way in future, as necessary.
>>>
>>> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
>>> ---
>>>  arch/arm64/kvm/hyp/switch.c |   6 ++
>>>  arch/arm64/kvm/sys_regs.c   | 224 +++++++++++++++++++++++++++++++++++---------
>>>  2 files changed, 185 insertions(+), 45 deletions(-)
>>>
> 
> [...]
> 
>>> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
>>> index 2e070d3..6583dd7 100644
>>> --- a/arch/arm64/kvm/sys_regs.c
>>> +++ b/arch/arm64/kvm/sys_regs.c
>>> @@ -892,6 +892,135 @@ static bool access_cntp_cval(struct kvm_vcpu *vcpu,
>>>  	return true;
>>>  }
>>>  
>>> +/* Read a sanitised cpufeature ID register by sys_reg_desc */
>>> +static u64 read_id_reg(struct sys_reg_desc const *r, bool raz)
>>> +{
>>> +	u32 id = sys_reg((u32)r->Op0, (u32)r->Op1,
>>> +			 (u32)r->CRn, (u32)r->CRm, (u32)r->Op2);
>>> +
>>> +	return raz ? 0 : read_sanitised_ftr_reg(id);
>>> +}
>>> +
>>> +/* cpufeature ID register access trap handlers */
>>> +
>>> +static bool __access_id_reg(struct kvm_vcpu *vcpu,
>>> +			    struct sys_reg_params *p,
>>> +			    const struct sys_reg_desc const *r,
>>> +			    bool raz)
>>> +{
>>> +	if (p->is_write) {
>>> +		kvm_inject_undefined(vcpu);
>>> +		return false;
>>> +	}
>>
>> I don't think this is supposed to happen (should have UNDEF-ed at EL1).
>> You can call write_to_read_only() in that case, which will spit out a
>> warning and inject the exception.
> 
> I'll check this -- sounds about right.
> 
> If is should never happen, should I just delete that code or BUG()?  I
> notice a BUG_ON() for a similar situation in access_vm_reg() for example.
> 
> Or do we not quite trust hardware not to get this wrong?
> (It feels like the kind of thing that could slip through validation
> and/or would be considered not worth a respin, but it seems wrong to
> work around a theoretical hardware bug before it's confirmed to exist,
> unless we think for some reason that it's really likely.)

That's the way we handle this for the rest of the accessors. We used to
have a BUG_ON(), but it is pretty silly to kill the whole system for
such a small deviation from the architecture. And maybe it is useless,
but it doesn't hurt either.

>>> +
>>> +	p->regval = read_id_reg(r, raz);
>>> +	return true;
>>> +}
> 
> [...]
> 
>>> @@ -944,6 +1073,32 @@ static const struct sys_reg_desc sys_reg_descs[] = {
>>>  	{ SYS_DESC(SYS_DBGVCR32_EL2), NULL, reset_val, DBGVCR32_EL2, 0 },
>>>  
>>>  	{ SYS_DESC(SYS_MPIDR_EL1), NULL, reset_mpidr, MPIDR_EL1 },
>>> +
>>> +	/*
>>> +	 * All non-RAZ feature registers listed here must also be
>>> +	 * present in arm64_ftr_regs[].
>>> +	 */
>>> +
>>> +	/* AArch64 mappings of the AArch32 ID registers */
>>> +	/* ID_AFR0_EL1 not exposed to guests for now */
>>> +	ID(PFR0),	ID(PFR1),	ID(DFR0),	_ID_RAZ(1,3),
>>> +	ID(MMFR0),	ID(MMFR1),	ID(MMFR2),	ID(MMFR3),
>>> +	ID(ISAR0),	ID(ISAR1),	ID(ISAR2),	ID(ISAR3),
>>> +	ID(ISAR4),	ID(ISAR5),	ID(MMFR4),	_ID_RAZ(2,7),
>>> +	_ID(MVFR0),	_ID(MVFR1),	_ID(MVFR2),	_ID_RAZ(3,3),
>>> +	_ID_RAZ(3,4),	_ID_RAZ(3,5),	_ID_RAZ(3,6),	_ID_RAZ(3,7),
>>
>> #bikeshed:
>>
>> OK, this is giving me a headache. Too many variants with similar names.
>> ID and _ID
>> I'm also slightly perplexed with the amalgamation of RAZ because the
>> register is not defined yet in the architecture, and RAZ because we
>> don't expose it (like ID_AFR0_EL1). Yes, there is a number of comments
> 
> This "raz" overloading already seems present in other places, such as the
> cpufeatures code.  (Which is not necessarily a good reason for adding
> more of it...)
> 
>> to document that, but the code should aim to be be self-documenting. How
>> about IDRAZ() for those we want to "hide", and IDRSV for encodings that
>> are not allocated yet? It would look like this:
>>
>> 	IDREG(ID_PFR0),		IDREG(ID_PFR1),		IDREG(ID_DFR0),
>> 	IDRAZ(ID_AFR0),		IDREG(ID_MMFR0),	IDREG(ID_MMFR1),
>> 	IDREG(ID_MMFR2),	IDREG(ID_MMFR3),	IDREG(ID_ISAR0),
>> 	IDREG(ID_ISAR1),	IDREG(ID_ISAR2),	IDREG(ID_ISAR3),
>> 	IDREG(ID_ISAR4),	IDREG(ID_ISAR5),	IDREG(ID_MMFR4),
>> 	IDRSV(2,7),		IDREG(MVFR0),		IDREG(MVFR1),
>> 	IDREG(MVFR2),		IDRSV(3,3),		IDRSV(3,4),	
>> 	IDRSV(3,5),		IDRSV(3,6),		IDRSV(3,7),
>>
>> Yes, only 3 a line. Lines are cheap. And yes, they also have similar
>> names, but I said #bikeshed.
> 
> So, point taken, but the main reason for making this a table was to make
> it easy to see by eye how the entries map to the encoding while hacking
> this up, which helped me to make sure no entries were missed or in the
> wrong place etc.
> 
> With 3 entries per line that visual map is lost, and with 2 entries per
> line it's debatable whether it's worth having multiple entries per line
> at all.

Let's be clear. I don't care at all about the number of entries per
line. I can widen my editor to 200 columns if I need to. If you think 4
is the way, keep it to 4.

My point is about the readability of both the macros and the
identifiers, and your initial proposal did seem to lack on both counts.

> So now that the table exists maybe we should just have one entry per
> line like everything else -- it really depends on which option you think
> is best for ongoing maintenance.
> 
> 
> Having one per line allows much less cryptic names, allowing the
> temptingly short but ambiguous "RAZ" to be avoided:
> 
> 	ID_SANITISED(ID_ISAR5),
> 	ID_RAZ_FOR_GUEST(ID_AFR0),
> 	ID_UNALLOCATED(crm, op2)
> 
> With a whole line and different lengths, it's easier to pick out
> the different cases by eye, so they don't all look like IDRXX (and are a
> more tasteful colour perhaps).
> 
> Blank lines and/or comments can split the list into sensible blocks for
> readability if needed.
> 
> If you're happy with naming along those broad lines then I'm happy to
> see what it looks like.

Sure. If you're happy with that, so am I.

>>> +
>>> +	/* AArch64 ID registers */
>>> +	ID(AA64PFR0),	ID(AA64PFR1),	_ID_RAZ(4,2),	_ID_RAZ(4,3),
>>> +	_ID_RAZ(4,4),	_ID_RAZ(4,5),	_ID_RAZ(4,6),	_ID_RAZ(4,7),
>>> +	ID(AA64DFR0),	ID(AA64DFR1),	_ID_RAZ(5,2),	_ID_RAZ(5,3),
>>> +	/* ID_AA64AFR0_EL1 and ID_AA64AFR0_EL1 not exposed to guests for now */
> 
> There are no sysreg definitions for IA_AA64AFR{0,1}_EL1 yet.
> 
> If we want to macroise those rather than just commenting, I guess
> they'll need adding in sysreg.h.  I'd prefer not to imply these are
> "unallocated" or similar when the architecture does define them.
> 
> Can I take it there's no problem with zombie entries in sysreg.h so long
> as they're at least referenced somewhere?  (Arguably they wouldn't be
> zombies then, but hopefully you see what I mean.)

That'd be the right thing to do. The register exists, and KVM handles it
by returning 0 when a guest reads it. So I'd argue that it *must* be
defined in sysreg.h, and given its full visibility in that table.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 02/27] arm64: KVM: Hide unsupported AArch64 CPU features from guests

[Dave Martin]

On Thu, Aug 17, 2017 at 09:45:51AM +0100, Marc Zyngier wrote:
> On 16/08/17 21:32, Dave Martin wrote:
> > On Wed, Aug 16, 2017 at 12:10:38PM +0100, Marc Zyngier wrote:
> >> On 09/08/17 13:05, Dave Martin wrote:
> >>> Currently, a guest kernel sees the true CPU feature registers
> >>> (ID_*_EL1) when it reads them using MRS instructions.  This means
> >>> that the guest will observe features that are present in the
> >>> hardware but the host doesn't understand or doesn't provide support
> >>> for.  A guest may legimitately try to use such a feature as per the
> >>> architecture, but use of the feature may trap instead of working
> >>> normally, triggering undef injection into the guest.
> >>>
> >>> This is not a problem for the host, but the guest may go wrong when
> >>> running on newer hardware than the host knows about.
> >>>
> >>> This patch hides from guest VMs any AArch64-specific CPU features
> >>> that the host doesn't support, by exposing to the guest the
> >>> sanitised versions of the registers computed by the cpufeatures
> >>> framework, instead of the true hardware registers.  To achieve
> >>> this, HCR_EL2.TID3 is now set for AArch64 guests, and emulation
> >>> code is added to KVM to report the sanitised versions of the
> >>> affected registers in response to MRS and register reads from
> >>> userspace.
> >>>
> >>> The affected registers are removed from invariant_sys_regs[] (since
> >>> the invariant_sys_regs handling is no longer quite correct for
> >>> them) and added to sys_reg_desgs[], with appropriate access(),
> >>> get_user() and set_user() methods.  No runtime vcpu storage is
> >>> allocated for the registers: instead, they are read on demand from
> >>> the cpufeatures framework.  This may need modification in the
> >>> future if there is a need for userspace to customise the features
> >>> visible to the guest.
> >>>
> >>> Attempts by userspace to write the registers are handled similarly
> >>> to the current invariant_sys_regs handling: writes are permitted,
> >>> but only if they don't attempt to change the value.  This is
> >>> sufficient to support VM snapshot/restore from userspace.
> >>>
> >>> Because of the additional registers, restoring a VM on an older
> >>> kernel may not work unless userspace knows how to handle the extra
> >>> VM registers exposed to the KVM user ABI by this patch.
> >>>
> >>> Under the principle of least damage, this patch makes no attempt to
> >>> handle any of the other registers currently in
> >>> invariant_sys_regs[], or to emulate registers for AArch32: however,
> >>> these could be handled in a similar way in future, as necessary.
> >>>
> >>> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> >>> ---
> >>>  arch/arm64/kvm/hyp/switch.c |   6 ++
> >>>  arch/arm64/kvm/sys_regs.c   | 224 +++++++++++++++++++++++++++++++++++---------
> >>>  2 files changed, 185 insertions(+), 45 deletions(-)

[...]

> >>> +static bool __access_id_reg(struct kvm_vcpu *vcpu,
> >>> +			    struct sys_reg_params *p,
> >>> +			    const struct sys_reg_desc const *r,
> >>> +			    bool raz)
> >>> +{
> >>> +	if (p->is_write) {
> >>> +		kvm_inject_undefined(vcpu);
> >>> +		return false;
> >>> +	}
> >>
> >> I don't think this is supposed to happen (should have UNDEF-ed at EL1).
> >> You can call write_to_read_only() in that case, which will spit out a
> >> warning and inject the exception.
> > 
> > I'll check this -- sounds about right.
> > 
> > If is should never happen, should I just delete that code or BUG()?  I
> > notice a BUG_ON() for a similar situation in access_vm_reg() for example.
> > 
> > Or do we not quite trust hardware not to get this wrong?
> > (It feels like the kind of thing that could slip through validation
> > and/or would be considered not worth a respin, but it seems wrong to
> > work around a theoretical hardware bug before it's confirmed to exist,
> > unless we think for some reason that it's really likely.)
> 
> That's the way we handle this for the rest of the accessors. We used to
> have a BUG_ON(), but it is pretty silly to kill the whole system for
> such a small deviation from the architecture. And maybe it is useless,
> but it doesn't hurt either.

OK, that makes sense -- I'll follow the precedent here and call
write_to_read_only() if this happens.

> >>> +
> >>> +	p->regval = read_id_reg(r, raz);
> >>> +	return true;
> >>> +}
> > 
> > [...]
> > 
> >>> @@ -944,6 +1073,32 @@ static const struct sys_reg_desc sys_reg_descs[] = {
> >>>  	{ SYS_DESC(SYS_DBGVCR32_EL2), NULL, reset_val, DBGVCR32_EL2, 0 },
> >>>  
> >>>  	{ SYS_DESC(SYS_MPIDR_EL1), NULL, reset_mpidr, MPIDR_EL1 },
> >>> +
> >>> +	/*
> >>> +	 * All non-RAZ feature registers listed here must also be
> >>> +	 * present in arm64_ftr_regs[].
> >>> +	 */
> >>> +
> >>> +	/* AArch64 mappings of the AArch32 ID registers */
> >>> +	/* ID_AFR0_EL1 not exposed to guests for now */
> >>> +	ID(PFR0),	ID(PFR1),	ID(DFR0),	_ID_RAZ(1,3),
> >>> +	ID(MMFR0),	ID(MMFR1),	ID(MMFR2),	ID(MMFR3),
> >>> +	ID(ISAR0),	ID(ISAR1),	ID(ISAR2),	ID(ISAR3),
> >>> +	ID(ISAR4),	ID(ISAR5),	ID(MMFR4),	_ID_RAZ(2,7),
> >>> +	_ID(MVFR0),	_ID(MVFR1),	_ID(MVFR2),	_ID_RAZ(3,3),
> >>> +	_ID_RAZ(3,4),	_ID_RAZ(3,5),	_ID_RAZ(3,6),	_ID_RAZ(3,7),
> >>
> >> #bikeshed:
> >>
> >> OK, this is giving me a headache. Too many variants with similar names.
> >> ID and _ID
> >> I'm also slightly perplexed with the amalgamation of RAZ because the
> >> register is not defined yet in the architecture, and RAZ because we
> >> don't expose it (like ID_AFR0_EL1). Yes, there is a number of comments
> > 
> > This "raz" overloading already seems present in other places, such as the
> > cpufeatures code.  (Which is not necessarily a good reason for adding
> > more of it...)
> > 
> >> to document that, but the code should aim to be be self-documenting. How
> >> about IDRAZ() for those we want to "hide", and IDRSV for encodings that
> >> are not allocated yet? It would look like this:
> >>
> >> 	IDREG(ID_PFR0),		IDREG(ID_PFR1),		IDREG(ID_DFR0),
> >> 	IDRAZ(ID_AFR0),		IDREG(ID_MMFR0),	IDREG(ID_MMFR1),
> >> 	IDREG(ID_MMFR2),	IDREG(ID_MMFR3),	IDREG(ID_ISAR0),
> >> 	IDREG(ID_ISAR1),	IDREG(ID_ISAR2),	IDREG(ID_ISAR3),
> >> 	IDREG(ID_ISAR4),	IDREG(ID_ISAR5),	IDREG(ID_MMFR4),
> >> 	IDRSV(2,7),		IDREG(MVFR0),		IDREG(MVFR1),
> >> 	IDREG(MVFR2),		IDRSV(3,3),		IDRSV(3,4),	
> >> 	IDRSV(3,5),		IDRSV(3,6),		IDRSV(3,7),
> >>
> >> Yes, only 3 a line. Lines are cheap. And yes, they also have similar
> >> names, but I said #bikeshed.
> > 
> > So, point taken, but the main reason for making this a table was to make
> > it easy to see by eye how the entries map to the encoding while hacking
> > this up, which helped me to make sure no entries were missed or in the
> > wrong place etc.
> > 
> > With 3 entries per line that visual map is lost, and with 2 entries per
> > line it's debatable whether it's worth having multiple entries per line
> > at all.
> 
> Let's be clear. I don't care at all about the number of entries per
> line. I can widen my editor to 200 columns if I need to. If you think 4
> is the way, keep it to 4.
> 
> My point is about the readability of both the macros and the
> identifiers, and your initial proposal did seem to lack on both counts.

Agreed, I was just trying to explain why it ended up that way in the
first place, and I'm happy to change it.

> > So now that the table exists maybe we should just have one entry per
> > line like everything else -- it really depends on which option you think
> > is best for ongoing maintenance.
> > 
> > 
> > Having one per line allows much less cryptic names, allowing the
> > temptingly short but ambiguous "RAZ" to be avoided:
> > 
> > 	ID_SANITISED(ID_ISAR5),
> > 	ID_RAZ_FOR_GUEST(ID_AFR0),
> > 	ID_UNALLOCATED(crm, op2)
> > 
> > With a whole line and different lengths, it's easier to pick out
> > the different cases by eye, so they don't all look like IDRXX (and are a
> > more tasteful colour perhaps).
> > 
> > Blank lines and/or comments can split the list into sensible blocks for
> > readability if needed.
> > 
> > If you're happy with naming along those broad lines then I'm happy to
> > see what it looks like.
> 
> Sure. If you're happy with that, so am I.
> 
> >>> +
> >>> +	/* AArch64 ID registers */
> >>> +	ID(AA64PFR0),	ID(AA64PFR1),	_ID_RAZ(4,2),	_ID_RAZ(4,3),
> >>> +	_ID_RAZ(4,4),	_ID_RAZ(4,5),	_ID_RAZ(4,6),	_ID_RAZ(4,7),
> >>> +	ID(AA64DFR0),	ID(AA64DFR1),	_ID_RAZ(5,2),	_ID_RAZ(5,3),
> >>> +	/* ID_AA64AFR0_EL1 and ID_AA64AFR0_EL1 not exposed to guests for now */
> > 
> > There are no sysreg definitions for IA_AA64AFR{0,1}_EL1 yet.
> > 
> > If we want to macroise those rather than just commenting, I guess
> > they'll need adding in sysreg.h.  I'd prefer not to imply these are
> > "unallocated" or similar when the architecture does define them.
> > 
> > Can I take it there's no problem with zombie entries in sysreg.h so long
> > as they're at least referenced somewhere?  (Arguably they wouldn't be
> > zombies then, but hopefully you see what I mean.)
> 
> That'd be the right thing to do. The register exists, and KVM handles it
> by returning 0 when a guest reads it. So I'd argue that it *must* be
> defined in sysreg.h, and given its full visibility in that table.

OK, sounds good -- I'll reroll with that change.

Cheers
---Dave

Re: [PATCH 02/27] arm64: KVM: Hide unsupported AArch64 CPU features from guests

[Dave Martin]

On Thu, Aug 17, 2017 at 09:45:51AM +0100, Marc Zyngier wrote:
> On 16/08/17 21:32, Dave Martin wrote:
> > On Wed, Aug 16, 2017 at 12:10:38PM +0100, Marc Zyngier wrote:
> >> On 09/08/17 13:05, Dave Martin wrote:
> >>> Currently, a guest kernel sees the true CPU feature registers
> >>> (ID_*_EL1) when it reads them using MRS instructions.  This means
> >>> that the guest will observe features that are present in the
> >>> hardware but the host doesn't understand or doesn't provide support
> >>> for.  A guest may legimitately try to use such a feature as per the
> >>> architecture, but use of the feature may trap instead of working
> >>> normally, triggering undef injection into the guest.
> >>>
> >>> This is not a problem for the host, but the guest may go wrong when
> >>> running on newer hardware than the host knows about.
> >>>
> >>> This patch hides from guest VMs any AArch64-specific CPU features
> >>> that the host doesn't support, by exposing to the guest the
> >>> sanitised versions of the registers computed by the cpufeatures
> >>> framework, instead of the true hardware registers.  To achieve
> >>> this, HCR_EL2.TID3 is now set for AArch64 guests, and emulation
> >>> code is added to KVM to report the sanitised versions of the
> >>> affected registers in response to MRS and register reads from
> >>> userspace.
> >>>
> >>> The affected registers are removed from invariant_sys_regs[] (since
> >>> the invariant_sys_regs handling is no longer quite correct for
> >>> them) and added to sys_reg_desgs[], with appropriate access(),
> >>> get_user() and set_user() methods.  No runtime vcpu storage is
> >>> allocated for the registers: instead, they are read on demand from
> >>> the cpufeatures framework.  This may need modification in the
> >>> future if there is a need for userspace to customise the features
> >>> visible to the guest.
> >>>
> >>> Attempts by userspace to write the registers are handled similarly
> >>> to the current invariant_sys_regs handling: writes are permitted,
> >>> but only if they don't attempt to change the value.  This is
> >>> sufficient to support VM snapshot/restore from userspace.
> >>>
> >>> Because of the additional registers, restoring a VM on an older
> >>> kernel may not work unless userspace knows how to handle the extra
> >>> VM registers exposed to the KVM user ABI by this patch.
> >>>
> >>> Under the principle of least damage, this patch makes no attempt to
> >>> handle any of the other registers currently in
> >>> invariant_sys_regs[], or to emulate registers for AArch32: however,
> >>> these could be handled in a similar way in future, as necessary.
> >>>
> >>> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> >>> ---
> >>>  arch/arm64/kvm/hyp/switch.c |   6 ++
> >>>  arch/arm64/kvm/sys_regs.c   | 224 +++++++++++++++++++++++++++++++++++---------
> >>>  2 files changed, 185 insertions(+), 45 deletions(-)

[...]

> >>> +static bool __access_id_reg(struct kvm_vcpu *vcpu,
> >>> +			    struct sys_reg_params *p,
> >>> +			    const struct sys_reg_desc const *r,
> >>> +			    bool raz)
> >>> +{
> >>> +	if (p->is_write) {
> >>> +		kvm_inject_undefined(vcpu);
> >>> +		return false;
> >>> +	}
> >>
> >> I don't think this is supposed to happen (should have UNDEF-ed at EL1).
> >> You can call write_to_read_only() in that case, which will spit out a
> >> warning and inject the exception.
> > 
> > I'll check this -- sounds about right.
> > 
> > If is should never happen, should I just delete that code or BUG()?  I
> > notice a BUG_ON() for a similar situation in access_vm_reg() for example.
> > 
> > Or do we not quite trust hardware not to get this wrong?
> > (It feels like the kind of thing that could slip through validation
> > and/or would be considered not worth a respin, but it seems wrong to
> > work around a theoretical hardware bug before it's confirmed to exist,
> > unless we think for some reason that it's really likely.)
> 
> That's the way we handle this for the rest of the accessors. We used to
> have a BUG_ON(), but it is pretty silly to kill the whole system for
> such a small deviation from the architecture. And maybe it is useless,
> but it doesn't hurt either.

OK, that makes sense -- I'll follow the precedent here and call
write_to_read_only() if this happens.

> >>> +
> >>> +	p->regval = read_id_reg(r, raz);
> >>> +	return true;
> >>> +}
> > 
> > [...]
> > 
> >>> @@ -944,6 +1073,32 @@ static const struct sys_reg_desc sys_reg_descs[] = {
> >>>  	{ SYS_DESC(SYS_DBGVCR32_EL2), NULL, reset_val, DBGVCR32_EL2, 0 },
> >>>  
> >>>  	{ SYS_DESC(SYS_MPIDR_EL1), NULL, reset_mpidr, MPIDR_EL1 },
> >>> +
> >>> +	/*
> >>> +	 * All non-RAZ feature registers listed here must also be
> >>> +	 * present in arm64_ftr_regs[].
> >>> +	 */
> >>> +
> >>> +	/* AArch64 mappings of the AArch32 ID registers */
> >>> +	/* ID_AFR0_EL1 not exposed to guests for now */
> >>> +	ID(PFR0),	ID(PFR1),	ID(DFR0),	_ID_RAZ(1,3),
> >>> +	ID(MMFR0),	ID(MMFR1),	ID(MMFR2),	ID(MMFR3),
> >>> +	ID(ISAR0),	ID(ISAR1),	ID(ISAR2),	ID(ISAR3),
> >>> +	ID(ISAR4),	ID(ISAR5),	ID(MMFR4),	_ID_RAZ(2,7),
> >>> +	_ID(MVFR0),	_ID(MVFR1),	_ID(MVFR2),	_ID_RAZ(3,3),
> >>> +	_ID_RAZ(3,4),	_ID_RAZ(3,5),	_ID_RAZ(3,6),	_ID_RAZ(3,7),
> >>
> >> #bikeshed:
> >>
> >> OK, this is giving me a headache. Too many variants with similar names.
> >> ID and _ID
> >> I'm also slightly perplexed with the amalgamation of RAZ because the
> >> register is not defined yet in the architecture, and RAZ because we
> >> don't expose it (like ID_AFR0_EL1). Yes, there is a number of comments
> > 
> > This "raz" overloading already seems present in other places, such as the
> > cpufeatures code.  (Which is not necessarily a good reason for adding
> > more of it...)
> > 
> >> to document that, but the code should aim to be be self-documenting. How
> >> about IDRAZ() for those we want to "hide", and IDRSV for encodings that
> >> are not allocated yet? It would look like this:
> >>
> >> 	IDREG(ID_PFR0),		IDREG(ID_PFR1),		IDREG(ID_DFR0),
> >> 	IDRAZ(ID_AFR0),		IDREG(ID_MMFR0),	IDREG(ID_MMFR1),
> >> 	IDREG(ID_MMFR2),	IDREG(ID_MMFR3),	IDREG(ID_ISAR0),
> >> 	IDREG(ID_ISAR1),	IDREG(ID_ISAR2),	IDREG(ID_ISAR3),
> >> 	IDREG(ID_ISAR4),	IDREG(ID_ISAR5),	IDREG(ID_MMFR4),
> >> 	IDRSV(2,7),		IDREG(MVFR0),		IDREG(MVFR1),
> >> 	IDREG(MVFR2),		IDRSV(3,3),		IDRSV(3,4),	
> >> 	IDRSV(3,5),		IDRSV(3,6),		IDRSV(3,7),
> >>
> >> Yes, only 3 a line. Lines are cheap. And yes, they also have similar
> >> names, but I said #bikeshed.
> > 
> > So, point taken, but the main reason for making this a table was to make
> > it easy to see by eye how the entries map to the encoding while hacking
> > this up, which helped me to make sure no entries were missed or in the
> > wrong place etc.
> > 
> > With 3 entries per line that visual map is lost, and with 2 entries per
> > line it's debatable whether it's worth having multiple entries per line
> > at all.
> 
> Let's be clear. I don't care at all about the number of entries per
> line. I can widen my editor to 200 columns if I need to. If you think 4
> is the way, keep it to 4.
> 
> My point is about the readability of both the macros and the
> identifiers, and your initial proposal did seem to lack on both counts.

Agreed, I was just trying to explain why it ended up that way in the
first place, and I'm happy to change it.

> > So now that the table exists maybe we should just have one entry per
> > line like everything else -- it really depends on which option you think
> > is best for ongoing maintenance.
> > 
> > 
> > Having one per line allows much less cryptic names, allowing the
> > temptingly short but ambiguous "RAZ" to be avoided:
> > 
> > 	ID_SANITISED(ID_ISAR5),
> > 	ID_RAZ_FOR_GUEST(ID_AFR0),
> > 	ID_UNALLOCATED(crm, op2)
> > 
> > With a whole line and different lengths, it's easier to pick out
> > the different cases by eye, so they don't all look like IDRXX (and are a
> > more tasteful colour perhaps).
> > 
> > Blank lines and/or comments can split the list into sensible blocks for
> > readability if needed.
> > 
> > If you're happy with naming along those broad lines then I'm happy to
> > see what it looks like.
> 
> Sure. If you're happy with that, so am I.
> 
> >>> +
> >>> +	/* AArch64 ID registers */
> >>> +	ID(AA64PFR0),	ID(AA64PFR1),	_ID_RAZ(4,2),	_ID_RAZ(4,3),
> >>> +	_ID_RAZ(4,4),	_ID_RAZ(4,5),	_ID_RAZ(4,6),	_ID_RAZ(4,7),
> >>> +	ID(AA64DFR0),	ID(AA64DFR1),	_ID_RAZ(5,2),	_ID_RAZ(5,3),
> >>> +	/* ID_AA64AFR0_EL1 and ID_AA64AFR0_EL1 not exposed to guests for now */
> > 
> > There are no sysreg definitions for IA_AA64AFR{0,1}_EL1 yet.
> > 
> > If we want to macroise those rather than just commenting, I guess
> > they'll need adding in sysreg.h.  I'd prefer not to imply these are
> > "unallocated" or similar when the architecture does define them.
> > 
> > Can I take it there's no problem with zombie entries in sysreg.h so long
> > as they're at least referenced somewhere?  (Arguably they wouldn't be
> > zombies then, but hopefully you see what I mean.)
> 
> That'd be the right thing to do. The register exists, and KVM handles it
> by returning 0 when a guest reads it. So I'd argue that it *must* be
> defined in sysreg.h, and given its full visibility in that table.

OK, sounds good -- I'll reroll with that change.

Cheers
---Dave

Re: [PATCH 24/27] arm64/sve: Detect SVE and activate runtime support

[Dave Martin]

On Wed, Aug 16, 2017 at 06:53:07PM +0100, Suzuki K Poulose wrote:
> On 09/08/17 13:05, Dave Martin wrote:
> >[This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]
> >
> >This patch enables detection of hardware SVE support via the
> >cpufeatures framework, and reports its presence to the kernel and
> >userspace via the new ARM64_SVE cpucap and HWCAP_SVE hwcap
> >respectively.
> >
> >Userspace can also detect SVE using ID_AA64PFR0_EL1, using the
> >cpufeatures MRS emulation.
> >
> >When running on hardware that supports SVE, this enables runtime
> >kernel support for SVE, and allows user tasks to execute SVE
> >instructions and make of the of the SVE-specific user/kernel
> >interface extensions implemented by this series.
> >
> >Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> 
> 
> >---
> > arch/arm64/include/asm/cpucaps.h    |  3 ++-
> > arch/arm64/include/asm/cpufeature.h |  3 ++-
> > arch/arm64/include/uapi/asm/hwcap.h |  1 +
> > arch/arm64/kernel/cpufeature.c      | 16 ++++++++++++++++
> > arch/arm64/kernel/cpuinfo.c         |  1 +
> > 5 files changed, 22 insertions(+), 2 deletions(-)
> >
> 
> ...
> 
> >
> >@@ -944,6 +957,9 @@ static const struct arm64_cpu_capabilities arm64_elf_hwcaps[] = {
> >        HWCAP_CAP(SYS_ID_AA64ISAR1_EL1, ID_AA64ISAR1_JSCVT_SHIFT, FTR_UNSIGNED, 1, CAP_HWCAP, HWCAP_JSCVT),
> >        HWCAP_CAP(SYS_ID_AA64ISAR1_EL1, ID_AA64ISAR1_FCMA_SHIFT, FTR_UNSIGNED, 1, CAP_HWCAP, HWCAP_FCMA),
> >        HWCAP_CAP(SYS_ID_AA64ISAR1_EL1, ID_AA64ISAR1_LRCPC_SHIFT, FTR_UNSIGNED, 1, CAP_HWCAP, HWCAP_LRCPC),
> >+#ifdef CONFIG_ARM64_SVE
> >+       HWCAP_CAP(SYS_ID_AA64PFR0_EL1, ID_AA64PFR0_SVE_SHIFT, FTR_UNSIGNED, ID_AA64PFR0_SVE, CAP_HWCAP, HWCAP_SVE),
> >+#endif
> >        {},
> > };
> >
> >diff --git a/arch/arm64/kernel/cpuinfo.c b/arch/arm64/kernel/cpuinfo.c
> >index 0e087d7..943aad0 100644
> >--- a/arch/arm64/kernel/cpuinfo.c
> >+++ b/arch/arm64/kernel/cpuinfo.c
> >@@ -69,6 +69,7 @@ static const char *const hwcap_str[] = {
> >        "jscvt",
> >        "fcma",
> >        "lrcpc",
> >+       "sve",
> >        NULL
> > };
> >
> 
> 
> Dave,
> 
> Could you please update the Documentation for the exposed CPU feature
> registers to include the SVE field ? Rest looks good to me. With that
> change,

Oops, yes, I'll update that.  Rutland's hwcaps documentation will also
need an update.

> Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>

Thanks for the review
---Dave

Re: [PATCH 24/27] arm64/sve: Detect SVE and activate runtime support

[Dave Martin]

On Wed, Aug 16, 2017 at 06:53:07PM +0100, Suzuki K Poulose wrote:
> On 09/08/17 13:05, Dave Martin wrote:
> >[This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]
> >
> >This patch enables detection of hardware SVE support via the
> >cpufeatures framework, and reports its presence to the kernel and
> >userspace via the new ARM64_SVE cpucap and HWCAP_SVE hwcap
> >respectively.
> >
> >Userspace can also detect SVE using ID_AA64PFR0_EL1, using the
> >cpufeatures MRS emulation.
> >
> >When running on hardware that supports SVE, this enables runtime
> >kernel support for SVE, and allows user tasks to execute SVE
> >instructions and make of the of the SVE-specific user/kernel
> >interface extensions implemented by this series.
> >
> >Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> 
> 
> >---
> > arch/arm64/include/asm/cpucaps.h    |  3 ++-
> > arch/arm64/include/asm/cpufeature.h |  3 ++-
> > arch/arm64/include/uapi/asm/hwcap.h |  1 +
> > arch/arm64/kernel/cpufeature.c      | 16 ++++++++++++++++
> > arch/arm64/kernel/cpuinfo.c         |  1 +
> > 5 files changed, 22 insertions(+), 2 deletions(-)
> >
> 
> ...
> 
> >
> >@@ -944,6 +957,9 @@ static const struct arm64_cpu_capabilities arm64_elf_hwcaps[] = {
> >        HWCAP_CAP(SYS_ID_AA64ISAR1_EL1, ID_AA64ISAR1_JSCVT_SHIFT, FTR_UNSIGNED, 1, CAP_HWCAP, HWCAP_JSCVT),
> >        HWCAP_CAP(SYS_ID_AA64ISAR1_EL1, ID_AA64ISAR1_FCMA_SHIFT, FTR_UNSIGNED, 1, CAP_HWCAP, HWCAP_FCMA),
> >        HWCAP_CAP(SYS_ID_AA64ISAR1_EL1, ID_AA64ISAR1_LRCPC_SHIFT, FTR_UNSIGNED, 1, CAP_HWCAP, HWCAP_LRCPC),
> >+#ifdef CONFIG_ARM64_SVE
> >+       HWCAP_CAP(SYS_ID_AA64PFR0_EL1, ID_AA64PFR0_SVE_SHIFT, FTR_UNSIGNED, ID_AA64PFR0_SVE, CAP_HWCAP, HWCAP_SVE),
> >+#endif
> >        {},
> > };
> >
> >diff --git a/arch/arm64/kernel/cpuinfo.c b/arch/arm64/kernel/cpuinfo.c
> >index 0e087d7..943aad0 100644
> >--- a/arch/arm64/kernel/cpuinfo.c
> >+++ b/arch/arm64/kernel/cpuinfo.c
> >@@ -69,6 +69,7 @@ static const char *const hwcap_str[] = {
> >        "jscvt",
> >        "fcma",
> >        "lrcpc",
> >+       "sve",
> >        NULL
> > };
> >
> 
> 
> Dave,
> 
> Could you please update the Documentation for the exposed CPU feature
> registers to include the SVE field ? Rest looks good to me. With that
> change,

Oops, yes, I'll update that.  Rutland's hwcaps documentation will also
need an update.

> Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>

Thanks for the review
---Dave

Re: [PATCH 15/27] arm64/sve: Probe SVE capabilities and usable vector lengths

[Dave Martin]

On Wed, Aug 16, 2017 at 06:48:01PM +0100, Suzuki K Poulose wrote:
> On 09/08/17 13:05, Dave Martin wrote:
> >[This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]

Any idea what this is ^ ?  I don't know if this is caused by me or you,
but I only seem to see it on subthreads you've replied to.

> >This patch uses the cpufeatures framework to determine common SVE
> >capabilities and vector lengths, and configures the runtime SVE
> >support code appropriately.
> >
> >ZCR_ELx is not really a feature register, but it is convenient to
> >use it as a template for recording the maximum vector length
> >supported by a CPU, using the LEN field.  This field is similar to
> >a feature field in that it is a contiguous bitfield for which we
> >want to determine the minimum system-wide value.  This patch adds
> >ZCR as a pseudo-register in cpuinfo/cpufeatures, with appropriate
> >custom code to populate it.  Finding the minimum supported value of
> >the LEN field is left to the cpufeatures framework in the usual
> >way.
> >
> >The meaning of ID_AA64ZFR0_EL1 is not architecturally defined yet,
> >so for now we just require it to be zero.
> >
> >Note that much of this code is dormant and SVE still won't be used
> >yet, since system_supports_sve() remains hardwired to false.
> >
> >Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> 
> Dave,
> 
> The cpufeature bits look good to me, with one minor comment.
> 
> 
> >diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> >index bce95de..955c873 100644
> >--- a/arch/arm64/kernel/fpsimd.c
> >+++ b/arch/arm64/kernel/fpsimd.c
> 
> ...
> 
> >+void __init sve_setup(void)
> >+{
> >+       u64 zcr;
> >+       unsigned int max_vl;
> >+
> >+       if (!system_supports_sve())
> >+               return;
> >+
> >+       /*
> >+        * The architecture mandates 128-bit vectors be supported, and
> >+        * the code assumes elsewhere that sve_vq_map is non-empty:
> >+        */
> >+       BUG_ON(!test_bit(vq_to_bit(1), sve_vq_map));
> >+
> >+       sve_vq_map_finalised = true;
> 
> We have something local in cpufeature.c, sys_caps_initialised. May be we could
> reuse it here ? With or without that change, FWIW.

I'll take a look at that.  Inventing that here seemed a little ugly, and
this is all driven from the cpufreatures code anyway now which ensures a
certain ordering.

If I can reuse sys_caps_initialised for this, I will -- seems pointless
to reinvent it.

> Acked-by: Suzuki K Poulose <suzuki.poulose@arm.com>

Thanks
---Dave

Re: [PATCH 15/27] arm64/sve: Probe SVE capabilities and usable vector lengths

[Dave Martin]

On Wed, Aug 16, 2017 at 06:48:01PM +0100, Suzuki K Poulose wrote:
> On 09/08/17 13:05, Dave Martin wrote:
> >[This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]

Any idea what this is ^ ?  I don't know if this is caused by me or you,
but I only seem to see it on subthreads you've replied to.

> >This patch uses the cpufeatures framework to determine common SVE
> >capabilities and vector lengths, and configures the runtime SVE
> >support code appropriately.
> >
> >ZCR_ELx is not really a feature register, but it is convenient to
> >use it as a template for recording the maximum vector length
> >supported by a CPU, using the LEN field.  This field is similar to
> >a feature field in that it is a contiguous bitfield for which we
> >want to determine the minimum system-wide value.  This patch adds
> >ZCR as a pseudo-register in cpuinfo/cpufeatures, with appropriate
> >custom code to populate it.  Finding the minimum supported value of
> >the LEN field is left to the cpufeatures framework in the usual
> >way.
> >
> >The meaning of ID_AA64ZFR0_EL1 is not architecturally defined yet,
> >so for now we just require it to be zero.
> >
> >Note that much of this code is dormant and SVE still won't be used
> >yet, since system_supports_sve() remains hardwired to false.
> >
> >Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> 
> Dave,
> 
> The cpufeature bits look good to me, with one minor comment.
> 
> 
> >diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> >index bce95de..955c873 100644
> >--- a/arch/arm64/kernel/fpsimd.c
> >+++ b/arch/arm64/kernel/fpsimd.c
> 
> ...
> 
> >+void __init sve_setup(void)
> >+{
> >+       u64 zcr;
> >+       unsigned int max_vl;
> >+
> >+       if (!system_supports_sve())
> >+               return;
> >+
> >+       /*
> >+        * The architecture mandates 128-bit vectors be supported, and
> >+        * the code assumes elsewhere that sve_vq_map is non-empty:
> >+        */
> >+       BUG_ON(!test_bit(vq_to_bit(1), sve_vq_map));
> >+
> >+       sve_vq_map_finalised = true;
> 
> We have something local in cpufeature.c, sys_caps_initialised. May be we could
> reuse it here ? With or without that change, FWIW.

I'll take a look at that.  Inventing that here seemed a little ugly, and
this is all driven from the cpufreatures code anyway now which ensures a
certain ordering.

If I can reuse sys_caps_initialised for this, I will -- seems pointless
to reinvent it.

> Acked-by: Suzuki K Poulose <suzuki.poulose@arm.com>

Thanks
---Dave

Re: [PATCH 15/27] arm64/sve: Probe SVE capabilities and usable vector lengths

[Suzuki K Poulose]

On 17/08/17 11:04, Dave Martin wrote:
> On Wed, Aug 16, 2017 at 06:48:01PM +0100, Suzuki K Poulose wrote:
>> On 09/08/17 13:05, Dave Martin wrote:
>>> [This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]
>
> Any idea what this is ^ ?  I don't know if this is caused by me or you,
> but I only seem to see it on subthreads you've replied to.
>

Dave,

Sorry about that, should have trimmed that. It looks like the mail server is unhappy
about email received via kvmarm list and I don't know why.

>>
>>> +void __init sve_setup(void)
>>> +{
>>> +       u64 zcr;
>>> +       unsigned int max_vl;
>>> +
>>> +       if (!system_supports_sve())
>>> +               return;
>>> +
>>> +       /*
>>> +        * The architecture mandates 128-bit vectors be supported, and
>>> +        * the code assumes elsewhere that sve_vq_map is non-empty:
>>> +        */
>>> +       BUG_ON(!test_bit(vq_to_bit(1), sve_vq_map));
>>> +
>>> +       sve_vq_map_finalised = true;
>>
>> We have something local in cpufeature.c, sys_caps_initialised. May be we could
>> reuse it here ? With or without that change, FWIW.
>
> I'll take a look at that.  Inventing that here seemed a little ugly, and
> this is all driven from the cpufreatures code anyway now which ensures a
> certain ordering.
>
> If I can reuse sys_caps_initialised for this, I will -- seems pointless
> to reinvent it.

Suzuki

Re: [PATCH 15/27] arm64/sve: Probe SVE capabilities and usable vector lengths

[Suzuki K Poulose]

On 17/08/17 11:04, Dave Martin wrote:
> On Wed, Aug 16, 2017 at 06:48:01PM +0100, Suzuki K Poulose wrote:
>> On 09/08/17 13:05, Dave Martin wrote:
>>> [This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing at http://aka.ms/LearnAboutSpoofing]
>
> Any idea what this is ^ ?  I don't know if this is caused by me or you,
> but I only seem to see it on subthreads you've replied to.
>

Dave,

Sorry about that, should have trimmed that. It looks like the mail server is unhappy
about email received via kvmarm list and I don't know why.

>>
>>> +void __init sve_setup(void)
>>> +{
>>> +       u64 zcr;
>>> +       unsigned int max_vl;
>>> +
>>> +       if (!system_supports_sve())
>>> +               return;
>>> +
>>> +       /*
>>> +        * The architecture mandates 128-bit vectors be supported, and
>>> +        * the code assumes elsewhere that sve_vq_map is non-empty:
>>> +        */
>>> +       BUG_ON(!test_bit(vq_to_bit(1), sve_vq_map));
>>> +
>>> +       sve_vq_map_finalised = true;
>>
>> We have something local in cpufeature.c, sys_caps_initialised. May be we could
>> reuse it here ? With or without that change, FWIW.
>
> I'll take a look at that.  Inventing that here seemed a little ugly, and
> this is all driven from the cpufreatures code anyway now which ensures a
> certain ordering.
>
> If I can reuse sys_caps_initialised for this, I will -- seems pointless
> to reinvent it.

Suzuki

Re: [PATCH 11/27] arm64/sve: Core task context handling

[Dave Martin]

On Tue, Aug 15, 2017 at 06:31:05PM +0100, Ard Biesheuvel wrote:
> Hi Dave,
> 
> On 9 August 2017 at 13:05, Dave Martin <Dave.Martin@arm.com> wrote:
> > This patch adds the core support for switching and managing the SVE
> > architectural state of user tasks.

[...]

> > +static u64 sve_cpacr_trap_on(u64 cpacr)
> > +{
> > +       return cpacr & ~(u64)CPACR_EL1_ZEN_EL0EN;
> > +}
> > +
> > +static u64 sve_cpacr_trap_off(u64 cpacr)
> > +{
> > +       return cpacr | CPACR_EL1_ZEN_EL0EN;
> > +}
> > +
> > +static void change_cpacr(u64 old, u64 new)
> > +{
> > +       if (old != new)
> > +               write_sysreg(new, CPACR_EL1);
> > +}

[...]

> > +static void task_fpsimd_load(void)
> > +{
> > +       if (system_supports_sve() && test_thread_flag(TIF_SVE)) {
> > +               unsigned int vl = current->thread.sve_vl;
> > +
> > +               BUG_ON(!sve_vl_valid(vl));
> > +               sve_load_state(sve_pffr(current),
> > +                              &current->thread.fpsimd_state.fpsr,
> > +                              sve_vq_from_vl(vl) - 1);
> > +       } else
> > +               fpsimd_load_state(&current->thread.fpsimd_state);
> > +
> 
> Please use braces consistently on all branches of an if ()
> 
> > +       if (system_supports_sve()) {
> > +               u64 cpacr = read_sysreg(CPACR_EL1);
> > +               u64 new_cpacr;
> > +
> > +               /* Toggle SVE trapping for userspace if needed */
> > +               if (test_thread_flag(TIF_SVE))
> > +                       new_cpacr = sve_cpacr_trap_off(cpacr);
> > +               else
> > +                       new_cpacr = sve_cpacr_trap_on(cpacr);
> > +
> > +               change_cpacr(cpacr, new_cpacr);
> 
> I understand you want to avoid setting CPACR to the same value, but
> this does look a bit clunky IMO. Wouldn't it be much better to have a
> generic accessor with a mask and a value that encapsulates this?

For this I now have:

static void change_cpacr(u64 val, u64 mask)
{
	u64 cpacr = read_sysreg(CPACR_EL1);
	u64 new = (cpacr & ~mask) | val;

	if (new != cpacr)
		write_sysreg(new, CPACR_EL1);
}

static void sve_cpacr_trap_on(void)
{
	change_cpacr(0, CPACR_EL1_ZEN_EL0EN);
}

static void sve_cpacr_trap_off(void)
{
	change_cpacr(CPACR_EL1_ZEN_EL0EN, CPACR_EL1_ZEN_EL0EN);
}


This is stilla little verbose, but fairly clean.  Possibly this was the
sort of thing you meant by a generic accessor.

What do you think?

[...]

Cheers
---Dave

Re: [PATCH 11/27] arm64/sve: Core task context handling

[Ard Biesheuvel]

On 17 August 2017 at 17:42, Dave Martin <Dave.Martin@arm.com> wrote:
> On Tue, Aug 15, 2017 at 06:31:05PM +0100, Ard Biesheuvel wrote:
>> Hi Dave,
>>
>> On 9 August 2017 at 13:05, Dave Martin <Dave.Martin@arm.com> wrote:
>> > This patch adds the core support for switching and managing the SVE
>> > architectural state of user tasks.
>
> [...]
>
>> > +static u64 sve_cpacr_trap_on(u64 cpacr)
>> > +{
>> > +       return cpacr & ~(u64)CPACR_EL1_ZEN_EL0EN;
>> > +}
>> > +
>> > +static u64 sve_cpacr_trap_off(u64 cpacr)
>> > +{
>> > +       return cpacr | CPACR_EL1_ZEN_EL0EN;
>> > +}
>> > +
>> > +static void change_cpacr(u64 old, u64 new)
>> > +{
>> > +       if (old != new)
>> > +               write_sysreg(new, CPACR_EL1);
>> > +}
>
> [...]
>
>> > +static void task_fpsimd_load(void)
>> > +{
>> > +       if (system_supports_sve() && test_thread_flag(TIF_SVE)) {
>> > +               unsigned int vl = current->thread.sve_vl;
>> > +
>> > +               BUG_ON(!sve_vl_valid(vl));
>> > +               sve_load_state(sve_pffr(current),
>> > +                              &current->thread.fpsimd_state.fpsr,
>> > +                              sve_vq_from_vl(vl) - 1);
>> > +       } else
>> > +               fpsimd_load_state(&current->thread.fpsimd_state);
>> > +
>>
>> Please use braces consistently on all branches of an if ()
>>
>> > +       if (system_supports_sve()) {
>> > +               u64 cpacr = read_sysreg(CPACR_EL1);
>> > +               u64 new_cpacr;
>> > +
>> > +               /* Toggle SVE trapping for userspace if needed */
>> > +               if (test_thread_flag(TIF_SVE))
>> > +                       new_cpacr = sve_cpacr_trap_off(cpacr);
>> > +               else
>> > +                       new_cpacr = sve_cpacr_trap_on(cpacr);
>> > +
>> > +               change_cpacr(cpacr, new_cpacr);
>>
>> I understand you want to avoid setting CPACR to the same value, but
>> this does look a bit clunky IMO. Wouldn't it be much better to have a
>> generic accessor with a mask and a value that encapsulates this?
>
> For this I now have:
>
> static void change_cpacr(u64 val, u64 mask)
> {
>         u64 cpacr = read_sysreg(CPACR_EL1);
>         u64 new = (cpacr & ~mask) | val;
>
>         if (new != cpacr)
>                 write_sysreg(new, CPACR_EL1);
> }
>
> static void sve_cpacr_trap_on(void)
> {
>         change_cpacr(0, CPACR_EL1_ZEN_EL0EN);
> }
>
> static void sve_cpacr_trap_off(void)
> {
>         change_cpacr(CPACR_EL1_ZEN_EL0EN, CPACR_EL1_ZEN_EL0EN);
> }
>
>
> This is stilla little verbose, but fairly clean.  Possibly this was the
> sort of thing you meant by a generic accessor.
>
> What do you think?
>

In my opinion, this does look a lot better. Having mask/val pairs like
this is a fairly common pattern, so it should be quite obvious to most
readers what is going on.

Re: [PATCH 01/27] regset: Add support for dynamically sized regsets

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> Currently the regset API doesn't allow for the possibility that
> regsets (or at least, the amount of meaningful data in a regset)
> may change in size.
>
> In particular, this results in useless padding being added to
> coredumps in a regset's current size is smaller than its
> theoretical maximum size.
>
> This patch adds a get_size() function to struct user_regset.
> Individual regset implementations can implement this function to
> return the current size of the regset data.  A regset_size()
> function is added to provide callers with an abstract interface for
> determining the size of a regset without needing to know whether
> the regset is dynamically sized or not.
>
> The only affected user of this interface is the ELF coredump code:
> This patch ports ELF coredump to dump regsets with their actual
> size in the coredump.  This has no effect except for new regsets
> that are dynamically sized and provide a get_size() implementation.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  fs/binfmt_elf.c        |  6 ++---
>  include/linux/regset.h | 67 ++++++++++++++++++++++++++++++++++++++++++++------
>  2 files changed, 63 insertions(+), 10 deletions(-)
>
> diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c
> index 879ff9c..16d2403 100644
> --- a/fs/binfmt_elf.c
> +++ b/fs/binfmt_elf.c
> @@ -1697,7 +1697,7 @@ static int fill_thread_core_info(struct elf_thread_core_info *t,
>  				 long signr, size_t *total)
>  {
>  	unsigned int i;
> -	unsigned int regset_size = view->regsets[0].n * view->regsets[0].size;
> +	unsigned int size = regset_size(t->task, &view->regsets[0]);
>
>  	/*
>  	 * NT_PRSTATUS is the one special case, because the regset data
> @@ -1706,7 +1706,7 @@ static int fill_thread_core_info(struct elf_thread_core_info *t,
>  	 * We assume that regset 0 is NT_PRSTATUS.
>  	 */
>  	fill_prstatus(&t->prstatus, t->task, signr);
> -	(void) view->regsets[0].get(t->task, &view->regsets[0], 0, regset_size,
> +	(void) view->regsets[0].get(t->task, &view->regsets[0], 0, size,
>  				    &t->prstatus.pr_reg, NULL);
>
>  	fill_note(&t->notes[0], "CORE", NT_PRSTATUS,
> @@ -1726,7 +1726,7 @@ static int fill_thread_core_info(struct elf_thread_core_info *t,
>  		if (regset->core_note_type && regset->get &&
>  		    (!regset->active || regset->active(t->task, regset))) {
>  			int ret;
> -			size_t size = regset->n * regset->size;
> +			size_t size = regset_size(t->task, regset);
>  			void *data = kmalloc(size, GFP_KERNEL);
>  			if (unlikely(!data))
>  				return 0;
> diff --git a/include/linux/regset.h b/include/linux/regset.h
> index 8e0c9fe..494ceda 100644
> --- a/include/linux/regset.h
> +++ b/include/linux/regset.h
> @@ -107,6 +107,28 @@ typedef int user_regset_writeback_fn(struct task_struct *target,
>  				     int immediate);
>
>  /**
> + * user_regset_get_size_fn - type of @get_size function in &struct user_regset
> + * @target:	thread being examined
> + * @regset:	regset being examined
> + *
> + * This call is optional; usually the pointer is %NULL.
> + *
> + * When provided, this function must return the current size of regset
> + * data, as observed by the @get function in &struct user_regset.  The
> + * value returned must be a multiple of @size.  The returned size is
> + * required to be valid only until the next time (if any) @regset is
> + * modified for @target.
> + *
> + * This function is intended for dynamically sized regsets.  A regset
> + * that is statically sized does not need to implement it.
> + *
> + * This function should not be called directly: instead, callers should
> + * call regset_size() to determine the current size of a regset.
> + */
> +typedef unsigned int user_regset_get_size_fn(struct task_struct *target,
> +					     const struct user_regset *regset);
> +
> +/**
>   * struct user_regset - accessible thread CPU state
>   * @n:			Number of slots (registers).
>   * @size:		Size in bytes of a slot (register).
> @@ -117,19 +139,33 @@ typedef int user_regset_writeback_fn(struct task_struct *target,
>   * @set:		Function to store values.
>   * @active:		Function to report if regset is active, or %NULL.
>   * @writeback:		Function to write data back to user memory, or %NULL.
> + * @get_size:		Function to return the regset's size, or %NULL.
>   *
>   * This data structure describes a machine resource we call a register set.
>   * This is part of the state of an individual thread, not necessarily
>   * actual CPU registers per se.  A register set consists of a number of
>   * similar slots, given by @n.  Each slot is @size bytes, and aligned to
> - * @align bytes (which is at least @size).
> + * @align bytes (which is at least @size).  For dynamically-sized
> + * regsets, @n must contain the maximum possible number of slots for the
> + * regset, and @get_size must point to a function that returns the
> + * current regset size.
>   *
> - * These functions must be called only on the current thread or on a
> - * thread that is in %TASK_STOPPED or %TASK_TRACED state, that we are
> - * guaranteed will not be woken up and return to user mode, and that we
> - * have called wait_task_inactive() on.  (The target thread always might
> - * wake up for SIGKILL while these functions are working, in which case
> - * that thread's user_regset state might be scrambled.)
> + * Callers that need to know only the current size of the regset and do
> + * not care about its internal structure should call regset_size()
> + * instead of inspecting @n or calling @get_size.
> + *
> + * For backward compatibility, the @get and @set methods must pad to, or
> + * accept, @n * @size bytes, even if the current regset size is smaller.
> + * The precise semantics of these operations depend on the regset being
> + * accessed.
> + *
> + * The functions to which &struct user_regset members point must be
> + * called only on the current thread or on a thread that is in
> + * %TASK_STOPPED or %TASK_TRACED state, that we are guaranteed will not
> + * be woken up and return to user mode, and that we have called
> + * wait_task_inactive() on.  (The target thread always might wake up for
> + * SIGKILL while these functions are working, in which case that
> + * thread's user_regset state might be scrambled.)
>   *
>   * The @pos argument must be aligned according to @align; the @count
>   * argument must be a multiple of @size.  These functions are not
> @@ -156,6 +192,7 @@ struct user_regset {
>  	user_regset_set_fn		*set;
>  	user_regset_active_fn		*active;
>  	user_regset_writeback_fn	*writeback;
> +	user_regset_get_size_fn		*get_size;
>  	unsigned int			n;
>  	unsigned int 			size;
>  	unsigned int 			align;
> @@ -371,5 +408,21 @@ static inline int copy_regset_from_user(struct task_struct *target,
>  	return regset->set(target, regset, offset, size, NULL, data);
>  }
>
> +/**
> + * regset_size - determine the current size of a regset
> + * @target:	thread to be examined
> + * @regset:	regset to be examined
> + *
> + * Note that the returned size is valid only until the next time
> + * (if any) @regset is modified for @target.
> + */
> +static inline unsigned int regset_size(struct task_struct *target,
> +				       const struct user_regset *regset)
> +{
> +	if (!regset->get_size)
> +		return regset->n * regset->size;
> +	else
> +		return regset->get_size(target, regset);
> +}
>
>  #endif	/* <linux/regset.h> */

Re: [PATCH 01/27] regset: Add support for dynamically sized regsets

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> Currently the regset API doesn't allow for the possibility that
> regsets (or at least, the amount of meaningful data in a regset)
> may change in size.
>
> In particular, this results in useless padding being added to
> coredumps in a regset's current size is smaller than its
> theoretical maximum size.
>
> This patch adds a get_size() function to struct user_regset.
> Individual regset implementations can implement this function to
> return the current size of the regset data.  A regset_size()
> function is added to provide callers with an abstract interface for
> determining the size of a regset without needing to know whether
> the regset is dynamically sized or not.
>
> The only affected user of this interface is the ELF coredump code:
> This patch ports ELF coredump to dump regsets with their actual
> size in the coredump.  This has no effect except for new regsets
> that are dynamically sized and provide a get_size() implementation.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  fs/binfmt_elf.c        |  6 ++---
>  include/linux/regset.h | 67 ++++++++++++++++++++++++++++++++++++++++++++------
>  2 files changed, 63 insertions(+), 10 deletions(-)
>
> diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c
> index 879ff9c..16d2403 100644
> --- a/fs/binfmt_elf.c
> +++ b/fs/binfmt_elf.c
> @@ -1697,7 +1697,7 @@ static int fill_thread_core_info(struct elf_thread_core_info *t,
>  				 long signr, size_t *total)
>  {
>  	unsigned int i;
> -	unsigned int regset_size = view->regsets[0].n * view->regsets[0].size;
> +	unsigned int size = regset_size(t->task, &view->regsets[0]);
>
>  	/*
>  	 * NT_PRSTATUS is the one special case, because the regset data
> @@ -1706,7 +1706,7 @@ static int fill_thread_core_info(struct elf_thread_core_info *t,
>  	 * We assume that regset 0 is NT_PRSTATUS.
>  	 */
>  	fill_prstatus(&t->prstatus, t->task, signr);
> -	(void) view->regsets[0].get(t->task, &view->regsets[0], 0, regset_size,
> +	(void) view->regsets[0].get(t->task, &view->regsets[0], 0, size,
>  				    &t->prstatus.pr_reg, NULL);
>
>  	fill_note(&t->notes[0], "CORE", NT_PRSTATUS,
> @@ -1726,7 +1726,7 @@ static int fill_thread_core_info(struct elf_thread_core_info *t,
>  		if (regset->core_note_type && regset->get &&
>  		    (!regset->active || regset->active(t->task, regset))) {
>  			int ret;
> -			size_t size = regset->n * regset->size;
> +			size_t size = regset_size(t->task, regset);
>  			void *data = kmalloc(size, GFP_KERNEL);
>  			if (unlikely(!data))
>  				return 0;
> diff --git a/include/linux/regset.h b/include/linux/regset.h
> index 8e0c9fe..494ceda 100644
> --- a/include/linux/regset.h
> +++ b/include/linux/regset.h
> @@ -107,6 +107,28 @@ typedef int user_regset_writeback_fn(struct task_struct *target,
>  				     int immediate);
>
>  /**
> + * user_regset_get_size_fn - type of @get_size function in &struct user_regset
> + * @target:	thread being examined
> + * @regset:	regset being examined
> + *
> + * This call is optional; usually the pointer is %NULL.
> + *
> + * When provided, this function must return the current size of regset
> + * data, as observed by the @get function in &struct user_regset.  The
> + * value returned must be a multiple of @size.  The returned size is
> + * required to be valid only until the next time (if any) @regset is
> + * modified for @target.
> + *
> + * This function is intended for dynamically sized regsets.  A regset
> + * that is statically sized does not need to implement it.
> + *
> + * This function should not be called directly: instead, callers should
> + * call regset_size() to determine the current size of a regset.
> + */
> +typedef unsigned int user_regset_get_size_fn(struct task_struct *target,
> +					     const struct user_regset *regset);
> +
> +/**
>   * struct user_regset - accessible thread CPU state
>   * @n:			Number of slots (registers).
>   * @size:		Size in bytes of a slot (register).
> @@ -117,19 +139,33 @@ typedef int user_regset_writeback_fn(struct task_struct *target,
>   * @set:		Function to store values.
>   * @active:		Function to report if regset is active, or %NULL.
>   * @writeback:		Function to write data back to user memory, or %NULL.
> + * @get_size:		Function to return the regset's size, or %NULL.
>   *
>   * This data structure describes a machine resource we call a register set.
>   * This is part of the state of an individual thread, not necessarily
>   * actual CPU registers per se.  A register set consists of a number of
>   * similar slots, given by @n.  Each slot is @size bytes, and aligned to
> - * @align bytes (which is at least @size).
> + * @align bytes (which is at least @size).  For dynamically-sized
> + * regsets, @n must contain the maximum possible number of slots for the
> + * regset, and @get_size must point to a function that returns the
> + * current regset size.
>   *
> - * These functions must be called only on the current thread or on a
> - * thread that is in %TASK_STOPPED or %TASK_TRACED state, that we are
> - * guaranteed will not be woken up and return to user mode, and that we
> - * have called wait_task_inactive() on.  (The target thread always might
> - * wake up for SIGKILL while these functions are working, in which case
> - * that thread's user_regset state might be scrambled.)
> + * Callers that need to know only the current size of the regset and do
> + * not care about its internal structure should call regset_size()
> + * instead of inspecting @n or calling @get_size.
> + *
> + * For backward compatibility, the @get and @set methods must pad to, or
> + * accept, @n * @size bytes, even if the current regset size is smaller.
> + * The precise semantics of these operations depend on the regset being
> + * accessed.
> + *
> + * The functions to which &struct user_regset members point must be
> + * called only on the current thread or on a thread that is in
> + * %TASK_STOPPED or %TASK_TRACED state, that we are guaranteed will not
> + * be woken up and return to user mode, and that we have called
> + * wait_task_inactive() on.  (The target thread always might wake up for
> + * SIGKILL while these functions are working, in which case that
> + * thread's user_regset state might be scrambled.)
>   *
>   * The @pos argument must be aligned according to @align; the @count
>   * argument must be a multiple of @size.  These functions are not
> @@ -156,6 +192,7 @@ struct user_regset {
>  	user_regset_set_fn		*set;
>  	user_regset_active_fn		*active;
>  	user_regset_writeback_fn	*writeback;
> +	user_regset_get_size_fn		*get_size;
>  	unsigned int			n;
>  	unsigned int 			size;
>  	unsigned int 			align;
> @@ -371,5 +408,21 @@ static inline int copy_regset_from_user(struct task_struct *target,
>  	return regset->set(target, regset, offset, size, NULL, data);
>  }
>
> +/**
> + * regset_size - determine the current size of a regset
> + * @target:	thread to be examined
> + * @regset:	regset to be examined
> + *
> + * Note that the returned size is valid only until the next time
> + * (if any) @regset is modified for @target.
> + */
> +static inline unsigned int regset_size(struct task_struct *target,
> +				       const struct user_regset *regset)
> +{
> +	if (!regset->get_size)
> +		return regset->n * regset->size;
> +	else
> +		return regset->get_size(target, regset);
> +}
>
>  #endif	/* <linux/regset.h> */


--
Alex Bennée

Re: [PATCH 03/27] arm64: efi: Add missing Kconfig dependency on KERNEL_MODE_NEON

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> The EFI runtime services ABI permits calls to EFI to clobber
> certain FPSIMD/NEON registers, as per the AArch64 procedure call
> standard.
>
> Saving/restoring the clobbered registers around such calls needs
> KERNEL_MODE_NEON, but the dependency is missing from Kconfig.
>
> This patch adds the missing dependency.
>
> This will aid bisection of the patches implementing support for the
> ARM Scalable Vector Extension (SVE).
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  arch/arm64/Kconfig | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index dfd9086..fdc302e 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -1050,6 +1050,7 @@ config EFI_STUB
>  config EFI
>  	bool "UEFI runtime support"
>  	depends on OF && !CPU_BIG_ENDIAN
> +	depends on KERNEL_MODE_NEON
>  	select LIBFDT
>  	select UCS2_STRING
>  	select EFI_PARAMS_FROM_FDT


--
Alex Bennée
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [PATCH 03/27] arm64: efi: Add missing Kconfig dependency on KERNEL_MODE_NEON

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> The EFI runtime services ABI permits calls to EFI to clobber
> certain FPSIMD/NEON registers, as per the AArch64 procedure call
> standard.
>
> Saving/restoring the clobbered registers around such calls needs
> KERNEL_MODE_NEON, but the dependency is missing from Kconfig.
>
> This patch adds the missing dependency.
>
> This will aid bisection of the patches implementing support for the
> ARM Scalable Vector Extension (SVE).
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  arch/arm64/Kconfig | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index dfd9086..fdc302e 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -1050,6 +1050,7 @@ config EFI_STUB
>  config EFI
>  	bool "UEFI runtime support"
>  	depends on OF && !CPU_BIG_ENDIAN
> +	depends on KERNEL_MODE_NEON
>  	select LIBFDT
>  	select UCS2_STRING
>  	select EFI_PARAMS_FROM_FDT


--
Alex Bennée

Re: [PATCH 04/27] arm64: Port deprecated instruction emulation to new sysctl interface

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> Currently, armv8_deprected.c takes charge of the "abi" sysctl
> directory, which makes life difficult for other code that wants to
> register sysctls in the same directory.
>
> There is a "new" [1] sysctl registration interface that removes the
> need to define ctl_tables for parent directories explicitly, which
> is ideal here.
>
> This patch ports register_insn_emulation_sysctl() over to the
> register_sysctl() interface and removes the redundant ctl_table for
> "abi".
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

>
> [1] fea478d4101a (sysctl: Add register_sysctl for normal sysctl
> users)
> The commit message notes an intent to port users of the
> pre-existing interfaces over to register_sysctl(), though the
> number of users of the new interface currently appears negligible.
> ---
>  arch/arm64/kernel/armv8_deprecated.c | 15 +++------------
>  1 file changed, 3 insertions(+), 12 deletions(-)
>
> diff --git a/arch/arm64/kernel/armv8_deprecated.c b/arch/arm64/kernel/armv8_deprecated.c
> index f0e6d71..e15eb2d 100644
> --- a/arch/arm64/kernel/armv8_deprecated.c
> +++ b/arch/arm64/kernel/armv8_deprecated.c
> @@ -228,15 +228,7 @@ static int emulation_proc_handler(struct ctl_table *table, int write,
>  	return ret;
>  }
>
> -static struct ctl_table ctl_abi[] = {
> -	{
> -		.procname = "abi",
> -		.mode = 0555,
> -	},
> -	{ }
> -};
> -
> -static void __init register_insn_emulation_sysctl(struct ctl_table *table)
> +static void __init register_insn_emulation_sysctl(void)
>  {
>  	unsigned long flags;
>  	int i = 0;
> @@ -262,8 +254,7 @@ static void __init register_insn_emulation_sysctl(struct ctl_table *table)
>  	}
>  	raw_spin_unlock_irqrestore(&insn_emulation_lock, flags);
>
> -	table->child = insns_sysctl;
> -	register_sysctl_table(table);
> +	register_sysctl("abi", insns_sysctl);
>  }
>
>  /*
> @@ -644,7 +635,7 @@ static int __init armv8_deprecated_init(void)
>  	cpuhp_setup_state_nocalls(CPUHP_AP_ARM64_ISNDEP_STARTING,
>  				  "arm64/isndep:starting",
>  				  run_all_insn_set_hw_mode, NULL);
> -	register_insn_emulation_sysctl(ctl_abi);
> +	register_insn_emulation_sysctl();
>
>  	return 0;
>  }


--
Alex Bennée
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [PATCH 04/27] arm64: Port deprecated instruction emulation to new sysctl interface

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> Currently, armv8_deprected.c takes charge of the "abi" sysctl
> directory, which makes life difficult for other code that wants to
> register sysctls in the same directory.
>
> There is a "new" [1] sysctl registration interface that removes the
> need to define ctl_tables for parent directories explicitly, which
> is ideal here.
>
> This patch ports register_insn_emulation_sysctl() over to the
> register_sysctl() interface and removes the redundant ctl_table for
> "abi".
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

>
> [1] fea478d4101a (sysctl: Add register_sysctl for normal sysctl
> users)
> The commit message notes an intent to port users of the
> pre-existing interfaces over to register_sysctl(), though the
> number of users of the new interface currently appears negligible.
> ---
>  arch/arm64/kernel/armv8_deprecated.c | 15 +++------------
>  1 file changed, 3 insertions(+), 12 deletions(-)
>
> diff --git a/arch/arm64/kernel/armv8_deprecated.c b/arch/arm64/kernel/armv8_deprecated.c
> index f0e6d71..e15eb2d 100644
> --- a/arch/arm64/kernel/armv8_deprecated.c
> +++ b/arch/arm64/kernel/armv8_deprecated.c
> @@ -228,15 +228,7 @@ static int emulation_proc_handler(struct ctl_table *table, int write,
>  	return ret;
>  }
>
> -static struct ctl_table ctl_abi[] = {
> -	{
> -		.procname = "abi",
> -		.mode = 0555,
> -	},
> -	{ }
> -};
> -
> -static void __init register_insn_emulation_sysctl(struct ctl_table *table)
> +static void __init register_insn_emulation_sysctl(void)
>  {
>  	unsigned long flags;
>  	int i = 0;
> @@ -262,8 +254,7 @@ static void __init register_insn_emulation_sysctl(struct ctl_table *table)
>  	}
>  	raw_spin_unlock_irqrestore(&insn_emulation_lock, flags);
>
> -	table->child = insns_sysctl;
> -	register_sysctl_table(table);
> +	register_sysctl("abi", insns_sysctl);
>  }
>
>  /*
> @@ -644,7 +635,7 @@ static int __init armv8_deprecated_init(void)
>  	cpuhp_setup_state_nocalls(CPUHP_AP_ARM64_ISNDEP_STARTING,
>  				  "arm64/isndep:starting",
>  				  run_all_insn_set_hw_mode, NULL);
> -	register_insn_emulation_sysctl(ctl_abi);
> +	register_insn_emulation_sysctl();
>
>  	return 0;
>  }


--
Alex Bennée

Re: [PATCH 05/27] arm64: fpsimd: Simplify uses of {set, clear}_ti_thread_flag()

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> The existing FPSIMD context switch code contains a couple of
> instances of {set,clear}_ti_thread(task_thread_info(task)).  Since
> there are thread flag manipulators that operate directly on
> task_struct, this verbosity isn't strictly needed.
>
> For consistency, this patch simplifies the affected calls.  This
> should have no impact on behaviour.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  arch/arm64/kernel/fpsimd.c | 6 ++----
>  1 file changed, 2 insertions(+), 4 deletions(-)
>
> diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> index 138fcfa..9c1f268e 100644
> --- a/arch/arm64/kernel/fpsimd.c
> +++ b/arch/arm64/kernel/fpsimd.c
> @@ -159,11 +159,9 @@ void fpsimd_thread_switch(struct task_struct *next)
>
>  		if (__this_cpu_read(fpsimd_last_state) == st
>  		    && st->cpu == smp_processor_id())
> -			clear_ti_thread_flag(task_thread_info(next),
> -					     TIF_FOREIGN_FPSTATE);
> +			clear_tsk_thread_flag(next, TIF_FOREIGN_FPSTATE);
>  		else
> -			set_ti_thread_flag(task_thread_info(next),
> -					   TIF_FOREIGN_FPSTATE);
> +			set_tsk_thread_flag(next, TIF_FOREIGN_FPSTATE);
>  	}
>  }

Re: [PATCH 05/27] arm64: fpsimd: Simplify uses of {set, clear}_ti_thread_flag()

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> The existing FPSIMD context switch code contains a couple of
> instances of {set,clear}_ti_thread(task_thread_info(task)).  Since
> there are thread flag manipulators that operate directly on
> task_struct, this verbosity isn't strictly needed.
>
> For consistency, this patch simplifies the affected calls.  This
> should have no impact on behaviour.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  arch/arm64/kernel/fpsimd.c | 6 ++----
>  1 file changed, 2 insertions(+), 4 deletions(-)
>
> diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> index 138fcfa..9c1f268e 100644
> --- a/arch/arm64/kernel/fpsimd.c
> +++ b/arch/arm64/kernel/fpsimd.c
> @@ -159,11 +159,9 @@ void fpsimd_thread_switch(struct task_struct *next)
>
>  		if (__this_cpu_read(fpsimd_last_state) == st
>  		    && st->cpu == smp_processor_id())
> -			clear_ti_thread_flag(task_thread_info(next),
> -					     TIF_FOREIGN_FPSTATE);
> +			clear_tsk_thread_flag(next, TIF_FOREIGN_FPSTATE);
>  		else
> -			set_ti_thread_flag(task_thread_info(next),
> -					   TIF_FOREIGN_FPSTATE);
> +			set_tsk_thread_flag(next, TIF_FOREIGN_FPSTATE);
>  	}
>  }


--
Alex Bennée

Re: [PATCH 06/27] arm64/sve: System register and exception syndrome definitions

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> The SVE architecture adds some system registers, ID register fields
> and a dedicated ESR exception class.
>
> This patch adds the appropriate definitions that will be needed by
> the kernel.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/include/asm/esr.h     |  3 ++-
>  arch/arm64/include/asm/kvm_arm.h |  1 +
>  arch/arm64/include/asm/sysreg.h  | 16 ++++++++++++++++
>  arch/arm64/kernel/traps.c        |  1 +
>  4 files changed, 20 insertions(+), 1 deletion(-)
>
> diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h
> index 8cabd57..813629e 100644
> --- a/arch/arm64/include/asm/esr.h
> +++ b/arch/arm64/include/asm/esr.h
> @@ -43,7 +43,8 @@
>  #define ESR_ELx_EC_HVC64	(0x16)
>  #define ESR_ELx_EC_SMC64	(0x17)
>  #define ESR_ELx_EC_SYS64	(0x18)
> -/* Unallocated EC: 0x19 - 0x1E */
> +#define ESR_ELx_EC_SVE		(0x19)
> +/* Unallocated EC: 0x1A - 0x1E */
>  #define ESR_ELx_EC_IMP_DEF	(0x1f)
>  #define ESR_ELx_EC_IABT_LOW	(0x20)
>  #define ESR_ELx_EC_IABT_CUR	(0x21)
> diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
> index 61d694c..dbf0537 100644
> --- a/arch/arm64/include/asm/kvm_arm.h
> +++ b/arch/arm64/include/asm/kvm_arm.h
> @@ -185,6 +185,7 @@
>  #define CPTR_EL2_TCPAC	(1 << 31)
>  #define CPTR_EL2_TTA	(1 << 20)
>  #define CPTR_EL2_TFP	(1 << CPTR_EL2_TFP_SHIFT)
> +#define CPTR_EL2_TZ	(1 << 8)
>  #define CPTR_EL2_DEFAULT	0x000033ff
>
>  /* Hyp Debug Configuration Register bits */
> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
> index 248339e..2d259e8 100644
> --- a/arch/arm64/include/asm/sysreg.h
> +++ b/arch/arm64/include/asm/sysreg.h
> @@ -145,6 +145,7 @@
>
>  #define SYS_ID_AA64PFR0_EL1		sys_reg(3, 0, 0, 4, 0)
>  #define SYS_ID_AA64PFR1_EL1		sys_reg(3, 0, 0, 4, 1)
> +#define SYS_ID_AA64ZFR0_EL1		sys_reg(3, 0, 0, 4, 4)
>
>  #define SYS_ID_AA64DFR0_EL1		sys_reg(3, 0, 0, 5, 0)
>  #define SYS_ID_AA64DFR1_EL1		sys_reg(3, 0, 0, 5, 1)
> @@ -160,6 +161,8 @@
>  #define SYS_ACTLR_EL1			sys_reg(3, 0, 1, 0, 1)
>  #define SYS_CPACR_EL1			sys_reg(3, 0, 1, 0, 2)
>
> +#define SYS_ZCR_EL1			sys_reg(3, 0, 1, 2, 0)
> +

I'll have to take these on trust. They are mentioned in both the ARM ARM
and the SVE supplement but I can't see any actual definitions of them.

>  #define SYS_TTBR0_EL1			sys_reg(3, 0, 2, 0, 0)
>  #define SYS_TTBR1_EL1			sys_reg(3, 0, 2, 0, 1)
>  #define SYS_TCR_EL1			sys_reg(3, 0, 2, 0, 2)
> @@ -250,6 +253,8 @@
>
>  #define SYS_PMCCFILTR_EL0		sys_reg (3, 3, 14, 15, 7)
>
> +#define SYS_ZCR_EL2			sys_reg(3, 4, 1, 2, 0)
> +
>  #define SYS_DACR32_EL2			sys_reg(3, 4, 3, 0, 0)
>  #define SYS_IFSR32_EL2			sys_reg(3, 4, 5, 0, 1)
>  #define SYS_FPEXC32_EL2			sys_reg(3, 4, 5, 3, 0)
> @@ -331,6 +336,7 @@
>  #define ID_AA64ISAR1_JSCVT_SHIFT	12
>
>  /* id_aa64pfr0 */
> +#define ID_AA64PFR0_SVE_SHIFT		32
>  #define ID_AA64PFR0_GIC_SHIFT		24
>  #define ID_AA64PFR0_ASIMD_SHIFT		20
>  #define ID_AA64PFR0_FP_SHIFT		16
> @@ -339,6 +345,7 @@
>  #define ID_AA64PFR0_EL1_SHIFT		4
>  #define ID_AA64PFR0_EL0_SHIFT		0
>
> +#define ID_AA64PFR0_SVE			0x1
>  #define ID_AA64PFR0_FP_NI		0xf
>  #define ID_AA64PFR0_FP_SUPPORTED	0x0
>  #define ID_AA64PFR0_ASIMD_NI		0xf
> @@ -440,6 +447,15 @@
>  #endif
>
>
> +#define ZCR_ELx_LEN_SHIFT	0
> +#define ZCR_ELx_LEN_SIZE	9
> +#define ZCR_ELx_LEN_MASK	0x1ff
> +
> +#define CPACR_EL1_ZEN_EL1EN	(1 << 16)
> +#define CPACR_EL1_ZEN_EL0EN	(1 << 17)
> +#define CPACR_EL1_ZEN		(CPACR_EL1_ZEN_EL1EN |
> CPACR_EL1_ZEN_EL0EN)

This is a little weird as it is a 2 bit field in which 00 and 11 are not
simply the sum of their bits. If the code wrote CPACR_EL1_ZEN_EL0EN |
CPACR_EL1_ZEN_EL1EN to the CPACR_EL1 you wouldn't get the expected behaviour.

> +
> +
>  /* Safe value for MPIDR_EL1: Bit31:RES1, Bit30:U:0, Bit24:MT:0 */
>  #define SYS_MPIDR_SAFE_VAL		(1UL << 31)
>
> diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
> index 0f047e9..8964795 100644
> --- a/arch/arm64/kernel/traps.c
> +++ b/arch/arm64/kernel/traps.c
> @@ -621,6 +621,7 @@ static const char *esr_class_str[] = {
>  	[ESR_ELx_EC_HVC64]		= "HVC (AArch64)",
>  	[ESR_ELx_EC_SMC64]		= "SMC (AArch64)",
>  	[ESR_ELx_EC_SYS64]		= "MSR/MRS (AArch64)",
> +	[ESR_ELx_EC_SVE]		= "SVE",
>  	[ESR_ELx_EC_IMP_DEF]		= "EL3 IMP DEF",
>  	[ESR_ELx_EC_IABT_LOW]		= "IABT (lower EL)",
>  	[ESR_ELx_EC_IABT_CUR]		= "IABT (current EL)",


--
Alex Bennée

Re: [PATCH 06/27] arm64/sve: System register and exception syndrome definitions

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> The SVE architecture adds some system registers, ID register fields
> and a dedicated ESR exception class.
>
> This patch adds the appropriate definitions that will be needed by
> the kernel.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/include/asm/esr.h     |  3 ++-
>  arch/arm64/include/asm/kvm_arm.h |  1 +
>  arch/arm64/include/asm/sysreg.h  | 16 ++++++++++++++++
>  arch/arm64/kernel/traps.c        |  1 +
>  4 files changed, 20 insertions(+), 1 deletion(-)
>
> diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h
> index 8cabd57..813629e 100644
> --- a/arch/arm64/include/asm/esr.h
> +++ b/arch/arm64/include/asm/esr.h
> @@ -43,7 +43,8 @@
>  #define ESR_ELx_EC_HVC64	(0x16)
>  #define ESR_ELx_EC_SMC64	(0x17)
>  #define ESR_ELx_EC_SYS64	(0x18)
> -/* Unallocated EC: 0x19 - 0x1E */
> +#define ESR_ELx_EC_SVE		(0x19)
> +/* Unallocated EC: 0x1A - 0x1E */
>  #define ESR_ELx_EC_IMP_DEF	(0x1f)
>  #define ESR_ELx_EC_IABT_LOW	(0x20)
>  #define ESR_ELx_EC_IABT_CUR	(0x21)
> diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
> index 61d694c..dbf0537 100644
> --- a/arch/arm64/include/asm/kvm_arm.h
> +++ b/arch/arm64/include/asm/kvm_arm.h
> @@ -185,6 +185,7 @@
>  #define CPTR_EL2_TCPAC	(1 << 31)
>  #define CPTR_EL2_TTA	(1 << 20)
>  #define CPTR_EL2_TFP	(1 << CPTR_EL2_TFP_SHIFT)
> +#define CPTR_EL2_TZ	(1 << 8)
>  #define CPTR_EL2_DEFAULT	0x000033ff
>
>  /* Hyp Debug Configuration Register bits */
> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
> index 248339e..2d259e8 100644
> --- a/arch/arm64/include/asm/sysreg.h
> +++ b/arch/arm64/include/asm/sysreg.h
> @@ -145,6 +145,7 @@
>
>  #define SYS_ID_AA64PFR0_EL1		sys_reg(3, 0, 0, 4, 0)
>  #define SYS_ID_AA64PFR1_EL1		sys_reg(3, 0, 0, 4, 1)
> +#define SYS_ID_AA64ZFR0_EL1		sys_reg(3, 0, 0, 4, 4)
>
>  #define SYS_ID_AA64DFR0_EL1		sys_reg(3, 0, 0, 5, 0)
>  #define SYS_ID_AA64DFR1_EL1		sys_reg(3, 0, 0, 5, 1)
> @@ -160,6 +161,8 @@
>  #define SYS_ACTLR_EL1			sys_reg(3, 0, 1, 0, 1)
>  #define SYS_CPACR_EL1			sys_reg(3, 0, 1, 0, 2)
>
> +#define SYS_ZCR_EL1			sys_reg(3, 0, 1, 2, 0)
> +

I'll have to take these on trust. They are mentioned in both the ARM ARM
and the SVE supplement but I can't see any actual definitions of them.

>  #define SYS_TTBR0_EL1			sys_reg(3, 0, 2, 0, 0)
>  #define SYS_TTBR1_EL1			sys_reg(3, 0, 2, 0, 1)
>  #define SYS_TCR_EL1			sys_reg(3, 0, 2, 0, 2)
> @@ -250,6 +253,8 @@
>
>  #define SYS_PMCCFILTR_EL0		sys_reg (3, 3, 14, 15, 7)
>
> +#define SYS_ZCR_EL2			sys_reg(3, 4, 1, 2, 0)
> +
>  #define SYS_DACR32_EL2			sys_reg(3, 4, 3, 0, 0)
>  #define SYS_IFSR32_EL2			sys_reg(3, 4, 5, 0, 1)
>  #define SYS_FPEXC32_EL2			sys_reg(3, 4, 5, 3, 0)
> @@ -331,6 +336,7 @@
>  #define ID_AA64ISAR1_JSCVT_SHIFT	12
>
>  /* id_aa64pfr0 */
> +#define ID_AA64PFR0_SVE_SHIFT		32
>  #define ID_AA64PFR0_GIC_SHIFT		24
>  #define ID_AA64PFR0_ASIMD_SHIFT		20
>  #define ID_AA64PFR0_FP_SHIFT		16
> @@ -339,6 +345,7 @@
>  #define ID_AA64PFR0_EL1_SHIFT		4
>  #define ID_AA64PFR0_EL0_SHIFT		0
>
> +#define ID_AA64PFR0_SVE			0x1
>  #define ID_AA64PFR0_FP_NI		0xf
>  #define ID_AA64PFR0_FP_SUPPORTED	0x0
>  #define ID_AA64PFR0_ASIMD_NI		0xf
> @@ -440,6 +447,15 @@
>  #endif
>
>
> +#define ZCR_ELx_LEN_SHIFT	0
> +#define ZCR_ELx_LEN_SIZE	9
> +#define ZCR_ELx_LEN_MASK	0x1ff
> +
> +#define CPACR_EL1_ZEN_EL1EN	(1 << 16)
> +#define CPACR_EL1_ZEN_EL0EN	(1 << 17)
> +#define CPACR_EL1_ZEN		(CPACR_EL1_ZEN_EL1EN |
> CPACR_EL1_ZEN_EL0EN)

This is a little weird as it is a 2 bit field in which 00 and 11 are not
simply the sum of their bits. If the code wrote CPACR_EL1_ZEN_EL0EN |
CPACR_EL1_ZEN_EL1EN to the CPACR_EL1 you wouldn't get the expected behaviour.

> +
> +
>  /* Safe value for MPIDR_EL1: Bit31:RES1, Bit30:U:0, Bit24:MT:0 */
>  #define SYS_MPIDR_SAFE_VAL		(1UL << 31)
>
> diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
> index 0f047e9..8964795 100644
> --- a/arch/arm64/kernel/traps.c
> +++ b/arch/arm64/kernel/traps.c
> @@ -621,6 +621,7 @@ static const char *esr_class_str[] = {
>  	[ESR_ELx_EC_HVC64]		= "HVC (AArch64)",
>  	[ESR_ELx_EC_SMC64]		= "SMC (AArch64)",
>  	[ESR_ELx_EC_SYS64]		= "MSR/MRS (AArch64)",
> +	[ESR_ELx_EC_SVE]		= "SVE",
>  	[ESR_ELx_EC_IMP_DEF]		= "EL3 IMP DEF",
>  	[ESR_ELx_EC_IABT_LOW]		= "IABT (lower EL)",
>  	[ESR_ELx_EC_IABT_CUR]		= "IABT (current EL)",


--
Alex Bennée

Re: [PATCH 07/27] arm64/sve: Low-level SVE architectural state manipulation functions

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> Manipulating the SVE architectural state, including the vector and
> predicate registers, first-fault register and the vector length,
> requires the use of dedicated instructions added by SVE.
>
> This patch adds suitable assembly functions for saving and
> restoring the SVE registers and querying the vector length.
> Setting of the vector length is done as part of register restore.
>
> Since people building kernels may not all get an SVE-enabled
> toolchain for a while, this patch uses macros that generate
> explicit opcodes in place of assembler mnemonics.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/include/asm/fpsimd.h       |   5 ++
>  arch/arm64/include/asm/fpsimdmacros.h | 137 ++++++++++++++++++++++++++++++++++
>  arch/arm64/kernel/entry-fpsimd.S      |  17 +++++
>  3 files changed, 159 insertions(+)
>
> diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
> index 410c481..026a7c7 100644
> --- a/arch/arm64/include/asm/fpsimd.h
> +++ b/arch/arm64/include/asm/fpsimd.h
> @@ -67,6 +67,11 @@ extern void fpsimd_update_current_state(struct fpsimd_state *state);
>
>  extern void fpsimd_flush_task_state(struct task_struct *target);
>
> +extern void sve_save_state(void *state, u32 *pfpsr);
> +extern void sve_load_state(void const *state, u32 const *pfpsr,
> +			   unsigned long vq_minus_1);
> +extern unsigned int sve_get_vl(void);
> +
>  /* For use by EFI runtime services calls only */
>  extern void __efi_fpsimd_begin(void);
>  extern void __efi_fpsimd_end(void);
> diff --git a/arch/arm64/include/asm/fpsimdmacros.h b/arch/arm64/include/asm/fpsimdmacros.h
> index 0f5fdd3..5023064 100644
> --- a/arch/arm64/include/asm/fpsimdmacros.h
> +++ b/arch/arm64/include/asm/fpsimdmacros.h
> @@ -75,3 +75,140 @@
>  	ldr	w\tmpnr, [\state, #16 * 2 + 4]
>  	fpsimd_restore_fpcr x\tmpnr, \state
>  .endm
> +
> +.macro _check_general_reg nr
> +	.if (\nr) < 0 || (\nr) > 30
> +		.error "Bad register number \nr."
> +	.endif
> +.endm
> +
> +.macro _sve_check_zreg znr
> +	.if (\znr) < 0 || (\znr) > 31
> +		.error "Bad Scalable Vector Extension vector register number \znr."
> +	.endif
> +.endm
> +
> +.macro _sve_check_preg pnr
> +	.if (\pnr) < 0 || (\pnr) > 15
> +		.error "Bad Scalable Vector Extension predicate register number \pnr."
> +	.endif
> +.endm
> +
> +.macro _check_num n, min, max
> +	.if (\n) < (\min) || (\n) > (\max)
> +		.error "Number \n out of range [\min,\max]"
> +	.endif
> +.endm
> +
> +.macro _sve_str_v nz, nxbase, offset=0
> +	_sve_check_zreg \nz
> +	_check_general_reg \nxbase
> +	_check_num (\offset), -0x100, 0xff
> +	.inst	0xe5804000			\
> +		| (\nz)				\
> +		| ((\nxbase) << 5)		\
> +		| (((\offset) & 7) << 10)	\
> +		| (((\offset) & 0x1f8) << 13)
> +.endm

Can we have references to the ARM ARM titles (the section numbers keep
changing) for each hand-hacked instruction please. Otherwise it is
impossible to verify each one.

> +
> +.macro _sve_ldr_v nz, nxbase, offset=0
> +	_sve_check_zreg \nz
> +	_check_general_reg \nxbase
> +	_check_num (\offset), -0x100, 0xff
> +	.inst	0x85804000			\
> +		| (\nz)				\
> +		| ((\nxbase) << 5)		\
> +		| (((\offset) & 7) << 10)	\
> +		| (((\offset) & 0x1f8) << 13)
> +.endm
> +
> +.macro _sve_str_p np, nxbase, offset=0
> +	_sve_check_preg \np
> +	_check_general_reg \nxbase
> +	_check_num (\offset), -0x100, 0xff
> +	.inst	0xe5800000			\
> +		| (\np)				\
> +		| ((\nxbase) << 5)		\
> +		| (((\offset) & 7) << 10)	\
> +		| (((\offset) & 0x1f8) << 13)
> +.endm
> +
> +.macro _sve_ldr_p np, nxbase, offset=0
> +	_sve_check_preg \np
> +	_check_general_reg \nxbase
> +	_check_num (\offset), -0x100, 0xff
> +	.inst	0x85800000			\
> +		| (\np)				\
> +		| ((\nxbase) << 5)		\
> +		| (((\offset) & 7) << 10)	\
> +		| (((\offset) & 0x1f8) << 13)
> +.endm
> +
> +.macro _sve_rdvl nx, imm
> +	_check_general_reg \nx
> +	_check_num (\imm), -0x20, 0x1f
> +	.inst	0x04bf5000			\
> +		| (\nx)				\
> +		| (((\imm) & 0x3f) << 5)
> +.endm
> +
> +.macro _sve_rdffr np
> +	_sve_check_preg \np
> +	.inst	0x2519f000			\
> +		| (\np)
> +.endm
> +
> +.macro _sve_wrffr np
> +	_sve_check_preg \np
> +	.inst	0x25289000			\
> +		| ((\np) << 5)
> +.endm
> +
> +.macro __for from:req, to:req
> +	.if (\from) == (\to)
> +		_for__body \from
> +	.else
> +		__for \from, (\from) + ((\to) - (\from)) / 2
> +		__for (\from) + ((\to) - (\from)) / 2 + 1, \to
> +	.endif
> +.endm
> +
> +.macro _for var:req, from:req, to:req, insn:vararg
> +	.macro _for__body \var:req
> +		\insn
> +	.endm
> +
> +	__for \from, \to
> +
> +	.purgem _for__body
> +.endm
> +
> +.macro sve_save nxbase, xpfpsr, nxtmp
> + _for n, 0, 31,	_sve_str_v	\n, \nxbase, \n - 34
> + _for n, 0, 15,	_sve_str_p	\n, \nxbase, \n - 16
> +		_sve_rdffr	0
> +		_sve_str_p	0, \nxbase
> +		_sve_ldr_p	0, \nxbase, -16
> +
> +		mrs		x\nxtmp, fpsr
> +		str		w\nxtmp, [\xpfpsr]
> +		mrs		x\nxtmp, fpcr
> +		str		w\nxtmp, [\xpfpsr, #4]
> +.endm
> +
> +.macro sve_load nxbase, xpfpsr, xvqminus1, nxtmp
> +		mrs_s		x\nxtmp, SYS_ZCR_EL1
> +		bic		x\nxtmp, x\nxtmp, ZCR_ELx_LEN_MASK
> +		orr		x\nxtmp, x\nxtmp, \xvqminus1
> +		msr_s		SYS_ZCR_EL1, x\nxtmp	// self-synchronising
> +
> + _for n, 0, 31,	_sve_ldr_v	\n, \nxbase, \n - 34
> +		_sve_ldr_p	0, \nxbase
> +		_sve_wrffr	0
> + _for n, 0, 15,	_sve_ldr_p	\n, \nxbase, \n - 16
> +
> +		ldr		w\nxtmp, [\xpfpsr]
> +		msr		fpsr, x\nxtmp
> +		ldr		w\nxtmp, [\xpfpsr, #4]
> +		msr		fpcr, x\nxtmp
> +.endm
> diff --git a/arch/arm64/kernel/entry-fpsimd.S b/arch/arm64/kernel/entry-fpsimd.S
> index 6a27cd6..73f17bf 100644
> --- a/arch/arm64/kernel/entry-fpsimd.S
> +++ b/arch/arm64/kernel/entry-fpsimd.S
> @@ -41,3 +41,20 @@ ENTRY(fpsimd_load_state)
>  	fpsimd_restore x0, 8
>  	ret
>  ENDPROC(fpsimd_load_state)
> +
> +#ifdef CONFIG_ARM64_SVE
> +ENTRY(sve_save_state)
> +	sve_save 0, x1, 2
> +	ret
> +ENDPROC(sve_save_state)
> +
> +ENTRY(sve_load_state)
> +	sve_load 0, x1, x2, 3
> +	ret
> +ENDPROC(sve_load_state)
> +
> +ENTRY(sve_get_vl)
> +	_sve_rdvl	0, 1
> +	ret
> +ENDPROC(sve_get_vl)
> +#endif /* CONFIG_ARM64_SVE */


--
Alex Bennée
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [PATCH 07/27] arm64/sve: Low-level SVE architectural state manipulation functions

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> Manipulating the SVE architectural state, including the vector and
> predicate registers, first-fault register and the vector length,
> requires the use of dedicated instructions added by SVE.
>
> This patch adds suitable assembly functions for saving and
> restoring the SVE registers and querying the vector length.
> Setting of the vector length is done as part of register restore.
>
> Since people building kernels may not all get an SVE-enabled
> toolchain for a while, this patch uses macros that generate
> explicit opcodes in place of assembler mnemonics.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/include/asm/fpsimd.h       |   5 ++
>  arch/arm64/include/asm/fpsimdmacros.h | 137 ++++++++++++++++++++++++++++++++++
>  arch/arm64/kernel/entry-fpsimd.S      |  17 +++++
>  3 files changed, 159 insertions(+)
>
> diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
> index 410c481..026a7c7 100644
> --- a/arch/arm64/include/asm/fpsimd.h
> +++ b/arch/arm64/include/asm/fpsimd.h
> @@ -67,6 +67,11 @@ extern void fpsimd_update_current_state(struct fpsimd_state *state);
>
>  extern void fpsimd_flush_task_state(struct task_struct *target);
>
> +extern void sve_save_state(void *state, u32 *pfpsr);
> +extern void sve_load_state(void const *state, u32 const *pfpsr,
> +			   unsigned long vq_minus_1);
> +extern unsigned int sve_get_vl(void);
> +
>  /* For use by EFI runtime services calls only */
>  extern void __efi_fpsimd_begin(void);
>  extern void __efi_fpsimd_end(void);
> diff --git a/arch/arm64/include/asm/fpsimdmacros.h b/arch/arm64/include/asm/fpsimdmacros.h
> index 0f5fdd3..5023064 100644
> --- a/arch/arm64/include/asm/fpsimdmacros.h
> +++ b/arch/arm64/include/asm/fpsimdmacros.h
> @@ -75,3 +75,140 @@
>  	ldr	w\tmpnr, [\state, #16 * 2 + 4]
>  	fpsimd_restore_fpcr x\tmpnr, \state
>  .endm
> +
> +.macro _check_general_reg nr
> +	.if (\nr) < 0 || (\nr) > 30
> +		.error "Bad register number \nr."
> +	.endif
> +.endm
> +
> +.macro _sve_check_zreg znr
> +	.if (\znr) < 0 || (\znr) > 31
> +		.error "Bad Scalable Vector Extension vector register number \znr."
> +	.endif
> +.endm
> +
> +.macro _sve_check_preg pnr
> +	.if (\pnr) < 0 || (\pnr) > 15
> +		.error "Bad Scalable Vector Extension predicate register number \pnr."
> +	.endif
> +.endm
> +
> +.macro _check_num n, min, max
> +	.if (\n) < (\min) || (\n) > (\max)
> +		.error "Number \n out of range [\min,\max]"
> +	.endif
> +.endm
> +
> +.macro _sve_str_v nz, nxbase, offset=0
> +	_sve_check_zreg \nz
> +	_check_general_reg \nxbase
> +	_check_num (\offset), -0x100, 0xff
> +	.inst	0xe5804000			\
> +		| (\nz)				\
> +		| ((\nxbase) << 5)		\
> +		| (((\offset) & 7) << 10)	\
> +		| (((\offset) & 0x1f8) << 13)
> +.endm

Can we have references to the ARM ARM titles (the section numbers keep
changing) for each hand-hacked instruction please. Otherwise it is
impossible to verify each one.

> +
> +.macro _sve_ldr_v nz, nxbase, offset=0
> +	_sve_check_zreg \nz
> +	_check_general_reg \nxbase
> +	_check_num (\offset), -0x100, 0xff
> +	.inst	0x85804000			\
> +		| (\nz)				\
> +		| ((\nxbase) << 5)		\
> +		| (((\offset) & 7) << 10)	\
> +		| (((\offset) & 0x1f8) << 13)
> +.endm
> +
> +.macro _sve_str_p np, nxbase, offset=0
> +	_sve_check_preg \np
> +	_check_general_reg \nxbase
> +	_check_num (\offset), -0x100, 0xff
> +	.inst	0xe5800000			\
> +		| (\np)				\
> +		| ((\nxbase) << 5)		\
> +		| (((\offset) & 7) << 10)	\
> +		| (((\offset) & 0x1f8) << 13)
> +.endm
> +
> +.macro _sve_ldr_p np, nxbase, offset=0
> +	_sve_check_preg \np
> +	_check_general_reg \nxbase
> +	_check_num (\offset), -0x100, 0xff
> +	.inst	0x85800000			\
> +		| (\np)				\
> +		| ((\nxbase) << 5)		\
> +		| (((\offset) & 7) << 10)	\
> +		| (((\offset) & 0x1f8) << 13)
> +.endm
> +
> +.macro _sve_rdvl nx, imm
> +	_check_general_reg \nx
> +	_check_num (\imm), -0x20, 0x1f
> +	.inst	0x04bf5000			\
> +		| (\nx)				\
> +		| (((\imm) & 0x3f) << 5)
> +.endm
> +
> +.macro _sve_rdffr np
> +	_sve_check_preg \np
> +	.inst	0x2519f000			\
> +		| (\np)
> +.endm
> +
> +.macro _sve_wrffr np
> +	_sve_check_preg \np
> +	.inst	0x25289000			\
> +		| ((\np) << 5)
> +.endm
> +
> +.macro __for from:req, to:req
> +	.if (\from) == (\to)
> +		_for__body \from
> +	.else
> +		__for \from, (\from) + ((\to) - (\from)) / 2
> +		__for (\from) + ((\to) - (\from)) / 2 + 1, \to
> +	.endif
> +.endm
> +
> +.macro _for var:req, from:req, to:req, insn:vararg
> +	.macro _for__body \var:req
> +		\insn
> +	.endm
> +
> +	__for \from, \to
> +
> +	.purgem _for__body
> +.endm
> +
> +.macro sve_save nxbase, xpfpsr, nxtmp
> + _for n, 0, 31,	_sve_str_v	\n, \nxbase, \n - 34
> + _for n, 0, 15,	_sve_str_p	\n, \nxbase, \n - 16
> +		_sve_rdffr	0
> +		_sve_str_p	0, \nxbase
> +		_sve_ldr_p	0, \nxbase, -16
> +
> +		mrs		x\nxtmp, fpsr
> +		str		w\nxtmp, [\xpfpsr]
> +		mrs		x\nxtmp, fpcr
> +		str		w\nxtmp, [\xpfpsr, #4]
> +.endm
> +
> +.macro sve_load nxbase, xpfpsr, xvqminus1, nxtmp
> +		mrs_s		x\nxtmp, SYS_ZCR_EL1
> +		bic		x\nxtmp, x\nxtmp, ZCR_ELx_LEN_MASK
> +		orr		x\nxtmp, x\nxtmp, \xvqminus1
> +		msr_s		SYS_ZCR_EL1, x\nxtmp	// self-synchronising
> +
> + _for n, 0, 31,	_sve_ldr_v	\n, \nxbase, \n - 34
> +		_sve_ldr_p	0, \nxbase
> +		_sve_wrffr	0
> + _for n, 0, 15,	_sve_ldr_p	\n, \nxbase, \n - 16
> +
> +		ldr		w\nxtmp, [\xpfpsr]
> +		msr		fpsr, x\nxtmp
> +		ldr		w\nxtmp, [\xpfpsr, #4]
> +		msr		fpcr, x\nxtmp
> +.endm
> diff --git a/arch/arm64/kernel/entry-fpsimd.S b/arch/arm64/kernel/entry-fpsimd.S
> index 6a27cd6..73f17bf 100644
> --- a/arch/arm64/kernel/entry-fpsimd.S
> +++ b/arch/arm64/kernel/entry-fpsimd.S
> @@ -41,3 +41,20 @@ ENTRY(fpsimd_load_state)
>  	fpsimd_restore x0, 8
>  	ret
>  ENDPROC(fpsimd_load_state)
> +
> +#ifdef CONFIG_ARM64_SVE
> +ENTRY(sve_save_state)
> +	sve_save 0, x1, 2
> +	ret
> +ENDPROC(sve_save_state)
> +
> +ENTRY(sve_load_state)
> +	sve_load 0, x1, x2, 3
> +	ret
> +ENDPROC(sve_load_state)
> +
> +ENTRY(sve_get_vl)
> +	_sve_rdvl	0, 1
> +	ret
> +ENDPROC(sve_get_vl)
> +#endif /* CONFIG_ARM64_SVE */


--
Alex Bennée

Re: [PATCH 08/27] arm64/sve: Kconfig update and conditional compilation support

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> This patch adds CONFIG_ARM64_SVE to control building of SVE support
> into the kernel, and adds a stub predicate system_supports_sve() to
> control conditional compilation and runtime SVE support.
>
> system_supports_sve() just returns false for now: it will be
> replaced with a non-trivial implementation in a later patch, once
> SVE support is complete enough to be enabled safely.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  arch/arm64/Kconfig                  | 11 +++++++++++
>  arch/arm64/include/asm/cpufeature.h |  5 +++++
>  2 files changed, 16 insertions(+)
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index fdc302e..fbf7a5c 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -962,6 +962,17 @@ config ARM64_UAO
>
>  endmenu
>
> +config ARM64_SVE
> +	bool "ARM Scalable Vector Extension support"
> +	default y
> +	help
> +	  The Scalable Vector Extension (SVE) is an extension to the AArch64
> +	  execution state which complements and extends the SIMD functionality
> +	  of the base architecture to support much larger vectors and to enable
> +	  additional vectorisation opportunities.
> +
> +	  To enable use of this extension on CPUs that implement it, say Y.
> +
>  config ARM64_MODULE_CMODEL_LARGE
>  	bool
>
> diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
> index 428ee1f..4ea3441 100644
> --- a/arch/arm64/include/asm/cpufeature.h
> +++ b/arch/arm64/include/asm/cpufeature.h
> @@ -262,6 +262,11 @@ static inline bool system_uses_ttbr0_pan(void)
>  		!cpus_have_const_cap(ARM64_HAS_PAN);
>  }
>
> +static inline bool system_supports_sve(void)
> +{
> +	return false;
> +}
> +
>  #endif /* __ASSEMBLY__ */
>
>  #endif

Re: [PATCH 08/27] arm64/sve: Kconfig update and conditional compilation support

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> This patch adds CONFIG_ARM64_SVE to control building of SVE support
> into the kernel, and adds a stub predicate system_supports_sve() to
> control conditional compilation and runtime SVE support.
>
> system_supports_sve() just returns false for now: it will be
> replaced with a non-trivial implementation in a later patch, once
> SVE support is complete enough to be enabled safely.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> ---
>  arch/arm64/Kconfig                  | 11 +++++++++++
>  arch/arm64/include/asm/cpufeature.h |  5 +++++
>  2 files changed, 16 insertions(+)
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index fdc302e..fbf7a5c 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -962,6 +962,17 @@ config ARM64_UAO
>
>  endmenu
>
> +config ARM64_SVE
> +	bool "ARM Scalable Vector Extension support"
> +	default y
> +	help
> +	  The Scalable Vector Extension (SVE) is an extension to the AArch64
> +	  execution state which complements and extends the SIMD functionality
> +	  of the base architecture to support much larger vectors and to enable
> +	  additional vectorisation opportunities.
> +
> +	  To enable use of this extension on CPUs that implement it, say Y.
> +
>  config ARM64_MODULE_CMODEL_LARGE
>  	bool
>
> diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
> index 428ee1f..4ea3441 100644
> --- a/arch/arm64/include/asm/cpufeature.h
> +++ b/arch/arm64/include/asm/cpufeature.h
> @@ -262,6 +262,11 @@ static inline bool system_uses_ttbr0_pan(void)
>  		!cpus_have_const_cap(ARM64_HAS_PAN);
>  }
>
> +static inline bool system_supports_sve(void)
> +{
> +	return false;
> +}
> +
>  #endif /* __ASSEMBLY__ */
>
>  #endif


--
Alex Bennée

Re: [PATCH 06/27] arm64/sve: System register and exception syndrome definitions

[Alex Bennée]

Alex Bennée <alex.bennee@linaro.org> writes:

> Dave Martin <Dave.Martin@arm.com> writes:
>
>> The SVE architecture adds some system registers, ID register fields
>> and a dedicated ESR exception class.
>>
>> This patch adds the appropriate definitions that will be needed by
>> the kernel.
>>
>> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
>> ---
>>  arch/arm64/include/asm/esr.h     |  3 ++-
>>  arch/arm64/include/asm/kvm_arm.h |  1 +
>>  arch/arm64/include/asm/sysreg.h  | 16 ++++++++++++++++
>>  arch/arm64/kernel/traps.c        |  1 +
>>  4 files changed, 20 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h
>> index 8cabd57..813629e 100644
>> --- a/arch/arm64/include/asm/esr.h
>> +++ b/arch/arm64/include/asm/esr.h
>> @@ -43,7 +43,8 @@
>>  #define ESR_ELx_EC_HVC64	(0x16)
>>  #define ESR_ELx_EC_SMC64	(0x17)
>>  #define ESR_ELx_EC_SYS64	(0x18)
>> -/* Unallocated EC: 0x19 - 0x1E */
>> +#define ESR_ELx_EC_SVE		(0x19)
>> +/* Unallocated EC: 0x1A - 0x1E */
>>  #define ESR_ELx_EC_IMP_DEF	(0x1f)
>>  #define ESR_ELx_EC_IABT_LOW	(0x20)
>>  #define ESR_ELx_EC_IABT_CUR	(0x21)
>> diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
>> index 61d694c..dbf0537 100644
>> --- a/arch/arm64/include/asm/kvm_arm.h
>> +++ b/arch/arm64/include/asm/kvm_arm.h
>> @@ -185,6 +185,7 @@
>>  #define CPTR_EL2_TCPAC	(1 << 31)
>>  #define CPTR_EL2_TTA	(1 << 20)
>>  #define CPTR_EL2_TFP	(1 << CPTR_EL2_TFP_SHIFT)
>> +#define CPTR_EL2_TZ	(1 << 8)
>>  #define CPTR_EL2_DEFAULT	0x000033ff
>>
>>  /* Hyp Debug Configuration Register bits */
>> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
>> index 248339e..2d259e8 100644
>> --- a/arch/arm64/include/asm/sysreg.h
>> +++ b/arch/arm64/include/asm/sysreg.h
>> @@ -145,6 +145,7 @@
>>
>>  #define SYS_ID_AA64PFR0_EL1		sys_reg(3, 0, 0, 4, 0)
>>  #define SYS_ID_AA64PFR1_EL1		sys_reg(3, 0, 0, 4, 1)
>> +#define SYS_ID_AA64ZFR0_EL1		sys_reg(3, 0, 0, 4, 4)
>>
>>  #define SYS_ID_AA64DFR0_EL1		sys_reg(3, 0, 0, 5, 0)
>>  #define SYS_ID_AA64DFR1_EL1		sys_reg(3, 0, 0, 5, 1)
>> @@ -160,6 +161,8 @@
>>  #define SYS_ACTLR_EL1			sys_reg(3, 0, 1, 0, 1)
>>  #define SYS_CPACR_EL1			sys_reg(3, 0, 1, 0, 2)
>>
>> +#define SYS_ZCR_EL1			sys_reg(3, 0, 1, 2, 0)
>> +
>
> I'll have to take these on trust. They are mentioned in both the ARM ARM
> and the SVE supplement but I can't see any actual definitions of them.
>
>>  #define SYS_TTBR0_EL1			sys_reg(3, 0, 2, 0, 0)
>>  #define SYS_TTBR1_EL1			sys_reg(3, 0, 2, 0, 1)
>>  #define SYS_TCR_EL1			sys_reg(3, 0, 2, 0, 2)
>> @@ -250,6 +253,8 @@
>>
>>  #define SYS_PMCCFILTR_EL0		sys_reg (3, 3, 14, 15, 7)
>>
>> +#define SYS_ZCR_EL2			sys_reg(3, 4, 1, 2, 0)
>> +

OK no I'm working directly from the unpacked ZIP file with the rest of
the details I think this should be:

  #define SYS_ZCR_EL2			sys_reg(3, 5, 1, 2, 0)

e.g. op1 = 101 / 5

>>  #define SYS_DACR32_EL2			sys_reg(3, 4, 3, 0, 0)
>>  #define SYS_IFSR32_EL2			sys_reg(3, 4, 5, 0, 1)
>>  #define SYS_FPEXC32_EL2			sys_reg(3, 4, 5, 3, 0)
>> @@ -331,6 +336,7 @@
>>  #define ID_AA64ISAR1_JSCVT_SHIFT	12
>>
>>  /* id_aa64pfr0 */
>> +#define ID_AA64PFR0_SVE_SHIFT		32
>>  #define ID_AA64PFR0_GIC_SHIFT		24
>>  #define ID_AA64PFR0_ASIMD_SHIFT		20
>>  #define ID_AA64PFR0_FP_SHIFT		16
>> @@ -339,6 +345,7 @@
>>  #define ID_AA64PFR0_EL1_SHIFT		4
>>  #define ID_AA64PFR0_EL0_SHIFT		0
>>
>> +#define ID_AA64PFR0_SVE			0x1
>>  #define ID_AA64PFR0_FP_NI		0xf
>>  #define ID_AA64PFR0_FP_SUPPORTED	0x0
>>  #define ID_AA64PFR0_ASIMD_NI		0xf
>> @@ -440,6 +447,15 @@
>>  #endif
>>
>>
>> +#define ZCR_ELx_LEN_SHIFT	0
>> +#define ZCR_ELx_LEN_SIZE	9
>> +#define ZCR_ELx_LEN_MASK	0x1ff
>> +

LEN should be 0/4/0xf

LEN, bits [3:0]

Constrains the scalable vector register length for EL1 and EL0 to (LEN+1)x128 bits.


>> +#define CPACR_EL1_ZEN_EL1EN	(1 << 16)
>> +#define CPACR_EL1_ZEN_EL0EN	(1 << 17)
>> +#define CPACR_EL1_ZEN		(CPACR_EL1_ZEN_EL1EN |
>> CPACR_EL1_ZEN_EL0EN)
>
> This is a little weird as it is a 2 bit field in which 00 and 11 are not
> simply the sum of their bits. If the code wrote CPACR_EL1_ZEN_EL0EN |
> CPACR_EL1_ZEN_EL1EN to the CPACR_EL1 you wouldn't get the expected behaviour.
>
>> +
>> +
>>  /* Safe value for MPIDR_EL1: Bit31:RES1, Bit30:U:0, Bit24:MT:0 */
>>  #define SYS_MPIDR_SAFE_VAL		(1UL << 31)
>>
>> diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
>> index 0f047e9..8964795 100644
>> --- a/arch/arm64/kernel/traps.c
>> +++ b/arch/arm64/kernel/traps.c
>> @@ -621,6 +621,7 @@ static const char *esr_class_str[] = {
>>  	[ESR_ELx_EC_HVC64]		= "HVC (AArch64)",
>>  	[ESR_ELx_EC_SMC64]		= "SMC (AArch64)",
>>  	[ESR_ELx_EC_SYS64]		= "MSR/MRS (AArch64)",
>> +	[ESR_ELx_EC_SVE]		= "SVE",
>>  	[ESR_ELx_EC_IMP_DEF]		= "EL3 IMP DEF",
>>  	[ESR_ELx_EC_IABT_LOW]		= "IABT (lower EL)",
>>  	[ESR_ELx_EC_IABT_CUR]		= "IABT (current EL)",

Re: [PATCH 06/27] arm64/sve: System register and exception syndrome definitions

[Alex Bennée]

Alex Bennée <alex.bennee@linaro.org> writes:

> Dave Martin <Dave.Martin@arm.com> writes:
>
>> The SVE architecture adds some system registers, ID register fields
>> and a dedicated ESR exception class.
>>
>> This patch adds the appropriate definitions that will be needed by
>> the kernel.
>>
>> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
>> ---
>>  arch/arm64/include/asm/esr.h     |  3 ++-
>>  arch/arm64/include/asm/kvm_arm.h |  1 +
>>  arch/arm64/include/asm/sysreg.h  | 16 ++++++++++++++++
>>  arch/arm64/kernel/traps.c        |  1 +
>>  4 files changed, 20 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h
>> index 8cabd57..813629e 100644
>> --- a/arch/arm64/include/asm/esr.h
>> +++ b/arch/arm64/include/asm/esr.h
>> @@ -43,7 +43,8 @@
>>  #define ESR_ELx_EC_HVC64	(0x16)
>>  #define ESR_ELx_EC_SMC64	(0x17)
>>  #define ESR_ELx_EC_SYS64	(0x18)
>> -/* Unallocated EC: 0x19 - 0x1E */
>> +#define ESR_ELx_EC_SVE		(0x19)
>> +/* Unallocated EC: 0x1A - 0x1E */
>>  #define ESR_ELx_EC_IMP_DEF	(0x1f)
>>  #define ESR_ELx_EC_IABT_LOW	(0x20)
>>  #define ESR_ELx_EC_IABT_CUR	(0x21)
>> diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
>> index 61d694c..dbf0537 100644
>> --- a/arch/arm64/include/asm/kvm_arm.h
>> +++ b/arch/arm64/include/asm/kvm_arm.h
>> @@ -185,6 +185,7 @@
>>  #define CPTR_EL2_TCPAC	(1 << 31)
>>  #define CPTR_EL2_TTA	(1 << 20)
>>  #define CPTR_EL2_TFP	(1 << CPTR_EL2_TFP_SHIFT)
>> +#define CPTR_EL2_TZ	(1 << 8)
>>  #define CPTR_EL2_DEFAULT	0x000033ff
>>
>>  /* Hyp Debug Configuration Register bits */
>> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
>> index 248339e..2d259e8 100644
>> --- a/arch/arm64/include/asm/sysreg.h
>> +++ b/arch/arm64/include/asm/sysreg.h
>> @@ -145,6 +145,7 @@
>>
>>  #define SYS_ID_AA64PFR0_EL1		sys_reg(3, 0, 0, 4, 0)
>>  #define SYS_ID_AA64PFR1_EL1		sys_reg(3, 0, 0, 4, 1)
>> +#define SYS_ID_AA64ZFR0_EL1		sys_reg(3, 0, 0, 4, 4)
>>
>>  #define SYS_ID_AA64DFR0_EL1		sys_reg(3, 0, 0, 5, 0)
>>  #define SYS_ID_AA64DFR1_EL1		sys_reg(3, 0, 0, 5, 1)
>> @@ -160,6 +161,8 @@
>>  #define SYS_ACTLR_EL1			sys_reg(3, 0, 1, 0, 1)
>>  #define SYS_CPACR_EL1			sys_reg(3, 0, 1, 0, 2)
>>
>> +#define SYS_ZCR_EL1			sys_reg(3, 0, 1, 2, 0)
>> +
>
> I'll have to take these on trust. They are mentioned in both the ARM ARM
> and the SVE supplement but I can't see any actual definitions of them.
>
>>  #define SYS_TTBR0_EL1			sys_reg(3, 0, 2, 0, 0)
>>  #define SYS_TTBR1_EL1			sys_reg(3, 0, 2, 0, 1)
>>  #define SYS_TCR_EL1			sys_reg(3, 0, 2, 0, 2)
>> @@ -250,6 +253,8 @@
>>
>>  #define SYS_PMCCFILTR_EL0		sys_reg (3, 3, 14, 15, 7)
>>
>> +#define SYS_ZCR_EL2			sys_reg(3, 4, 1, 2, 0)
>> +

OK no I'm working directly from the unpacked ZIP file with the rest of
the details I think this should be:

  #define SYS_ZCR_EL2			sys_reg(3, 5, 1, 2, 0)

e.g. op1 = 101 / 5

>>  #define SYS_DACR32_EL2			sys_reg(3, 4, 3, 0, 0)
>>  #define SYS_IFSR32_EL2			sys_reg(3, 4, 5, 0, 1)
>>  #define SYS_FPEXC32_EL2			sys_reg(3, 4, 5, 3, 0)
>> @@ -331,6 +336,7 @@
>>  #define ID_AA64ISAR1_JSCVT_SHIFT	12
>>
>>  /* id_aa64pfr0 */
>> +#define ID_AA64PFR0_SVE_SHIFT		32
>>  #define ID_AA64PFR0_GIC_SHIFT		24
>>  #define ID_AA64PFR0_ASIMD_SHIFT		20
>>  #define ID_AA64PFR0_FP_SHIFT		16
>> @@ -339,6 +345,7 @@
>>  #define ID_AA64PFR0_EL1_SHIFT		4
>>  #define ID_AA64PFR0_EL0_SHIFT		0
>>
>> +#define ID_AA64PFR0_SVE			0x1
>>  #define ID_AA64PFR0_FP_NI		0xf
>>  #define ID_AA64PFR0_FP_SUPPORTED	0x0
>>  #define ID_AA64PFR0_ASIMD_NI		0xf
>> @@ -440,6 +447,15 @@
>>  #endif
>>
>>
>> +#define ZCR_ELx_LEN_SHIFT	0
>> +#define ZCR_ELx_LEN_SIZE	9
>> +#define ZCR_ELx_LEN_MASK	0x1ff
>> +

LEN should be 0/4/0xf

LEN, bits [3:0]

Constrains the scalable vector register length for EL1 and EL0 to (LEN+1)x128 bits.


>> +#define CPACR_EL1_ZEN_EL1EN	(1 << 16)
>> +#define CPACR_EL1_ZEN_EL0EN	(1 << 17)
>> +#define CPACR_EL1_ZEN		(CPACR_EL1_ZEN_EL1EN |
>> CPACR_EL1_ZEN_EL0EN)
>
> This is a little weird as it is a 2 bit field in which 00 and 11 are not
> simply the sum of their bits. If the code wrote CPACR_EL1_ZEN_EL0EN |
> CPACR_EL1_ZEN_EL1EN to the CPACR_EL1 you wouldn't get the expected behaviour.
>
>> +
>> +
>>  /* Safe value for MPIDR_EL1: Bit31:RES1, Bit30:U:0, Bit24:MT:0 */
>>  #define SYS_MPIDR_SAFE_VAL		(1UL << 31)
>>
>> diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
>> index 0f047e9..8964795 100644
>> --- a/arch/arm64/kernel/traps.c
>> +++ b/arch/arm64/kernel/traps.c
>> @@ -621,6 +621,7 @@ static const char *esr_class_str[] = {
>>  	[ESR_ELx_EC_HVC64]		= "HVC (AArch64)",
>>  	[ESR_ELx_EC_SMC64]		= "SMC (AArch64)",
>>  	[ESR_ELx_EC_SYS64]		= "MSR/MRS (AArch64)",
>> +	[ESR_ELx_EC_SVE]		= "SVE",
>>  	[ESR_ELx_EC_IMP_DEF]		= "EL3 IMP DEF",
>>  	[ESR_ELx_EC_IABT_LOW]		= "IABT (lower EL)",
>>  	[ESR_ELx_EC_IABT_CUR]		= "IABT (current EL)",


--
Alex Bennée

Re: [PATCH 06/27] arm64/sve: System register and exception syndrome definitions

[Dave Martin]

On Mon, Aug 21, 2017 at 10:33:55AM +0100, Alex Bennée wrote:
> 
> Dave Martin <Dave.Martin@arm.com> writes:
> 
> > The SVE architecture adds some system registers, ID register fields
> > and a dedicated ESR exception class.
> >
> > This patch adds the appropriate definitions that will be needed by
> > the kernel.
> >
> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> > ---
> >  arch/arm64/include/asm/esr.h     |  3 ++-
> >  arch/arm64/include/asm/kvm_arm.h |  1 +
> >  arch/arm64/include/asm/sysreg.h  | 16 ++++++++++++++++
> >  arch/arm64/kernel/traps.c        |  1 +
> >  4 files changed, 20 insertions(+), 1 deletion(-)

[...]

> > diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
> > index 248339e..2d259e8 100644
> > --- a/arch/arm64/include/asm/sysreg.h
> > +++ b/arch/arm64/include/asm/sysreg.h

[...]

> > +#define SYS_ZCR_EL1			sys_reg(3, 0, 1, 2, 0)
> > +
> 
> I'll have to take these on trust. They are mentioned in both the ARM ARM
> and the SVE supplement but I can't see any actual definitions of them.

[I note from subsequent replies you've now found this in the
accompanying HTML]

[...]

> > +#define CPACR_EL1_ZEN_EL1EN	(1 << 16)
> > +#define CPACR_EL1_ZEN_EL0EN	(1 << 17)
> > +#define CPACR_EL1_ZEN		(CPACR_EL1_ZEN_EL1EN |
> > CPACR_EL1_ZEN_EL0EN)
> 
> This is a little weird as it is a 2 bit field in which 00 and 11 are not
> simply the sum of their bits. If the code wrote CPACR_EL1_ZEN_EL0EN |
> CPACR_EL1_ZEN_EL1EN to the CPACR_EL1 you wouldn't get the expected behaviour.

This seemed natural-ish if you believe that disabling at EL1 implies
disabling at EL0.  This is consistent with the way the traps at EL2/3
work, and lack of this property would be a sort of privilege inversion.

The meanings of the bits are not orthogonal, but it's still useful to be
able to twiddle EL0EN by itself when EL1EN is set (since we wan't to
control access for EL0 but leave EL1 access enabled).

Maybe comments would be sufficient:

#define CPACR_EL1_ZEN_EL1EN ... /* enable EL1 access */
#define CPACR_EL1_ZEN_EL0EN ... /* enable EL0 access, if EL1EN is also set */

I'm not sure how to make the names both reasonably terse and self-
escribing, but I'm open to ideas.

Cheers
---Dave

Re: [PATCH 06/27] arm64/sve: System register and exception syndrome definitions

[Dave Martin]

On Mon, Aug 21, 2017 at 10:33:55AM +0100, Alex Bennée wrote:
> 
> Dave Martin <Dave.Martin@arm.com> writes:
> 
> > The SVE architecture adds some system registers, ID register fields
> > and a dedicated ESR exception class.
> >
> > This patch adds the appropriate definitions that will be needed by
> > the kernel.
> >
> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> > ---
> >  arch/arm64/include/asm/esr.h     |  3 ++-
> >  arch/arm64/include/asm/kvm_arm.h |  1 +
> >  arch/arm64/include/asm/sysreg.h  | 16 ++++++++++++++++
> >  arch/arm64/kernel/traps.c        |  1 +
> >  4 files changed, 20 insertions(+), 1 deletion(-)

[...]

> > diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
> > index 248339e..2d259e8 100644
> > --- a/arch/arm64/include/asm/sysreg.h
> > +++ b/arch/arm64/include/asm/sysreg.h

[...]

> > +#define SYS_ZCR_EL1			sys_reg(3, 0, 1, 2, 0)
> > +
> 
> I'll have to take these on trust. They are mentioned in both the ARM ARM
> and the SVE supplement but I can't see any actual definitions of them.

[I note from subsequent replies you've now found this in the
accompanying HTML]

[...]

> > +#define CPACR_EL1_ZEN_EL1EN	(1 << 16)
> > +#define CPACR_EL1_ZEN_EL0EN	(1 << 17)
> > +#define CPACR_EL1_ZEN		(CPACR_EL1_ZEN_EL1EN |
> > CPACR_EL1_ZEN_EL0EN)
> 
> This is a little weird as it is a 2 bit field in which 00 and 11 are not
> simply the sum of their bits. If the code wrote CPACR_EL1_ZEN_EL0EN |
> CPACR_EL1_ZEN_EL1EN to the CPACR_EL1 you wouldn't get the expected behaviour.

This seemed natural-ish if you believe that disabling at EL1 implies
disabling at EL0.  This is consistent with the way the traps at EL2/3
work, and lack of this property would be a sort of privilege inversion.

The meanings of the bits are not orthogonal, but it's still useful to be
able to twiddle EL0EN by itself when EL1EN is set (since we wan't to
control access for EL0 but leave EL1 access enabled).

Maybe comments would be sufficient:

#define CPACR_EL1_ZEN_EL1EN ... /* enable EL1 access */
#define CPACR_EL1_ZEN_EL0EN ... /* enable EL0 access, if EL1EN is also set */

I'm not sure how to make the names both reasonably terse and self-
escribing, but I'm open to ideas.

Cheers
---Dave

Re: [PATCH 06/27] arm64/sve: System register and exception syndrome definitions

[Dave Martin]

On Mon, Aug 21, 2017 at 01:34:38PM +0100, Alex Bennée wrote:
> 
> Alex Bennée <alex.bennee@linaro.org> writes:
> 
> > Dave Martin <Dave.Martin@arm.com> writes:

[...]

> OK no I'm working directly from the unpacked ZIP file with the rest of
> the details I think this should be:
> 
>   #define SYS_ZCR_EL2			sys_reg(3, 5, 1, 2, 0)
> 
> e.g. op1 = 101 / 5

No, that's the encoding for ZCR_EL12.  Where did you get this from?

(This encoding follows the general pattern of the v8.1 Virtualization
Host Extensions.)

[...]

> >> +#define ZCR_ELx_LEN_SHIFT	0
> >> +#define ZCR_ELx_LEN_SIZE	9
> >> +#define ZCR_ELx_LEN_MASK	0x1ff
> >> +
> 
> LEN should be 0/4/0xf
> 
> LEN, bits [3:0]
> 
> Constrains the scalable vector register length for EL1 and EL0 to
> (LEN+1)x128 bits.

The SVE supplement is not very explicit about the meaning of bits [8:4],
but they are reserved to extend the LEN field in the future, in case
that's ever needed for future architecture revisions.  I've aimed for
Linux to cope with this.

Basically bits [8:4] are read-as-zero, write-ignore today, but in
the future some or all of them may be LEN field bits.

In particular, this means that writing all bits [8:0] with 1 will
configure the largest supported vector length, even on future
architecture versions that may have a larger LEN field.

It didn't seem useful to distinguish the two classes of bits here.

Cheers
---Dave

Re: [PATCH 06/27] arm64/sve: System register and exception syndrome definitions

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> On Mon, Aug 21, 2017 at 10:33:55AM +0100, Alex Bennée wrote:
>>
>> Dave Martin <Dave.Martin@arm.com> writes:
>>
>> > The SVE architecture adds some system registers, ID register fields
>> > and a dedicated ESR exception class.
>> >
>> > This patch adds the appropriate definitions that will be needed by
>> > the kernel.
>> >
>> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
>> > ---
>> >  arch/arm64/include/asm/esr.h     |  3 ++-
>> >  arch/arm64/include/asm/kvm_arm.h |  1 +
>> >  arch/arm64/include/asm/sysreg.h  | 16 ++++++++++++++++
>> >  arch/arm64/kernel/traps.c        |  1 +
>> >  4 files changed, 20 insertions(+), 1 deletion(-)
>
> [...]
>
>> > diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
>> > index 248339e..2d259e8 100644
>> > --- a/arch/arm64/include/asm/sysreg.h
>> > +++ b/arch/arm64/include/asm/sysreg.h
>
> [...]
>
>> > +#define SYS_ZCR_EL1			sys_reg(3, 0, 1, 2, 0)
>> > +
>>
>> I'll have to take these on trust. They are mentioned in both the ARM ARM
>> and the SVE supplement but I can't see any actual definitions of them.
>
> [I note from subsequent replies you've now found this in the
> accompanying HTML]
>
> [...]
>
>> > +#define CPACR_EL1_ZEN_EL1EN	(1 << 16)
>> > +#define CPACR_EL1_ZEN_EL0EN	(1 << 17)
>> > +#define CPACR_EL1_ZEN		(CPACR_EL1_ZEN_EL1EN |
>> > CPACR_EL1_ZEN_EL0EN)
>>
>> This is a little weird as it is a 2 bit field in which 00 and 11 are not
>> simply the sum of their bits. If the code wrote CPACR_EL1_ZEN_EL0EN |
>> CPACR_EL1_ZEN_EL1EN to the CPACR_EL1 you wouldn't get the expected behaviour.
>
> This seemed natural-ish if you believe that disabling at EL1 implies
> disabling at EL0.  This is consistent with the way the traps at EL2/3
> work, and lack of this property would be a sort of privilege inversion.
>
> The meanings of the bits are not orthogonal, but it's still useful to be
> able to twiddle EL0EN by itself when EL1EN is set (since we wan't to
> control access for EL0 but leave EL1 access enabled).
>
> Maybe comments would be sufficient:
>
> #define CPACR_EL1_ZEN_EL1EN ... /* enable EL1 access */
> #define CPACR_EL1_ZEN_EL0EN ... /* enable EL0 access, if EL1EN is also
> set */

Certainly it would help. I'll see as I go through the rest of the code
but it looks like a potential bear trap we should at least mitigate.

>
> I'm not sure how to make the names both reasonably terse and self-
> escribing, but I'm open to ideas.

Sorry no decent ideas. Naming things is hard as they say.

BTW see the follow-up mail for the other things I found....

Re: [PATCH 06/27] arm64/sve: System register and exception syndrome definitions

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> On Mon, Aug 21, 2017 at 10:33:55AM +0100, Alex Bennée wrote:
>>
>> Dave Martin <Dave.Martin@arm.com> writes:
>>
>> > The SVE architecture adds some system registers, ID register fields
>> > and a dedicated ESR exception class.
>> >
>> > This patch adds the appropriate definitions that will be needed by
>> > the kernel.
>> >
>> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
>> > ---
>> >  arch/arm64/include/asm/esr.h     |  3 ++-
>> >  arch/arm64/include/asm/kvm_arm.h |  1 +
>> >  arch/arm64/include/asm/sysreg.h  | 16 ++++++++++++++++
>> >  arch/arm64/kernel/traps.c        |  1 +
>> >  4 files changed, 20 insertions(+), 1 deletion(-)
>
> [...]
>
>> > diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
>> > index 248339e..2d259e8 100644
>> > --- a/arch/arm64/include/asm/sysreg.h
>> > +++ b/arch/arm64/include/asm/sysreg.h
>
> [...]
>
>> > +#define SYS_ZCR_EL1			sys_reg(3, 0, 1, 2, 0)
>> > +
>>
>> I'll have to take these on trust. They are mentioned in both the ARM ARM
>> and the SVE supplement but I can't see any actual definitions of them.
>
> [I note from subsequent replies you've now found this in the
> accompanying HTML]
>
> [...]
>
>> > +#define CPACR_EL1_ZEN_EL1EN	(1 << 16)
>> > +#define CPACR_EL1_ZEN_EL0EN	(1 << 17)
>> > +#define CPACR_EL1_ZEN		(CPACR_EL1_ZEN_EL1EN |
>> > CPACR_EL1_ZEN_EL0EN)
>>
>> This is a little weird as it is a 2 bit field in which 00 and 11 are not
>> simply the sum of their bits. If the code wrote CPACR_EL1_ZEN_EL0EN |
>> CPACR_EL1_ZEN_EL1EN to the CPACR_EL1 you wouldn't get the expected behaviour.
>
> This seemed natural-ish if you believe that disabling at EL1 implies
> disabling at EL0.  This is consistent with the way the traps at EL2/3
> work, and lack of this property would be a sort of privilege inversion.
>
> The meanings of the bits are not orthogonal, but it's still useful to be
> able to twiddle EL0EN by itself when EL1EN is set (since we wan't to
> control access for EL0 but leave EL1 access enabled).
>
> Maybe comments would be sufficient:
>
> #define CPACR_EL1_ZEN_EL1EN ... /* enable EL1 access */
> #define CPACR_EL1_ZEN_EL0EN ... /* enable EL0 access, if EL1EN is also
> set */

Certainly it would help. I'll see as I go through the rest of the code
but it looks like a potential bear trap we should at least mitigate.

>
> I'm not sure how to make the names both reasonably terse and self-
> escribing, but I'm open to ideas.

Sorry no decent ideas. Naming things is hard as they say.

BTW see the follow-up mail for the other things I found....

--
Alex Bennée

Re: [PATCH 07/27] arm64/sve: Low-level SVE architectural state manipulation functions

[Dave Martin]

On Mon, Aug 21, 2017 at 11:11:48AM +0100, Alex Bennée wrote:
> 
> Dave Martin <Dave.Martin@arm.com> writes:
> 
> > Manipulating the SVE architectural state, including the vector and
> > predicate registers, first-fault register and the vector length,
> > requires the use of dedicated instructions added by SVE.
> >
> > This patch adds suitable assembly functions for saving and
> > restoring the SVE registers and querying the vector length.
> > Setting of the vector length is done as part of register restore.
> >
> > Since people building kernels may not all get an SVE-enabled
> > toolchain for a while, this patch uses macros that generate
> > explicit opcodes in place of assembler mnemonics.
> >
> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> > ---
> >  arch/arm64/include/asm/fpsimd.h       |   5 ++
> >  arch/arm64/include/asm/fpsimdmacros.h | 137 ++++++++++++++++++++++++++++++++++
> >  arch/arm64/kernel/entry-fpsimd.S      |  17 +++++
> >  3 files changed, 159 insertions(+)

[...]

> > diff --git a/arch/arm64/include/asm/fpsimdmacros.h b/arch/arm64/include/asm/fpsimdmacros.h
> > index 0f5fdd3..5023064 100644
> > --- a/arch/arm64/include/asm/fpsimdmacros.h
> > +++ b/arch/arm64/include/asm/fpsimdmacros.h

[...]

> > +.macro _sve_str_v nz, nxbase, offset=0
> > +	_sve_check_zreg \nz
> > +	_check_general_reg \nxbase
> > +	_check_num (\offset), -0x100, 0xff
> > +	.inst	0xe5804000			\
> > +		| (\nz)				\
> > +		| ((\nxbase) << 5)		\
> > +		| (((\offset) & 7) << 10)	\
> > +		| (((\offset) & 0x1f8) << 13)
> > +.endm
> 
> Can we have references to the ARM ARM titles (the section numbers keep
> changing) for each hand-hacked instruction please. Otherwise it is
> impossible to verify each one.

That's fair -- will do.

The supplement didn't exist at the time I first wrote this code...

[...]

Cheers
---Dave

Re: [PATCH 07/27] arm64/sve: Low-level SVE architectural state manipulation functions

[Dave Martin]

On Mon, Aug 21, 2017 at 11:11:48AM +0100, Alex Bennée wrote:
> 
> Dave Martin <Dave.Martin@arm.com> writes:
> 
> > Manipulating the SVE architectural state, including the vector and
> > predicate registers, first-fault register and the vector length,
> > requires the use of dedicated instructions added by SVE.
> >
> > This patch adds suitable assembly functions for saving and
> > restoring the SVE registers and querying the vector length.
> > Setting of the vector length is done as part of register restore.
> >
> > Since people building kernels may not all get an SVE-enabled
> > toolchain for a while, this patch uses macros that generate
> > explicit opcodes in place of assembler mnemonics.
> >
> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> > ---
> >  arch/arm64/include/asm/fpsimd.h       |   5 ++
> >  arch/arm64/include/asm/fpsimdmacros.h | 137 ++++++++++++++++++++++++++++++++++
> >  arch/arm64/kernel/entry-fpsimd.S      |  17 +++++
> >  3 files changed, 159 insertions(+)

[...]

> > diff --git a/arch/arm64/include/asm/fpsimdmacros.h b/arch/arm64/include/asm/fpsimdmacros.h
> > index 0f5fdd3..5023064 100644
> > --- a/arch/arm64/include/asm/fpsimdmacros.h
> > +++ b/arch/arm64/include/asm/fpsimdmacros.h

[...]

> > +.macro _sve_str_v nz, nxbase, offset=0
> > +	_sve_check_zreg \nz
> > +	_check_general_reg \nxbase
> > +	_check_num (\offset), -0x100, 0xff
> > +	.inst	0xe5804000			\
> > +		| (\nz)				\
> > +		| ((\nxbase) << 5)		\
> > +		| (((\offset) & 7) << 10)	\
> > +		| (((\offset) & 0x1f8) << 13)
> > +.endm
> 
> Can we have references to the ARM ARM titles (the section numbers keep
> changing) for each hand-hacked instruction please. Otherwise it is
> impossible to verify each one.

That's fair -- will do.

The supplement didn't exist at the time I first wrote this code...

[...]

Cheers
---Dave

Re: [PATCH 06/27] arm64/sve: System register and exception syndrome definitions

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> On Mon, Aug 21, 2017 at 01:34:38PM +0100, Alex Bennée wrote:
>>
>> Alex Bennée <alex.bennee@linaro.org> writes:
>>
>> > Dave Martin <Dave.Martin@arm.com> writes:
>
> [...]
>
>> OK no I'm working directly from the unpacked ZIP file with the rest of
>> the details I think this should be:
>>
>>   #define SYS_ZCR_EL2			sys_reg(3, 5, 1, 2, 0)
>>
>> e.g. op1 = 101 / 5
>
> No, that's the encoding for ZCR_EL12.  Where did you get this from?

Sorry my mistake, -ETOOMANYBITTABLES....

>
> (This encoding follows the general pattern of the v8.1 Virtualization
> Host Extensions.)
>
> [...]
>
>> >> +#define ZCR_ELx_LEN_SHIFT	0
>> >> +#define ZCR_ELx_LEN_SIZE	9
>> >> +#define ZCR_ELx_LEN_MASK	0x1ff
>> >> +
>>
>> LEN should be 0/4/0xf
>>
>> LEN, bits [3:0]
>>
>> Constrains the scalable vector register length for EL1 and EL0 to
>> (LEN+1)x128 bits.
>
> The SVE supplement is not very explicit about the meaning of bits [8:4],
> but they are reserved to extend the LEN field in the future, in case
> that's ever needed for future architecture revisions.  I've aimed for
> Linux to cope with this.
>
> Basically bits [8:4] are read-as-zero, write-ignore today, but in
> the future some or all of them may be LEN field bits.
>
> In particular, this means that writing all bits [8:0] with 1 will
> configure the largest supported vector length, even on future
> architecture versions that may have a larger LEN field.

Ahh ok. It's not clear from the html and it is certainly implied in the
supplement (2.1.1) that the architectural max is:

  The size of every vector register is an IMPLEMENTATION DEFINED
  multiple of 128 bits, up to an architectural maximum of 2048 bits.

>
> It didn't seem useful to distinguish the two classes of bits here.

Maybe a comment clarifying would be useful then?

>
> Cheers
> ---Dave


--
Alex Bennée
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [PATCH 06/27] arm64/sve: System register and exception syndrome definitions

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> On Mon, Aug 21, 2017 at 01:34:38PM +0100, Alex Bennée wrote:
>>
>> Alex Bennée <alex.bennee@linaro.org> writes:
>>
>> > Dave Martin <Dave.Martin@arm.com> writes:
>
> [...]
>
>> OK no I'm working directly from the unpacked ZIP file with the rest of
>> the details I think this should be:
>>
>>   #define SYS_ZCR_EL2			sys_reg(3, 5, 1, 2, 0)
>>
>> e.g. op1 = 101 / 5
>
> No, that's the encoding for ZCR_EL12.  Where did you get this from?

Sorry my mistake, -ETOOMANYBITTABLES....

>
> (This encoding follows the general pattern of the v8.1 Virtualization
> Host Extensions.)
>
> [...]
>
>> >> +#define ZCR_ELx_LEN_SHIFT	0
>> >> +#define ZCR_ELx_LEN_SIZE	9
>> >> +#define ZCR_ELx_LEN_MASK	0x1ff
>> >> +
>>
>> LEN should be 0/4/0xf
>>
>> LEN, bits [3:0]
>>
>> Constrains the scalable vector register length for EL1 and EL0 to
>> (LEN+1)x128 bits.
>
> The SVE supplement is not very explicit about the meaning of bits [8:4],
> but they are reserved to extend the LEN field in the future, in case
> that's ever needed for future architecture revisions.  I've aimed for
> Linux to cope with this.
>
> Basically bits [8:4] are read-as-zero, write-ignore today, but in
> the future some or all of them may be LEN field bits.
>
> In particular, this means that writing all bits [8:0] with 1 will
> configure the largest supported vector length, even on future
> architecture versions that may have a larger LEN field.

Ahh ok. It's not clear from the html and it is certainly implied in the
supplement (2.1.1) that the architectural max is:

  The size of every vector register is an IMPLEMENTATION DEFINED
  multiple of 128 bits, up to an architectural maximum of 2048 bits.

>
> It didn't seem useful to distinguish the two classes of bits here.

Maybe a comment clarifying would be useful then?

>
> Cheers
> ---Dave


--
Alex Bennée

Re: [PATCH 06/27] arm64/sve: System register and exception syndrome definitions

[Dave Martin]

On Mon, Aug 21, 2017 at 03:50:32PM +0100, Alex Bennée wrote:
> Dave Martin <Dave.Martin@arm.com> writes:
> > On Mon, Aug 21, 2017 at 01:34:38PM +0100, Alex Bennée wrote:

[...]

> >> > Dave Martin <Dave.Martin@arm.com> writes:

[...]

> >> >> +#define ZCR_ELx_LEN_SHIFT	0
> >> >> +#define ZCR_ELx_LEN_SIZE	9
> >> >> +#define ZCR_ELx_LEN_MASK	0x1ff
> >> >> +
> >>
> >> LEN should be 0/4/0xf
> >>
> >> LEN, bits [3:0]
> >>
> >> Constrains the scalable vector register length for EL1 and EL0 to
> >> (LEN+1)x128 bits.
> >
> > The SVE supplement is not very explicit about the meaning of bits [8:4],
> > but they are reserved to extend the LEN field in the future, in case
> > that's ever needed for future architecture revisions.  I've aimed for
> > Linux to cope with this.
> >
> > Basically bits [8:4] are read-as-zero, write-ignore today, but in
> > the future some or all of them may be LEN field bits.
> >
> > In particular, this means that writing all bits [8:0] with 1 will
> > configure the largest supported vector length, even on future
> > architecture versions that may have a larger LEN field.
> 
> Ahh ok. It's not clear from the html and it is certainly implied in the
> supplement (2.1.1) that the architectural max is:
> 
>   The size of every vector register is an IMPLEMENTATION DEFINED
>   multiple of 128 bits, up to an architectural maximum of 2048 bits.
> 
> >
> > It didn't seem useful to distinguish the two classes of bits here.
> 
> Maybe a comment clarifying would be useful then?

OK, I think I can say something like:

/*
 * The ZCR_ELx_LEN_* definitions intentionally include bits [8:4]
 * which are reserved by the SVE architecture for future expansion of
 * the LEN field, with compatible semantics.
 */

Any good?

Cheers
---Dave

Re: [PATCH 06/27] arm64/sve: System register and exception syndrome definitions

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> On Mon, Aug 21, 2017 at 03:50:32PM +0100, Alex Bennée wrote:
>> Dave Martin <Dave.Martin@arm.com> writes:
>> > On Mon, Aug 21, 2017 at 01:34:38PM +0100, Alex Bennée wrote:
>
> [...]
>
>> >> > Dave Martin <Dave.Martin@arm.com> writes:
>
> [...]
>
>> >> >> +#define ZCR_ELx_LEN_SHIFT	0
>> >> >> +#define ZCR_ELx_LEN_SIZE	9
>> >> >> +#define ZCR_ELx_LEN_MASK	0x1ff
>> >> >> +
>> >>
>> >> LEN should be 0/4/0xf
>> >>
>> >> LEN, bits [3:0]
>> >>
>> >> Constrains the scalable vector register length for EL1 and EL0 to
>> >> (LEN+1)x128 bits.
>> >
>> > The SVE supplement is not very explicit about the meaning of bits [8:4],
>> > but they are reserved to extend the LEN field in the future, in case
>> > that's ever needed for future architecture revisions.  I've aimed for
>> > Linux to cope with this.
>> >
>> > Basically bits [8:4] are read-as-zero, write-ignore today, but in
>> > the future some or all of them may be LEN field bits.
>> >
>> > In particular, this means that writing all bits [8:0] with 1 will
>> > configure the largest supported vector length, even on future
>> > architecture versions that may have a larger LEN field.
>>
>> Ahh ok. It's not clear from the html and it is certainly implied in the
>> supplement (2.1.1) that the architectural max is:
>>
>>   The size of every vector register is an IMPLEMENTATION DEFINED
>>   multiple of 128 bits, up to an architectural maximum of 2048 bits.
>>
>> >
>> > It didn't seem useful to distinguish the two classes of bits here.
>>
>> Maybe a comment clarifying would be useful then?
>
> OK, I think I can say something like:
>
> /*
>  * The ZCR_ELx_LEN_* definitions intentionally include bits [8:4]
>  * which are reserved by the SVE architecture for future expansion of
>  * the LEN field, with compatible semantics.
>  */
>
> Any good?

Works for me ;-)

>
> Cheers
> ---Dave

Re: [PATCH 06/27] arm64/sve: System register and exception syndrome definitions

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> On Mon, Aug 21, 2017 at 03:50:32PM +0100, Alex Bennée wrote:
>> Dave Martin <Dave.Martin@arm.com> writes:
>> > On Mon, Aug 21, 2017 at 01:34:38PM +0100, Alex Bennée wrote:
>
> [...]
>
>> >> > Dave Martin <Dave.Martin@arm.com> writes:
>
> [...]
>
>> >> >> +#define ZCR_ELx_LEN_SHIFT	0
>> >> >> +#define ZCR_ELx_LEN_SIZE	9
>> >> >> +#define ZCR_ELx_LEN_MASK	0x1ff
>> >> >> +
>> >>
>> >> LEN should be 0/4/0xf
>> >>
>> >> LEN, bits [3:0]
>> >>
>> >> Constrains the scalable vector register length for EL1 and EL0 to
>> >> (LEN+1)x128 bits.
>> >
>> > The SVE supplement is not very explicit about the meaning of bits [8:4],
>> > but they are reserved to extend the LEN field in the future, in case
>> > that's ever needed for future architecture revisions.  I've aimed for
>> > Linux to cope with this.
>> >
>> > Basically bits [8:4] are read-as-zero, write-ignore today, but in
>> > the future some or all of them may be LEN field bits.
>> >
>> > In particular, this means that writing all bits [8:0] with 1 will
>> > configure the largest supported vector length, even on future
>> > architecture versions that may have a larger LEN field.
>>
>> Ahh ok. It's not clear from the html and it is certainly implied in the
>> supplement (2.1.1) that the architectural max is:
>>
>>   The size of every vector register is an IMPLEMENTATION DEFINED
>>   multiple of 128 bits, up to an architectural maximum of 2048 bits.
>>
>> >
>> > It didn't seem useful to distinguish the two classes of bits here.
>>
>> Maybe a comment clarifying would be useful then?
>
> OK, I think I can say something like:
>
> /*
>  * The ZCR_ELx_LEN_* definitions intentionally include bits [8:4]
>  * which are reserved by the SVE architecture for future expansion of
>  * the LEN field, with compatible semantics.
>  */
>
> Any good?

Works for me ;-)

>
> Cheers
> ---Dave


--
Alex Bennée

Re: [PATCH 09/27] arm64/sve: Signal frame and context structure definition

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> This patch defines the representation that will be used for the SVE
> register state in the signal frame, and implements support for
> saving and restoring the SVE registers around signals.
>
> The same layout will also be used for the in-kernel task state.
>
> Due to the variability of the SVE vector length, it is not possible
> to define a fixed C struct to describe all the registers.  Instead,
> Macros are defined in sigcontext.h to facilitate access to the
> parts of the structure.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/include/uapi/asm/sigcontext.h | 113 ++++++++++++++++++++++++++++++-
>  1 file changed, 112 insertions(+), 1 deletion(-)
>
> diff --git a/arch/arm64/include/uapi/asm/sigcontext.h b/arch/arm64/include/uapi/asm/sigcontext.h
> index f0a76b9..0533bdf 100644
> --- a/arch/arm64/include/uapi/asm/sigcontext.h
> +++ b/arch/arm64/include/uapi/asm/sigcontext.h
> @@ -16,6 +16,8 @@
>  #ifndef _UAPI__ASM_SIGCONTEXT_H
>  #define _UAPI__ASM_SIGCONTEXT_H
>
> +#ifndef __ASSEMBLY__
> +
>  #include <linux/types.h>
>
>  /*
> @@ -41,10 +43,11 @@ struct sigcontext {
>   *
>   *	0x210		fpsimd_context
>   *	 0x10		esr_context
> + *	0x8a0		sve_context (vl <= 64) (optional)
>   *	 0x20		extra_context (optional)
>   *	 0x10		terminator (null _aarch64_ctx)
>   *
> - *	0xdb0		(reserved for future allocation)
> + *	0x510		(reserved for future allocation)
>   *
>   * New records that can exceed this space need to be opt-in for userspace, so
>   * that an expanded signal frame is not generated unexpectedly.  The mechanism
> @@ -116,4 +119,112 @@ struct extra_context {
>  	__u32 __reserved[3];
>  };
>
> +#define SVE_MAGIC	0x53564501
> +
> +struct sve_context {
> +	struct _aarch64_ctx head;
> +	__u16 vl;
> +	__u16 __reserved[3];
> +};
> +
> +#endif /* !__ASSEMBLY__ */
> +
> +/*
> + * The SVE architecture leaves space for future expansion of the
> + * vector length beyond its initial architectural limit of 2048 bits
> + * (16 quadwords).
> + */
> +#define SVE_VQ_MIN		1
> +#define SVE_VQ_MAX		0x200
> +
> +#define SVE_VL_MIN		(SVE_VQ_MIN * 0x10)
> +#define SVE_VL_MAX		(SVE_VQ_MAX * 0x10)
> +
> +#define SVE_NUM_ZREGS		32
> +#define SVE_NUM_PREGS		16
> +
> +#define sve_vl_valid(vl) \
> +	((vl) % 0x10 == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
> +#define sve_vq_from_vl(vl)	((vl) / 0x10)
> +#define sve_vl_from_vq(vq)	((vq) * 0x10)

I got a little confused first time through over what VQ and VL where.
Maybe it would make sense to expand a little more from first principles?

  /*
   * The SVE architecture defines vector registers as a multiple of 128
   * bit quadwords. The current architectural limit is 2048 bits (16
   * quadwords) but there is room for future expansion beyond that.
    */

  #define SVE_VQ_BITS             128      /* 128 bits in one quadword */
  #define SVE_VQ_BYTES            (SVE_VQ_BITS / 8)

  #define SVE_VQ_MIN		1
  #define SVE_VQ_MAX		0x200    /* see ZCR_ELx[8:0] */

  #define SVE_VL_MIN_BYTES	(SVE_VQ_MIN * SVE_VQ_BYTES)
  #define SVE_VL_MAX_BYTES	(SVE_VQ_MAX * SVE_VQ_BYTES)

  #define SVE_NUM_ZREGS		32
  #define SVE_NUM_PREGS		16

  #define sve_vl_valid(vl)						\
          ((vl) % SVE_VQ_BYTES == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
  #define sve_vq_from_vl(vl)	((vl) / SVE_VQ_BYTES)
  #define sve_vl_from_vq(vq)	((vq) * SVE_VQ_BYTES)


> +
> +/*
> + * If the SVE registers are currently live for the thread at signal delivery,
> + * sve_context.head.size >=
> + *	SVE_SIG_CONTEXT_SIZE(sve_vq_from_vl(sve_context.vl))
> + * and the register data may be accessed using the SVE_SIG_*() macros.
> + *
> + * If sve_context.head.size <
> + *	SVE_SIG_CONTEXT_SIZE(sve_vq_from_vl(sve_context.vl)),
> + * the SVE registers were not live for the thread and no register data
> + * is included: in this case, the SVE_SIG_*() macros should not be
> + * used except for this check.
> + *
> + * The same convention applies when returning from a signal: a caller
> + * will need to remove or resize the sve_context block if it wants to
> + * make the SVE registers live when they were previously non-live or
> + * vice-versa.  This may require the the caller to allocate fresh
> + * memory and/or move other context blocks in the signal frame.
> + *
> + * Changing the vector length during signal return is not permitted:
> + * sve_context.vl must equal the thread's current vector length when
> + * doing a sigreturn.
> + *
> + *
> + * Note: for all these macros, the "vq" argument denotes the SVE
> + * vector length in quadwords (i.e., units of 128 bits).
> + *
> + * The correct way to obtain vq is to use sve_vq_from_vl(vl).  The
> + * result is valid if and only if sve_vl_valid(vl) is true.  This is
> + * guaranteed for a struct sve_context written by the kernel.
> + *
> + *
> + * Additional macros describe the contents and layout of the payload.
> + * For each, SVE_SIG_x_OFFSET(args) is the start offset relative to
> + * the start of struct sve_context, and SVE_SIG_x_SIZE(args) is the
> + * size in bytes:
> + *
> + *	x	type				description
> + *	-	----				-----------
> + *	REGS					the entire SVE context
> + *
> + *	ZREGS	__uint128_t[SVE_NUM_ZREGS][vq]	all Z-registers
> + *	ZREG	__uint128_t[vq]			individual Z-register Zn
> + *
> + *	PREGS	uint16_t[SVE_NUM_PREGS][vq]	all P-registers
> + *	PREG	uint16_t[vq]			individual P-register Pn
> + *
> + *	FFR	uint16_t[vq]			first-fault status register
> + *
> + * Additional data might be appended in the future.
> + */
> +
> +#define SVE_SIG_ZREG_SIZE(vq)	((__u32)(vq) * 16)
> +#define SVE_SIG_PREG_SIZE(vq)	((__u32)(vq) * 2)
> +#define SVE_SIG_FFR_SIZE(vq)	SVE_SIG_PREG_SIZE(vq)
> +
> +#define SVE_SIG_REGS_OFFSET	((sizeof(struct sve_context) + 15) / 16 * 16)
> +
> +#define SVE_SIG_ZREGS_OFFSET	SVE_SIG_REGS_OFFSET
> +#define SVE_SIG_ZREG_OFFSET(vq, n) \
> +	(SVE_SIG_ZREGS_OFFSET + SVE_SIG_ZREG_SIZE(vq) * (n))
> +#define SVE_SIG_ZREGS_SIZE(vq) \
> +	(SVE_SIG_ZREG_OFFSET(vq, SVE_NUM_ZREGS) - SVE_SIG_ZREGS_OFFSET)
> +
> +#define SVE_SIG_PREGS_OFFSET(vq) \
> +	(SVE_SIG_ZREGS_OFFSET + SVE_SIG_ZREGS_SIZE(vq))
> +#define SVE_SIG_PREG_OFFSET(vq, n) \
> +	(SVE_SIG_PREGS_OFFSET(vq) + SVE_SIG_PREG_SIZE(vq) * (n))
> +#define SVE_SIG_PREGS_SIZE(vq) \
> +	(SVE_SIG_PREG_OFFSET(vq, SVE_NUM_PREGS) - SVE_SIG_PREGS_OFFSET(vq))
> +
> +#define SVE_SIG_FFR_OFFSET(vq) \
> +	(SVE_SIG_PREGS_OFFSET(vq) + SVE_SIG_PREGS_SIZE(vq))
> +
> +#define SVE_SIG_REGS_SIZE(vq) \
> +	(SVE_SIG_FFR_OFFSET(vq) + SVE_SIG_FFR_SIZE(vq) - SVE_SIG_REGS_OFFSET)
> +
> +#define SVE_SIG_CONTEXT_SIZE(vq) (SVE_SIG_REGS_OFFSET + SVE_SIG_REGS_SIZE(vq))
> +
> +
>  #endif /* _UAPI__ASM_SIGCONTEXT_H */


--
Alex Bennée

Re: [PATCH 09/27] arm64/sve: Signal frame and context structure definition

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> This patch defines the representation that will be used for the SVE
> register state in the signal frame, and implements support for
> saving and restoring the SVE registers around signals.
>
> The same layout will also be used for the in-kernel task state.
>
> Due to the variability of the SVE vector length, it is not possible
> to define a fixed C struct to describe all the registers.  Instead,
> Macros are defined in sigcontext.h to facilitate access to the
> parts of the structure.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/include/uapi/asm/sigcontext.h | 113 ++++++++++++++++++++++++++++++-
>  1 file changed, 112 insertions(+), 1 deletion(-)
>
> diff --git a/arch/arm64/include/uapi/asm/sigcontext.h b/arch/arm64/include/uapi/asm/sigcontext.h
> index f0a76b9..0533bdf 100644
> --- a/arch/arm64/include/uapi/asm/sigcontext.h
> +++ b/arch/arm64/include/uapi/asm/sigcontext.h
> @@ -16,6 +16,8 @@
>  #ifndef _UAPI__ASM_SIGCONTEXT_H
>  #define _UAPI__ASM_SIGCONTEXT_H
>
> +#ifndef __ASSEMBLY__
> +
>  #include <linux/types.h>
>
>  /*
> @@ -41,10 +43,11 @@ struct sigcontext {
>   *
>   *	0x210		fpsimd_context
>   *	 0x10		esr_context
> + *	0x8a0		sve_context (vl <= 64) (optional)
>   *	 0x20		extra_context (optional)
>   *	 0x10		terminator (null _aarch64_ctx)
>   *
> - *	0xdb0		(reserved for future allocation)
> + *	0x510		(reserved for future allocation)
>   *
>   * New records that can exceed this space need to be opt-in for userspace, so
>   * that an expanded signal frame is not generated unexpectedly.  The mechanism
> @@ -116,4 +119,112 @@ struct extra_context {
>  	__u32 __reserved[3];
>  };
>
> +#define SVE_MAGIC	0x53564501
> +
> +struct sve_context {
> +	struct _aarch64_ctx head;
> +	__u16 vl;
> +	__u16 __reserved[3];
> +};
> +
> +#endif /* !__ASSEMBLY__ */
> +
> +/*
> + * The SVE architecture leaves space for future expansion of the
> + * vector length beyond its initial architectural limit of 2048 bits
> + * (16 quadwords).
> + */
> +#define SVE_VQ_MIN		1
> +#define SVE_VQ_MAX		0x200
> +
> +#define SVE_VL_MIN		(SVE_VQ_MIN * 0x10)
> +#define SVE_VL_MAX		(SVE_VQ_MAX * 0x10)
> +
> +#define SVE_NUM_ZREGS		32
> +#define SVE_NUM_PREGS		16
> +
> +#define sve_vl_valid(vl) \
> +	((vl) % 0x10 == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
> +#define sve_vq_from_vl(vl)	((vl) / 0x10)
> +#define sve_vl_from_vq(vq)	((vq) * 0x10)

I got a little confused first time through over what VQ and VL where.
Maybe it would make sense to expand a little more from first principles?

  /*
   * The SVE architecture defines vector registers as a multiple of 128
   * bit quadwords. The current architectural limit is 2048 bits (16
   * quadwords) but there is room for future expansion beyond that.
    */

  #define SVE_VQ_BITS             128      /* 128 bits in one quadword */
  #define SVE_VQ_BYTES            (SVE_VQ_BITS / 8)

  #define SVE_VQ_MIN		1
  #define SVE_VQ_MAX		0x200    /* see ZCR_ELx[8:0] */

  #define SVE_VL_MIN_BYTES	(SVE_VQ_MIN * SVE_VQ_BYTES)
  #define SVE_VL_MAX_BYTES	(SVE_VQ_MAX * SVE_VQ_BYTES)

  #define SVE_NUM_ZREGS		32
  #define SVE_NUM_PREGS		16

  #define sve_vl_valid(vl)						\
          ((vl) % SVE_VQ_BYTES == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
  #define sve_vq_from_vl(vl)	((vl) / SVE_VQ_BYTES)
  #define sve_vl_from_vq(vq)	((vq) * SVE_VQ_BYTES)


> +
> +/*
> + * If the SVE registers are currently live for the thread at signal delivery,
> + * sve_context.head.size >=
> + *	SVE_SIG_CONTEXT_SIZE(sve_vq_from_vl(sve_context.vl))
> + * and the register data may be accessed using the SVE_SIG_*() macros.
> + *
> + * If sve_context.head.size <
> + *	SVE_SIG_CONTEXT_SIZE(sve_vq_from_vl(sve_context.vl)),
> + * the SVE registers were not live for the thread and no register data
> + * is included: in this case, the SVE_SIG_*() macros should not be
> + * used except for this check.
> + *
> + * The same convention applies when returning from a signal: a caller
> + * will need to remove or resize the sve_context block if it wants to
> + * make the SVE registers live when they were previously non-live or
> + * vice-versa.  This may require the the caller to allocate fresh
> + * memory and/or move other context blocks in the signal frame.
> + *
> + * Changing the vector length during signal return is not permitted:
> + * sve_context.vl must equal the thread's current vector length when
> + * doing a sigreturn.
> + *
> + *
> + * Note: for all these macros, the "vq" argument denotes the SVE
> + * vector length in quadwords (i.e., units of 128 bits).
> + *
> + * The correct way to obtain vq is to use sve_vq_from_vl(vl).  The
> + * result is valid if and only if sve_vl_valid(vl) is true.  This is
> + * guaranteed for a struct sve_context written by the kernel.
> + *
> + *
> + * Additional macros describe the contents and layout of the payload.
> + * For each, SVE_SIG_x_OFFSET(args) is the start offset relative to
> + * the start of struct sve_context, and SVE_SIG_x_SIZE(args) is the
> + * size in bytes:
> + *
> + *	x	type				description
> + *	-	----				-----------
> + *	REGS					the entire SVE context
> + *
> + *	ZREGS	__uint128_t[SVE_NUM_ZREGS][vq]	all Z-registers
> + *	ZREG	__uint128_t[vq]			individual Z-register Zn
> + *
> + *	PREGS	uint16_t[SVE_NUM_PREGS][vq]	all P-registers
> + *	PREG	uint16_t[vq]			individual P-register Pn
> + *
> + *	FFR	uint16_t[vq]			first-fault status register
> + *
> + * Additional data might be appended in the future.
> + */
> +
> +#define SVE_SIG_ZREG_SIZE(vq)	((__u32)(vq) * 16)
> +#define SVE_SIG_PREG_SIZE(vq)	((__u32)(vq) * 2)
> +#define SVE_SIG_FFR_SIZE(vq)	SVE_SIG_PREG_SIZE(vq)
> +
> +#define SVE_SIG_REGS_OFFSET	((sizeof(struct sve_context) + 15) / 16 * 16)
> +
> +#define SVE_SIG_ZREGS_OFFSET	SVE_SIG_REGS_OFFSET
> +#define SVE_SIG_ZREG_OFFSET(vq, n) \
> +	(SVE_SIG_ZREGS_OFFSET + SVE_SIG_ZREG_SIZE(vq) * (n))
> +#define SVE_SIG_ZREGS_SIZE(vq) \
> +	(SVE_SIG_ZREG_OFFSET(vq, SVE_NUM_ZREGS) - SVE_SIG_ZREGS_OFFSET)
> +
> +#define SVE_SIG_PREGS_OFFSET(vq) \
> +	(SVE_SIG_ZREGS_OFFSET + SVE_SIG_ZREGS_SIZE(vq))
> +#define SVE_SIG_PREG_OFFSET(vq, n) \
> +	(SVE_SIG_PREGS_OFFSET(vq) + SVE_SIG_PREG_SIZE(vq) * (n))
> +#define SVE_SIG_PREGS_SIZE(vq) \
> +	(SVE_SIG_PREG_OFFSET(vq, SVE_NUM_PREGS) - SVE_SIG_PREGS_OFFSET(vq))
> +
> +#define SVE_SIG_FFR_OFFSET(vq) \
> +	(SVE_SIG_PREGS_OFFSET(vq) + SVE_SIG_PREGS_SIZE(vq))
> +
> +#define SVE_SIG_REGS_SIZE(vq) \
> +	(SVE_SIG_FFR_OFFSET(vq) + SVE_SIG_FFR_SIZE(vq) - SVE_SIG_REGS_OFFSET)
> +
> +#define SVE_SIG_CONTEXT_SIZE(vq) (SVE_SIG_REGS_OFFSET + SVE_SIG_REGS_SIZE(vq))
> +
> +
>  #endif /* _UAPI__ASM_SIGCONTEXT_H */


--
Alex Bennée

Re: [PATCH 09/27] arm64/sve: Signal frame and context structure definition

[Dave Martin]

On Tue, Aug 22, 2017 at 11:22:44AM +0100, Alex Bennée wrote:
> 
> Dave Martin <Dave.Martin@arm.com> writes:
> 
> > This patch defines the representation that will be used for the SVE
> > register state in the signal frame, and implements support for
> > saving and restoring the SVE registers around signals.
> >
> > The same layout will also be used for the in-kernel task state.
> >
> > Due to the variability of the SVE vector length, it is not possible
> > to define a fixed C struct to describe all the registers.  Instead,
> > Macros are defined in sigcontext.h to facilitate access to the
> > parts of the structure.
> >
> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> > ---
> >  arch/arm64/include/uapi/asm/sigcontext.h | 113 ++++++++++++++++++++++++++++++-
> >  1 file changed, 112 insertions(+), 1 deletion(-)
> >
> > diff --git a/arch/arm64/include/uapi/asm/sigcontext.h b/arch/arm64/include/uapi/asm/sigcontext.h
> > index f0a76b9..0533bdf 100644
> > --- a/arch/arm64/include/uapi/asm/sigcontext.h
> > +++ b/arch/arm64/include/uapi/asm/sigcontext.h
> > @@ -16,6 +16,8 @@
> >  #ifndef _UAPI__ASM_SIGCONTEXT_H
> >  #define _UAPI__ASM_SIGCONTEXT_H
> >
> > +#ifndef __ASSEMBLY__
> > +
> >  #include <linux/types.h>
> >
> >  /*
> > @@ -41,10 +43,11 @@ struct sigcontext {
> >   *
> >   *	0x210		fpsimd_context
> >   *	 0x10		esr_context
> > + *	0x8a0		sve_context (vl <= 64) (optional)
> >   *	 0x20		extra_context (optional)
> >   *	 0x10		terminator (null _aarch64_ctx)
> >   *
> > - *	0xdb0		(reserved for future allocation)
> > + *	0x510		(reserved for future allocation)
> >   *
> >   * New records that can exceed this space need to be opt-in for userspace, so
> >   * that an expanded signal frame is not generated unexpectedly.  The mechanism
> > @@ -116,4 +119,112 @@ struct extra_context {
> >  	__u32 __reserved[3];
> >  };
> >
> > +#define SVE_MAGIC	0x53564501
> > +
> > +struct sve_context {
> > +	struct _aarch64_ctx head;
> > +	__u16 vl;
> > +	__u16 __reserved[3];
> > +};
> > +
> > +#endif /* !__ASSEMBLY__ */
> > +
> > +/*
> > + * The SVE architecture leaves space for future expansion of the
> > + * vector length beyond its initial architectural limit of 2048 bits
> > + * (16 quadwords).
> > + */
> > +#define SVE_VQ_MIN		1
> > +#define SVE_VQ_MAX		0x200
> > +
> > +#define SVE_VL_MIN		(SVE_VQ_MIN * 0x10)
> > +#define SVE_VL_MAX		(SVE_VQ_MAX * 0x10)
> > +
> > +#define SVE_NUM_ZREGS		32
> > +#define SVE_NUM_PREGS		16
> > +
> > +#define sve_vl_valid(vl) \
> > +	((vl) % 0x10 == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
> > +#define sve_vq_from_vl(vl)	((vl) / 0x10)
> > +#define sve_vl_from_vq(vq)	((vq) * 0x10)
> 
> I got a little confused first time through over what VQ and VL where.
> Maybe it would make sense to expand a little more from first principles?
> 
>   /*
>    * The SVE architecture defines vector registers as a multiple of 128
>    * bit quadwords. The current architectural limit is 2048 bits (16
>    * quadwords) but there is room for future expansion beyond that.
>     */

This comes up in several places and so I didn't want to comment it
repeatedly everywhere.

Instead, I wrote up something in section 2 (Vector length terminology)
of Documentation/arm64/sve.txt -- see patch 25.  Can you take a look and
see whether that's adequate?

[...]

Cheers
---Dave

Re: [PATCH 09/27] arm64/sve: Signal frame and context structure definition

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> On Tue, Aug 22, 2017 at 11:22:44AM +0100, Alex Bennée wrote:
>>
>> Dave Martin <Dave.Martin@arm.com> writes:
>>
>> > This patch defines the representation that will be used for the SVE
>> > register state in the signal frame, and implements support for
>> > saving and restoring the SVE registers around signals.
>> >
>> > The same layout will also be used for the in-kernel task state.
>> >
>> > Due to the variability of the SVE vector length, it is not possible
>> > to define a fixed C struct to describe all the registers.  Instead,
>> > Macros are defined in sigcontext.h to facilitate access to the
>> > parts of the structure.
>> >
>> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
>> > ---
>> >  arch/arm64/include/uapi/asm/sigcontext.h | 113 ++++++++++++++++++++++++++++++-
>> >  1 file changed, 112 insertions(+), 1 deletion(-)
>> >
>> > diff --git a/arch/arm64/include/uapi/asm/sigcontext.h b/arch/arm64/include/uapi/asm/sigcontext.h
>> > index f0a76b9..0533bdf 100644
>> > --- a/arch/arm64/include/uapi/asm/sigcontext.h
>> > +++ b/arch/arm64/include/uapi/asm/sigcontext.h
>> > @@ -16,6 +16,8 @@
>> >  #ifndef _UAPI__ASM_SIGCONTEXT_H
>> >  #define _UAPI__ASM_SIGCONTEXT_H
>> >
>> > +#ifndef __ASSEMBLY__
>> > +
>> >  #include <linux/types.h>
>> >
>> >  /*
>> > @@ -41,10 +43,11 @@ struct sigcontext {
>> >   *
>> >   *	0x210		fpsimd_context
>> >   *	 0x10		esr_context
>> > + *	0x8a0		sve_context (vl <= 64) (optional)
>> >   *	 0x20		extra_context (optional)
>> >   *	 0x10		terminator (null _aarch64_ctx)
>> >   *
>> > - *	0xdb0		(reserved for future allocation)
>> > + *	0x510		(reserved for future allocation)
>> >   *
>> >   * New records that can exceed this space need to be opt-in for userspace, so
>> >   * that an expanded signal frame is not generated unexpectedly.  The mechanism
>> > @@ -116,4 +119,112 @@ struct extra_context {
>> >  	__u32 __reserved[3];
>> >  };
>> >
>> > +#define SVE_MAGIC	0x53564501
>> > +
>> > +struct sve_context {
>> > +	struct _aarch64_ctx head;
>> > +	__u16 vl;
>> > +	__u16 __reserved[3];
>> > +};
>> > +
>> > +#endif /* !__ASSEMBLY__ */
>> > +
>> > +/*
>> > + * The SVE architecture leaves space for future expansion of the
>> > + * vector length beyond its initial architectural limit of 2048 bits
>> > + * (16 quadwords).
>> > + */
>> > +#define SVE_VQ_MIN		1
>> > +#define SVE_VQ_MAX		0x200
>> > +
>> > +#define SVE_VL_MIN		(SVE_VQ_MIN * 0x10)
>> > +#define SVE_VL_MAX		(SVE_VQ_MAX * 0x10)
>> > +
>> > +#define SVE_NUM_ZREGS		32
>> > +#define SVE_NUM_PREGS		16
>> > +
>> > +#define sve_vl_valid(vl) \
>> > +	((vl) % 0x10 == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
>> > +#define sve_vq_from_vl(vl)	((vl) / 0x10)
>> > +#define sve_vl_from_vq(vq)	((vq) * 0x10)
>>
>> I got a little confused first time through over what VQ and VL where.
>> Maybe it would make sense to expand a little more from first principles?
>>
>>   /*
>>    * The SVE architecture defines vector registers as a multiple of 128
>>    * bit quadwords. The current architectural limit is 2048 bits (16
>>    * quadwords) but there is room for future expansion beyond that.
>>     */
>
> This comes up in several places and so I didn't want to comment it
> repeatedly everywhere.
>
> Instead, I wrote up something in section 2 (Vector length terminology)
> of Documentation/arm64/sve.txt -- see patch 25.  Can you take a look and
> see whether that's adequate?

Ahh, I hadn't got to that yet. I'm unsure to the order the kernel likes
to put things but I like to put design documents at the front of the
patch queue as they are useful primers and saves you having to patch a:

modified   arch/arm64/include/uapi/asm/sigcontext.h
@@ -132,19 +132,24 @@ struct sve_context {
 /*
  * The SVE architecture leaves space for future expansion of the
  * vector length beyond its initial architectural limit of 2048 bits
- * (16 quadwords).
+ * (16 quadwords). See Documentation/arm64/sve.txt for a summary of
+ * the terminology of Vector Quads (VQ) and Vector Lengths (VL).
  */
+
+#define SVE_VQ_BITS             128      /* 128 bits in one quadword */
+#define SVE_VQ_BYTES            (SVE_VQ_BITS / 8)
+
 #define SVE_VQ_MIN		1
 #define SVE_VQ_MAX		0x200

-#define SVE_VL_MIN		(SVE_VQ_MIN * 0x10)
-#define SVE_VL_MAX		(SVE_VQ_MAX * 0x10)
+#define SVE_VL_MIN		(SVE_VQ_MIN * SVE_VQ_BYTES)
+#define SVE_VL_MAX		(SVE_VQ_MAX * SVE_VQ_BYTES)

 #define SVE_NUM_ZREGS		32
 #define SVE_NUM_PREGS		16

 #define sve_vl_valid(vl) \
-	((vl) % 0x10 == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
+	((vl) % SVE_VQ_BYTES == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
 #define sve_vq_from_vl(vl)	((vl) / 0x10)
 #define sve_vl_from_vq(vq)	((vq) * 0x10)


>
> [...]
>
> Cheers
> ---Dave


--
Alex Bennée
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [PATCH 09/27] arm64/sve: Signal frame and context structure definition

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> On Tue, Aug 22, 2017 at 11:22:44AM +0100, Alex Bennée wrote:
>>
>> Dave Martin <Dave.Martin@arm.com> writes:
>>
>> > This patch defines the representation that will be used for the SVE
>> > register state in the signal frame, and implements support for
>> > saving and restoring the SVE registers around signals.
>> >
>> > The same layout will also be used for the in-kernel task state.
>> >
>> > Due to the variability of the SVE vector length, it is not possible
>> > to define a fixed C struct to describe all the registers.  Instead,
>> > Macros are defined in sigcontext.h to facilitate access to the
>> > parts of the structure.
>> >
>> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
>> > ---
>> >  arch/arm64/include/uapi/asm/sigcontext.h | 113 ++++++++++++++++++++++++++++++-
>> >  1 file changed, 112 insertions(+), 1 deletion(-)
>> >
>> > diff --git a/arch/arm64/include/uapi/asm/sigcontext.h b/arch/arm64/include/uapi/asm/sigcontext.h
>> > index f0a76b9..0533bdf 100644
>> > --- a/arch/arm64/include/uapi/asm/sigcontext.h
>> > +++ b/arch/arm64/include/uapi/asm/sigcontext.h
>> > @@ -16,6 +16,8 @@
>> >  #ifndef _UAPI__ASM_SIGCONTEXT_H
>> >  #define _UAPI__ASM_SIGCONTEXT_H
>> >
>> > +#ifndef __ASSEMBLY__
>> > +
>> >  #include <linux/types.h>
>> >
>> >  /*
>> > @@ -41,10 +43,11 @@ struct sigcontext {
>> >   *
>> >   *	0x210		fpsimd_context
>> >   *	 0x10		esr_context
>> > + *	0x8a0		sve_context (vl <= 64) (optional)
>> >   *	 0x20		extra_context (optional)
>> >   *	 0x10		terminator (null _aarch64_ctx)
>> >   *
>> > - *	0xdb0		(reserved for future allocation)
>> > + *	0x510		(reserved for future allocation)
>> >   *
>> >   * New records that can exceed this space need to be opt-in for userspace, so
>> >   * that an expanded signal frame is not generated unexpectedly.  The mechanism
>> > @@ -116,4 +119,112 @@ struct extra_context {
>> >  	__u32 __reserved[3];
>> >  };
>> >
>> > +#define SVE_MAGIC	0x53564501
>> > +
>> > +struct sve_context {
>> > +	struct _aarch64_ctx head;
>> > +	__u16 vl;
>> > +	__u16 __reserved[3];
>> > +};
>> > +
>> > +#endif /* !__ASSEMBLY__ */
>> > +
>> > +/*
>> > + * The SVE architecture leaves space for future expansion of the
>> > + * vector length beyond its initial architectural limit of 2048 bits
>> > + * (16 quadwords).
>> > + */
>> > +#define SVE_VQ_MIN		1
>> > +#define SVE_VQ_MAX		0x200
>> > +
>> > +#define SVE_VL_MIN		(SVE_VQ_MIN * 0x10)
>> > +#define SVE_VL_MAX		(SVE_VQ_MAX * 0x10)
>> > +
>> > +#define SVE_NUM_ZREGS		32
>> > +#define SVE_NUM_PREGS		16
>> > +
>> > +#define sve_vl_valid(vl) \
>> > +	((vl) % 0x10 == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
>> > +#define sve_vq_from_vl(vl)	((vl) / 0x10)
>> > +#define sve_vl_from_vq(vq)	((vq) * 0x10)
>>
>> I got a little confused first time through over what VQ and VL where.
>> Maybe it would make sense to expand a little more from first principles?
>>
>>   /*
>>    * The SVE architecture defines vector registers as a multiple of 128
>>    * bit quadwords. The current architectural limit is 2048 bits (16
>>    * quadwords) but there is room for future expansion beyond that.
>>     */
>
> This comes up in several places and so I didn't want to comment it
> repeatedly everywhere.
>
> Instead, I wrote up something in section 2 (Vector length terminology)
> of Documentation/arm64/sve.txt -- see patch 25.  Can you take a look and
> see whether that's adequate?

Ahh, I hadn't got to that yet. I'm unsure to the order the kernel likes
to put things but I like to put design documents at the front of the
patch queue as they are useful primers and saves you having to patch a:

modified   arch/arm64/include/uapi/asm/sigcontext.h
@@ -132,19 +132,24 @@ struct sve_context {
 /*
  * The SVE architecture leaves space for future expansion of the
  * vector length beyond its initial architectural limit of 2048 bits
- * (16 quadwords).
+ * (16 quadwords). See Documentation/arm64/sve.txt for a summary of
+ * the terminology of Vector Quads (VQ) and Vector Lengths (VL).
  */
+
+#define SVE_VQ_BITS             128      /* 128 bits in one quadword */
+#define SVE_VQ_BYTES            (SVE_VQ_BITS / 8)
+
 #define SVE_VQ_MIN		1
 #define SVE_VQ_MAX		0x200

-#define SVE_VL_MIN		(SVE_VQ_MIN * 0x10)
-#define SVE_VL_MAX		(SVE_VQ_MAX * 0x10)
+#define SVE_VL_MIN		(SVE_VQ_MIN * SVE_VQ_BYTES)
+#define SVE_VL_MAX		(SVE_VQ_MAX * SVE_VQ_BYTES)

 #define SVE_NUM_ZREGS		32
 #define SVE_NUM_PREGS		16

 #define sve_vl_valid(vl) \
-	((vl) % 0x10 == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
+	((vl) % SVE_VQ_BYTES == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
 #define sve_vq_from_vl(vl)	((vl) / 0x10)
 #define sve_vl_from_vq(vq)	((vq) * 0x10)


>
> [...]
>
> Cheers
> ---Dave


--
Alex Bennée

Re: [PATCH 09/27] arm64/sve: Signal frame and context structure definition

[Dave Martin]

On Tue, Aug 22, 2017 at 02:53:49PM +0100, Alex Bennée wrote:
> 
> Dave Martin <Dave.Martin@arm.com> writes:
> 
> > On Tue, Aug 22, 2017 at 11:22:44AM +0100, Alex Bennée wrote:
> >>
> >> Dave Martin <Dave.Martin@arm.com> writes:

[...]

> >> > +/*
> >> > + * The SVE architecture leaves space for future expansion of the
> >> > + * vector length beyond its initial architectural limit of 2048 bits
> >> > + * (16 quadwords).
> >> > + */
> >> > +#define SVE_VQ_MIN		1
> >> > +#define SVE_VQ_MAX		0x200
> >> > +
> >> > +#define SVE_VL_MIN		(SVE_VQ_MIN * 0x10)
> >> > +#define SVE_VL_MAX		(SVE_VQ_MAX * 0x10)
> >> > +
> >> > +#define SVE_NUM_ZREGS		32
> >> > +#define SVE_NUM_PREGS		16
> >> > +
> >> > +#define sve_vl_valid(vl) \
> >> > +	((vl) % 0x10 == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
> >> > +#define sve_vq_from_vl(vl)	((vl) / 0x10)
> >> > +#define sve_vl_from_vq(vq)	((vq) * 0x10)
> >>
> >> I got a little confused first time through over what VQ and VL where.
> >> Maybe it would make sense to expand a little more from first principles?
> >>
> >>   /*
> >>    * The SVE architecture defines vector registers as a multiple of 128
> >>    * bit quadwords. The current architectural limit is 2048 bits (16
> >>    * quadwords) but there is room for future expansion beyond that.
> >>     */
> >
> > This comes up in several places and so I didn't want to comment it
> > repeatedly everywhere.
> >
> > Instead, I wrote up something in section 2 (Vector length terminology)
> > of Documentation/arm64/sve.txt -- see patch 25.  Can you take a look and
> > see whether that's adequate?
> 
> Ahh, I hadn't got to that yet. I'm unsure to the order the kernel likes
> to put things but I like to put design documents at the front of the

I don't have a strong opinion on that -- I had preferred not to add a
document describing stuff that doesn't exist at the time of commit.
I could commit a stub document at the start of the series, and then
commit the real document later.

Either way, it seemed overkill.

Perhaps I should have drawn more attention to the documentation in the
cover letter, and encouraged reviewers to look at it first.  My
experience is that people don't often read cover letters...

Now the series is posted, I'm minded to keep the order as-is, unless you
think it's a big issue.

Adding a reference to the document seems a reasonable thing to do,
so I could add that.

> patch queue as they are useful primers and saves you having to patch a:
> 
> modified   arch/arm64/include/uapi/asm/sigcontext.h
> @@ -132,19 +132,24 @@ struct sve_context {
>  /*
>   * The SVE architecture leaves space for future expansion of the
>   * vector length beyond its initial architectural limit of 2048 bits
> - * (16 quadwords).
> + * (16 quadwords). See Documentation/arm64/sve.txt for a summary of
> + * the terminology of Vector Quads (VQ) and Vector Lengths (VL).
>   */
> +
> +#define SVE_VQ_BITS             128      /* 128 bits in one quadword */
> +#define SVE_VQ_BYTES            (SVE_VQ_BITS / 8)
> +

I was trying to keep extraneous #defines to a minimum, since this is a
uapi header, and people may depend on anything defined here.

I think SVE_VQ_BYTES is reasonable to have, and this allows us to
rewrite a few hard-coded 0x10s and 16s symbolically which is probably a
good idea -- I'll add this.

SVE_VQ_BITS looks redundant to me though.  It wouldn't be used for any
purpose other than defining SVE_VQ_BYTES.

>  #define SVE_VQ_MIN		1
>  #define SVE_VQ_MAX		0x200
> 
> -#define SVE_VL_MIN		(SVE_VQ_MIN * 0x10)
> -#define SVE_VL_MAX		(SVE_VQ_MAX * 0x10)
> +#define SVE_VL_MIN		(SVE_VQ_MIN * SVE_VQ_BYTES)
> +#define SVE_VL_MAX		(SVE_VQ_MAX * SVE_VQ_BYTES)
> 
>  #define SVE_NUM_ZREGS		32
>  #define SVE_NUM_PREGS		16
> 
>  #define sve_vl_valid(vl) \
> -	((vl) % 0x10 == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
> +	((vl) % SVE_VQ_BYTES == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
>  #define sve_vq_from_vl(vl)	((vl) / 0x10)
>  #define sve_vl_from_vq(vq)	((vq) * 0x10)

[...]

Cheers
---Dave

Re: [PATCH 09/27] arm64/sve: Signal frame and context structure definition

[Dave Martin]

On Tue, Aug 22, 2017 at 02:53:49PM +0100, Alex Bennée wrote:
> 
> Dave Martin <Dave.Martin@arm.com> writes:
> 
> > On Tue, Aug 22, 2017 at 11:22:44AM +0100, Alex Bennée wrote:
> >>
> >> Dave Martin <Dave.Martin@arm.com> writes:

[...]

> >> > +/*
> >> > + * The SVE architecture leaves space for future expansion of the
> >> > + * vector length beyond its initial architectural limit of 2048 bits
> >> > + * (16 quadwords).
> >> > + */
> >> > +#define SVE_VQ_MIN		1
> >> > +#define SVE_VQ_MAX		0x200
> >> > +
> >> > +#define SVE_VL_MIN		(SVE_VQ_MIN * 0x10)
> >> > +#define SVE_VL_MAX		(SVE_VQ_MAX * 0x10)
> >> > +
> >> > +#define SVE_NUM_ZREGS		32
> >> > +#define SVE_NUM_PREGS		16
> >> > +
> >> > +#define sve_vl_valid(vl) \
> >> > +	((vl) % 0x10 == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
> >> > +#define sve_vq_from_vl(vl)	((vl) / 0x10)
> >> > +#define sve_vl_from_vq(vq)	((vq) * 0x10)
> >>
> >> I got a little confused first time through over what VQ and VL where.
> >> Maybe it would make sense to expand a little more from first principles?
> >>
> >>   /*
> >>    * The SVE architecture defines vector registers as a multiple of 128
> >>    * bit quadwords. The current architectural limit is 2048 bits (16
> >>    * quadwords) but there is room for future expansion beyond that.
> >>     */
> >
> > This comes up in several places and so I didn't want to comment it
> > repeatedly everywhere.
> >
> > Instead, I wrote up something in section 2 (Vector length terminology)
> > of Documentation/arm64/sve.txt -- see patch 25.  Can you take a look and
> > see whether that's adequate?
> 
> Ahh, I hadn't got to that yet. I'm unsure to the order the kernel likes
> to put things but I like to put design documents at the front of the

I don't have a strong opinion on that -- I had preferred not to add a
document describing stuff that doesn't exist at the time of commit.
I could commit a stub document at the start of the series, and then
commit the real document later.

Either way, it seemed overkill.

Perhaps I should have drawn more attention to the documentation in the
cover letter, and encouraged reviewers to look at it first.  My
experience is that people don't often read cover letters...

Now the series is posted, I'm minded to keep the order as-is, unless you
think it's a big issue.

Adding a reference to the document seems a reasonable thing to do,
so I could add that.

> patch queue as they are useful primers and saves you having to patch a:
> 
> modified   arch/arm64/include/uapi/asm/sigcontext.h
> @@ -132,19 +132,24 @@ struct sve_context {
>  /*
>   * The SVE architecture leaves space for future expansion of the
>   * vector length beyond its initial architectural limit of 2048 bits
> - * (16 quadwords).
> + * (16 quadwords). See Documentation/arm64/sve.txt for a summary of
> + * the terminology of Vector Quads (VQ) and Vector Lengths (VL).
>   */
> +
> +#define SVE_VQ_BITS             128      /* 128 bits in one quadword */
> +#define SVE_VQ_BYTES            (SVE_VQ_BITS / 8)
> +

I was trying to keep extraneous #defines to a minimum, since this is a
uapi header, and people may depend on anything defined here.

I think SVE_VQ_BYTES is reasonable to have, and this allows us to
rewrite a few hard-coded 0x10s and 16s symbolically which is probably a
good idea -- I'll add this.

SVE_VQ_BITS looks redundant to me though.  It wouldn't be used for any
purpose other than defining SVE_VQ_BYTES.

>  #define SVE_VQ_MIN		1
>  #define SVE_VQ_MAX		0x200
> 
> -#define SVE_VL_MIN		(SVE_VQ_MIN * 0x10)
> -#define SVE_VL_MAX		(SVE_VQ_MAX * 0x10)
> +#define SVE_VL_MIN		(SVE_VQ_MIN * SVE_VQ_BYTES)
> +#define SVE_VL_MAX		(SVE_VQ_MAX * SVE_VQ_BYTES)
> 
>  #define SVE_NUM_ZREGS		32
>  #define SVE_NUM_PREGS		16
> 
>  #define sve_vl_valid(vl) \
> -	((vl) % 0x10 == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
> +	((vl) % SVE_VQ_BYTES == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
>  #define sve_vq_from_vl(vl)	((vl) / 0x10)
>  #define sve_vl_from_vq(vq)	((vq) * 0x10)

[...]

Cheers
---Dave

Re: [PATCH 09/27] arm64/sve: Signal frame and context structure definition

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> On Tue, Aug 22, 2017 at 02:53:49PM +0100, Alex Bennée wrote:
>>
>> Dave Martin <Dave.Martin@arm.com> writes:
>>
>> > On Tue, Aug 22, 2017 at 11:22:44AM +0100, Alex Bennée wrote:
>> >>
>> >> Dave Martin <Dave.Martin@arm.com> writes:
>
> [...]
>
>> >> > +/*
>> >> > + * The SVE architecture leaves space for future expansion of the
>> >> > + * vector length beyond its initial architectural limit of 2048 bits
>> >> > + * (16 quadwords).
>> >> > + */
>> >> > +#define SVE_VQ_MIN		1
>> >> > +#define SVE_VQ_MAX		0x200
>> >> > +
>> >> > +#define SVE_VL_MIN		(SVE_VQ_MIN * 0x10)
>> >> > +#define SVE_VL_MAX		(SVE_VQ_MAX * 0x10)
>> >> > +
>> >> > +#define SVE_NUM_ZREGS		32
>> >> > +#define SVE_NUM_PREGS		16
>> >> > +
>> >> > +#define sve_vl_valid(vl) \
>> >> > +	((vl) % 0x10 == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
>> >> > +#define sve_vq_from_vl(vl)	((vl) / 0x10)
>> >> > +#define sve_vl_from_vq(vq)	((vq) * 0x10)
>> >>
>> >> I got a little confused first time through over what VQ and VL where.
>> >> Maybe it would make sense to expand a little more from first principles?
>> >>
>> >>   /*
>> >>    * The SVE architecture defines vector registers as a multiple of 128
>> >>    * bit quadwords. The current architectural limit is 2048 bits (16
>> >>    * quadwords) but there is room for future expansion beyond that.
>> >>     */
>> >
>> > This comes up in several places and so I didn't want to comment it
>> > repeatedly everywhere.
>> >
>> > Instead, I wrote up something in section 2 (Vector length terminology)
>> > of Documentation/arm64/sve.txt -- see patch 25.  Can you take a look and
>> > see whether that's adequate?
>>
>> Ahh, I hadn't got to that yet. I'm unsure to the order the kernel likes
>> to put things but I like to put design documents at the front of the
>
> I don't have a strong opinion on that -- I had preferred not to add a
> document describing stuff that doesn't exist at the time of commit.
> I could commit a stub document at the start of the series, and then
> commit the real document later.
>
> Either way, it seemed overkill.
>
> Perhaps I should have drawn more attention to the documentation in the
> cover letter, and encouraged reviewers to look at it first.  My
> experience is that people don't often read cover letters...
>
> Now the series is posted, I'm minded to keep the order as-is, unless you
> think it's a big issue.
>
> Adding a reference to the document seems a reasonable thing to do,
> so I could add that.
>
>> patch queue as they are useful primers and saves you having to patch a:
>>
>> modified   arch/arm64/include/uapi/asm/sigcontext.h
>> @@ -132,19 +132,24 @@ struct sve_context {
>>  /*
>>   * The SVE architecture leaves space for future expansion of the
>>   * vector length beyond its initial architectural limit of 2048 bits
>> - * (16 quadwords).
>> + * (16 quadwords). See Documentation/arm64/sve.txt for a summary of
>> + * the terminology of Vector Quads (VQ) and Vector Lengths (VL).
>>   */
>> +
>> +#define SVE_VQ_BITS             128      /* 128 bits in one quadword */
>> +#define SVE_VQ_BYTES            (SVE_VQ_BITS / 8)
>> +
>
> I was trying to keep extraneous #defines to a minimum, since this is a
> uapi header, and people may depend on anything defined here.
>
> I think SVE_VQ_BYTES is reasonable to have, and this allows us to
> rewrite a few hard-coded 0x10s and 16s symbolically which is probably a
> good idea -- I'll add this.
>
> SVE_VQ_BITS looks redundant to me though.  It wouldn't be used for any
> purpose other than defining SVE_VQ_BYTES.

Yeah I was more concerned with getting rid of the magic 0x10's than
showing exactly how many bits something is.

>
>>  #define SVE_VQ_MIN		1
>>  #define SVE_VQ_MAX		0x200
>>
>> -#define SVE_VL_MIN		(SVE_VQ_MIN * 0x10)
>> -#define SVE_VL_MAX		(SVE_VQ_MAX * 0x10)
>> +#define SVE_VL_MIN		(SVE_VQ_MIN * SVE_VQ_BYTES)
>> +#define SVE_VL_MAX		(SVE_VQ_MAX * SVE_VQ_BYTES)
>>
>>  #define SVE_NUM_ZREGS		32
>>  #define SVE_NUM_PREGS		16
>>
>>  #define sve_vl_valid(vl) \
>> -	((vl) % 0x10 == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
>> +	((vl) % SVE_VQ_BYTES == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
>>  #define sve_vq_from_vl(vl)	((vl) / 0x10)
>>  #define sve_vl_from_vq(vq)	((vq) * 0x10)
>
> [...]
>
> Cheers
> ---Dave

Re: [PATCH 09/27] arm64/sve: Signal frame and context structure definition

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> On Tue, Aug 22, 2017 at 02:53:49PM +0100, Alex Bennée wrote:
>>
>> Dave Martin <Dave.Martin@arm.com> writes:
>>
>> > On Tue, Aug 22, 2017 at 11:22:44AM +0100, Alex Bennée wrote:
>> >>
>> >> Dave Martin <Dave.Martin@arm.com> writes:
>
> [...]
>
>> >> > +/*
>> >> > + * The SVE architecture leaves space for future expansion of the
>> >> > + * vector length beyond its initial architectural limit of 2048 bits
>> >> > + * (16 quadwords).
>> >> > + */
>> >> > +#define SVE_VQ_MIN		1
>> >> > +#define SVE_VQ_MAX		0x200
>> >> > +
>> >> > +#define SVE_VL_MIN		(SVE_VQ_MIN * 0x10)
>> >> > +#define SVE_VL_MAX		(SVE_VQ_MAX * 0x10)
>> >> > +
>> >> > +#define SVE_NUM_ZREGS		32
>> >> > +#define SVE_NUM_PREGS		16
>> >> > +
>> >> > +#define sve_vl_valid(vl) \
>> >> > +	((vl) % 0x10 == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
>> >> > +#define sve_vq_from_vl(vl)	((vl) / 0x10)
>> >> > +#define sve_vl_from_vq(vq)	((vq) * 0x10)
>> >>
>> >> I got a little confused first time through over what VQ and VL where.
>> >> Maybe it would make sense to expand a little more from first principles?
>> >>
>> >>   /*
>> >>    * The SVE architecture defines vector registers as a multiple of 128
>> >>    * bit quadwords. The current architectural limit is 2048 bits (16
>> >>    * quadwords) but there is room for future expansion beyond that.
>> >>     */
>> >
>> > This comes up in several places and so I didn't want to comment it
>> > repeatedly everywhere.
>> >
>> > Instead, I wrote up something in section 2 (Vector length terminology)
>> > of Documentation/arm64/sve.txt -- see patch 25.  Can you take a look and
>> > see whether that's adequate?
>>
>> Ahh, I hadn't got to that yet. I'm unsure to the order the kernel likes
>> to put things but I like to put design documents at the front of the
>
> I don't have a strong opinion on that -- I had preferred not to add a
> document describing stuff that doesn't exist at the time of commit.
> I could commit a stub document at the start of the series, and then
> commit the real document later.
>
> Either way, it seemed overkill.
>
> Perhaps I should have drawn more attention to the documentation in the
> cover letter, and encouraged reviewers to look at it first.  My
> experience is that people don't often read cover letters...
>
> Now the series is posted, I'm minded to keep the order as-is, unless you
> think it's a big issue.
>
> Adding a reference to the document seems a reasonable thing to do,
> so I could add that.
>
>> patch queue as they are useful primers and saves you having to patch a:
>>
>> modified   arch/arm64/include/uapi/asm/sigcontext.h
>> @@ -132,19 +132,24 @@ struct sve_context {
>>  /*
>>   * The SVE architecture leaves space for future expansion of the
>>   * vector length beyond its initial architectural limit of 2048 bits
>> - * (16 quadwords).
>> + * (16 quadwords). See Documentation/arm64/sve.txt for a summary of
>> + * the terminology of Vector Quads (VQ) and Vector Lengths (VL).
>>   */
>> +
>> +#define SVE_VQ_BITS             128      /* 128 bits in one quadword */
>> +#define SVE_VQ_BYTES            (SVE_VQ_BITS / 8)
>> +
>
> I was trying to keep extraneous #defines to a minimum, since this is a
> uapi header, and people may depend on anything defined here.
>
> I think SVE_VQ_BYTES is reasonable to have, and this allows us to
> rewrite a few hard-coded 0x10s and 16s symbolically which is probably a
> good idea -- I'll add this.
>
> SVE_VQ_BITS looks redundant to me though.  It wouldn't be used for any
> purpose other than defining SVE_VQ_BYTES.

Yeah I was more concerned with getting rid of the magic 0x10's than
showing exactly how many bits something is.

>
>>  #define SVE_VQ_MIN		1
>>  #define SVE_VQ_MAX		0x200
>>
>> -#define SVE_VL_MIN		(SVE_VQ_MIN * 0x10)
>> -#define SVE_VL_MAX		(SVE_VQ_MAX * 0x10)
>> +#define SVE_VL_MIN		(SVE_VQ_MIN * SVE_VQ_BYTES)
>> +#define SVE_VL_MAX		(SVE_VQ_MAX * SVE_VQ_BYTES)
>>
>>  #define SVE_NUM_ZREGS		32
>>  #define SVE_NUM_PREGS		16
>>
>>  #define sve_vl_valid(vl) \
>> -	((vl) % 0x10 == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
>> +	((vl) % SVE_VQ_BYTES == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX)
>>  #define sve_vq_from_vl(vl)	((vl) / 0x10)
>>  #define sve_vl_from_vq(vq)	((vq) * 0x10)
>
> [...]
>
> Cheers
> ---Dave


--
Alex Bennée

Re: [PATCH 10/27] arm64/sve: Low-level CPU setup

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> To enable the kernel to use SVE, all SVE traps from EL1 must be
> disabled.  To take maximum advantage of the hardware, the full
> available vector length also needs to be enabled for EL1 by
> programming ZCR_EL2.LEN.  (The kernel will program ZCR_EL1.LEN as
> required, but this cannot override the limit set by ZCR_EL2.)
>
> In advance of full SVE support being implemented for userspace, it
> also necessary to ensure that SVE traps from EL0 are enabled.
>
> This patch makes the appropriate changes to the primary and
> secondary CPU initialisation code.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/kernel/head.S | 13 ++++++++++++-
>  arch/arm64/mm/proc.S     | 14 ++++++++++++--
>  2 files changed, 24 insertions(+), 3 deletions(-)
>
> diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
> index 973df7d..0ae1713 100644
> --- a/arch/arm64/kernel/head.S
> +++ b/arch/arm64/kernel/head.S
> @@ -514,8 +514,19 @@ CPU_LE(	movk	x0, #0x30d0, lsl #16	)	// Clear EE and E0E on LE systems
>  	mov	x0, #0x33ff
>  	msr	cptr_el2, x0			// Disable copro. traps to EL2
>
> +	/* SVE register access */
> +	mrs	x1, id_aa64pfr0_el1
> +	ubfx	x1, x1, #ID_AA64PFR0_SVE_SHIFT, #4
> +	cbz	x1, 7f
> +
> +	bic	x0, x0, #CPTR_EL2_TZ		// Also disable SVE traps
> +	msr	cptr_el2, x0			// Disable copro. traps
> to EL2

It seems a shame to write to cptr_el2 twice rather than compute and
write.

> +	isb

Especially as the second one needs an isb :-/

But I don't see a much neater way of doing it so:

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> +	mov	x1, #ZCR_ELx_LEN_MASK		// SVE: Enable full vector
> +	msr_s	SYS_ZCR_EL2, x1			// length for EL1.
> +
>  	/* Hypervisor stub */
> -	adr_l	x0, __hyp_stub_vectors
> +7:	adr_l	x0, __hyp_stub_vectors
>  	msr	vbar_el2, x0
>
>  	/* spsr */
> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
> index 877d42f..dd22ef2 100644
> --- a/arch/arm64/mm/proc.S
> +++ b/arch/arm64/mm/proc.S
> @@ -27,6 +27,7 @@
>  #include <asm/pgtable-hwdef.h>
>  #include <asm/cpufeature.h>
>  #include <asm/alternative.h>
> +#include <asm/sysreg.h>
>
>  #ifdef CONFIG_ARM64_64K_PAGES
>  #define TCR_TG_FLAGS	TCR_TG0_64K | TCR_TG1_64K
> @@ -186,8 +187,17 @@ ENTRY(__cpu_setup)
>  	tlbi	vmalle1				// Invalidate local TLB
>  	dsb	nsh
>
> -	mov	x0, #3 << 20
> -	msr	cpacr_el1, x0			// Enable FP/ASIMD
> +	mov	x0, #3 << 20			// FEN
> +
> +	/* SVE */
> +	mrs	x5, id_aa64pfr0_el1
> +	ubfx	x5, x5, #ID_AA64PFR0_SVE_SHIFT, #4
> +	cbz	x5, 1f
> +
> +	bic	x0, x0, #CPACR_EL1_ZEN
> +	orr	x0, x0, #CPACR_EL1_ZEN_EL1EN	// SVE: trap for EL0, not EL1
> +1:	msr	cpacr_el1, x0			// Enable FP/ASIMD
> +
>  	mov	x0, #1 << 12			// Reset mdscr_el1 and disable
>  	msr	mdscr_el1, x0			// access to the DCC from EL0
>  	isb					// Unmask debug exceptions now,

Re: [PATCH 10/27] arm64/sve: Low-level CPU setup

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> To enable the kernel to use SVE, all SVE traps from EL1 must be
> disabled.  To take maximum advantage of the hardware, the full
> available vector length also needs to be enabled for EL1 by
> programming ZCR_EL2.LEN.  (The kernel will program ZCR_EL1.LEN as
> required, but this cannot override the limit set by ZCR_EL2.)
>
> In advance of full SVE support being implemented for userspace, it
> also necessary to ensure that SVE traps from EL0 are enabled.
>
> This patch makes the appropriate changes to the primary and
> secondary CPU initialisation code.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/kernel/head.S | 13 ++++++++++++-
>  arch/arm64/mm/proc.S     | 14 ++++++++++++--
>  2 files changed, 24 insertions(+), 3 deletions(-)
>
> diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
> index 973df7d..0ae1713 100644
> --- a/arch/arm64/kernel/head.S
> +++ b/arch/arm64/kernel/head.S
> @@ -514,8 +514,19 @@ CPU_LE(	movk	x0, #0x30d0, lsl #16	)	// Clear EE and E0E on LE systems
>  	mov	x0, #0x33ff
>  	msr	cptr_el2, x0			// Disable copro. traps to EL2
>
> +	/* SVE register access */
> +	mrs	x1, id_aa64pfr0_el1
> +	ubfx	x1, x1, #ID_AA64PFR0_SVE_SHIFT, #4
> +	cbz	x1, 7f
> +
> +	bic	x0, x0, #CPTR_EL2_TZ		// Also disable SVE traps
> +	msr	cptr_el2, x0			// Disable copro. traps
> to EL2

It seems a shame to write to cptr_el2 twice rather than compute and
write.

> +	isb

Especially as the second one needs an isb :-/

But I don't see a much neater way of doing it so:

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

> +	mov	x1, #ZCR_ELx_LEN_MASK		// SVE: Enable full vector
> +	msr_s	SYS_ZCR_EL2, x1			// length for EL1.
> +
>  	/* Hypervisor stub */
> -	adr_l	x0, __hyp_stub_vectors
> +7:	adr_l	x0, __hyp_stub_vectors
>  	msr	vbar_el2, x0
>
>  	/* spsr */
> diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
> index 877d42f..dd22ef2 100644
> --- a/arch/arm64/mm/proc.S
> +++ b/arch/arm64/mm/proc.S
> @@ -27,6 +27,7 @@
>  #include <asm/pgtable-hwdef.h>
>  #include <asm/cpufeature.h>
>  #include <asm/alternative.h>
> +#include <asm/sysreg.h>
>
>  #ifdef CONFIG_ARM64_64K_PAGES
>  #define TCR_TG_FLAGS	TCR_TG0_64K | TCR_TG1_64K
> @@ -186,8 +187,17 @@ ENTRY(__cpu_setup)
>  	tlbi	vmalle1				// Invalidate local TLB
>  	dsb	nsh
>
> -	mov	x0, #3 << 20
> -	msr	cpacr_el1, x0			// Enable FP/ASIMD
> +	mov	x0, #3 << 20			// FEN
> +
> +	/* SVE */
> +	mrs	x5, id_aa64pfr0_el1
> +	ubfx	x5, x5, #ID_AA64PFR0_SVE_SHIFT, #4
> +	cbz	x5, 1f
> +
> +	bic	x0, x0, #CPACR_EL1_ZEN
> +	orr	x0, x0, #CPACR_EL1_ZEN_EL1EN	// SVE: trap for EL0, not EL1
> +1:	msr	cpacr_el1, x0			// Enable FP/ASIMD
> +
>  	mov	x0, #1 << 12			// Reset mdscr_el1 and disable
>  	msr	mdscr_el1, x0			// access to the DCC from EL0
>  	isb					// Unmask debug exceptions now,


--
Alex Bennée

Re: [PATCH 10/27] arm64/sve: Low-level CPU setup

[Dave Martin]

On Tue, Aug 22, 2017 at 04:04:28PM +0100, Alex Bennée wrote:
> 
> Dave Martin <Dave.Martin@arm.com> writes:
> 
> > To enable the kernel to use SVE, all SVE traps from EL1 must be
> > disabled.  To take maximum advantage of the hardware, the full
> > available vector length also needs to be enabled for EL1 by
> > programming ZCR_EL2.LEN.  (The kernel will program ZCR_EL1.LEN as
> > required, but this cannot override the limit set by ZCR_EL2.)
> >
> > In advance of full SVE support being implemented for userspace, it
> > also necessary to ensure that SVE traps from EL0 are enabled.
> >
> > This patch makes the appropriate changes to the primary and
> > secondary CPU initialisation code.
> >
> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> > ---
> >  arch/arm64/kernel/head.S | 13 ++++++++++++-
> >  arch/arm64/mm/proc.S     | 14 ++++++++++++--
> >  2 files changed, 24 insertions(+), 3 deletions(-)
> >
> > diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
> > index 973df7d..0ae1713 100644
> > --- a/arch/arm64/kernel/head.S
> > +++ b/arch/arm64/kernel/head.S
> > @@ -514,8 +514,19 @@ CPU_LE(	movk	x0, #0x30d0, lsl #16	)	// Clear EE and E0E on LE systems
> >  	mov	x0, #0x33ff
> >  	msr	cptr_el2, x0			// Disable copro. traps to EL2
> >
> > +	/* SVE register access */
> > +	mrs	x1, id_aa64pfr0_el1
> > +	ubfx	x1, x1, #ID_AA64PFR0_SVE_SHIFT, #4
> > +	cbz	x1, 7f
> > +
> > +	bic	x0, x0, #CPTR_EL2_TZ		// Also disable SVE traps
> > +	msr	cptr_el2, x0			// Disable copro. traps
> > to EL2
> 
> It seems a shame to write to cptr_el2 twice rather than compute and
> write.
> 
> > +	isb
> 
> Especially as the second one needs an isb :-/
> 
> But I don't see a much neater way of doing it so:
> 
> Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

Thanks

Originally I think I did merge the CPTR writes here, but since this is
slow-path anyway, I figured it was better to keep the code simple rather
than introducing new interdependencies.

I could revisit it if someone has a strong view.

Cheers
---Dave

> 
> > +	mov	x1, #ZCR_ELx_LEN_MASK		// SVE: Enable full vector
> > +	msr_s	SYS_ZCR_EL2, x1			// length for EL1.
> > +
> >  	/* Hypervisor stub */
> > -	adr_l	x0, __hyp_stub_vectors
> > +7:	adr_l	x0, __hyp_stub_vectors
> >  	msr	vbar_el2, x0

[...]

Re: [PATCH 09/27] arm64/sve: Signal frame and context structure definition

[Dave Martin]

On Tue, Aug 22, 2017 at 04:03:20PM +0100, Alex Bennée wrote:
> 
> Dave Martin <Dave.Martin@arm.com> writes:
> 
> > On Tue, Aug 22, 2017 at 02:53:49PM +0100, Alex Bennée wrote:

[...]

> >> +
> >> +#define SVE_VQ_BITS             128      /* 128 bits in one quadword */
> >> +#define SVE_VQ_BYTES            (SVE_VQ_BITS / 8)
> >> +
> >
> > I was trying to keep extraneous #defines to a minimum, since this is a
> > uapi header, and people may depend on anything defined here.
> >
> > I think SVE_VQ_BYTES is reasonable to have, and this allows us to
> > rewrite a few hard-coded 0x10s and 16s symbolically which is probably a
> > good idea -- I'll add this.
> >
> > SVE_VQ_BITS looks redundant to me though.  It wouldn't be used for any
> > purpose other than defining SVE_VQ_BYTES.
> 
> Yeah I was more concerned with getting rid of the magic 0x10's than
> showing exactly how many bits something is.

OK, I'll take SVE_VQ_BYTES and use it in the appropriate places.
There are a few 0x10s/16s in the series that can use this instead
of being open-coded.

Cheers
---Dave

Re: [PATCH 11/27] arm64/sve: Core task context handling

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> This patch adds the core support for switching and managing the SVE
> architectural state of user tasks.
>
> Calls to the existing FPSIMD low-level save/restore functions are
> factored out as new functions task_fpsimd_{save,load}(), since SVE
> now dynamically may or may not need to be handled at these points
> depending on the kernel configuration, hardware features discovered
> at boot, and the runtime state of the task.  To make these
> decisions as fast as possible, const cpucaps are used where
> feasible, via the system_supports_sve() helper.
>
> The SVE registers are only tracked for threads that have explicitly
> used SVE, indicated by the new thread flag TIF_SVE.  Otherwise, the
> FPSIMD view of the architectural state is stored in
> thread.fpsimd_state as usual.
>
> When in use, the SVE registers are not stored directly in
> thread_struct due to their potentially large and variable size.
> Because the task_struct slab allocator must be configured very
> early during kernel boot, it is also tricky to configure it
> correctly to match the maximum vector length provided by the
> hardware, since this depends on examining secondary CPUs as well as
> the primary.  Instead, a pointer sve_state in thread_struct points
> to a dynamically allocated buffer containing the SVE register data,
> and code is added to allocate, duplicate and free this buffer at
> appropriate times.
>
> TIF_SVE is set when taking an SVE access trap from userspace, if
> suitable hardware support has been detected.  This enables SVE for
> the thread: a subsequent return to userspace will disable the trap
> accordingly.  If such a trap is taken without sufficient hardware
> support, SIGILL is sent to the thread instead as if an undefined
> instruction had been executed: this may happen if userspace tries
> to use SVE in a system where not all CPUs support it for example.
>
> The kernel may clear TIF_SVE and disable SVE for the thread
> whenever an explicit syscall is made by userspace, though this is
> considered an optimisation opportunity rather than a deterministic
> guarantee: the kernel may not do this on every syscall, but it is
> permitted to do so.  For backwards compatibility reasons and
> conformance with the spirit of the base AArch64 procedure call
> standard, the subset of the SVE register state that aliases the
> FPSIMD registers is still preserved across a syscall even if this
> happens.
>
> TIF_SVE is also cleared, and SVE disabled, on exec: this is an
> obvious slow path and a hint that we are running a new binary that
> may not use SVE.
>
> Code is added to sync data between thread.fpsimd_state and
> thread.sve_state whenever enabling/disabling SVE, in a manner
> consistent with the SVE architectural programmer's model.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/include/asm/fpsimd.h      |  19 +++
>  arch/arm64/include/asm/processor.h   |   2 +
>  arch/arm64/include/asm/thread_info.h |   1 +
>  arch/arm64/include/asm/traps.h       |   2 +
>  arch/arm64/kernel/entry.S            |  14 +-
>  arch/arm64/kernel/fpsimd.c           | 241 ++++++++++++++++++++++++++++++++++-
>  arch/arm64/kernel/process.c          |   6 +-
>  arch/arm64/kernel/traps.c            |   4 +-
>  8 files changed, 279 insertions(+), 10 deletions(-)
>
> diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
> index 026a7c7..72090a1 100644
> --- a/arch/arm64/include/asm/fpsimd.h
> +++ b/arch/arm64/include/asm/fpsimd.h
> @@ -20,6 +20,8 @@
>
>  #ifndef __ASSEMBLY__
>
> +#include <linux/stddef.h>
> +
>  /*
>   * FP/SIMD storage area has:
>   *  - FPSR and FPCR
> @@ -72,6 +74,23 @@ extern void sve_load_state(void const *state, u32 const *pfpsr,
>  			   unsigned long vq_minus_1);
>  extern unsigned int sve_get_vl(void);
>
> +#ifdef CONFIG_ARM64_SVE
> +
> +extern size_t sve_state_size(struct task_struct const *task);
> +
> +extern void sve_alloc(struct task_struct *task);
> +extern void fpsimd_release_thread(struct task_struct *task);
> +extern void fpsimd_dup_sve(struct task_struct *dst,
> +			   struct task_struct const *src);
> +
> +#else /* ! CONFIG_ARM64_SVE */
> +
> +static void __maybe_unused sve_alloc(struct task_struct *task) { }
> +static void __maybe_unused fpsimd_release_thread(struct task_struct *task) { }
> +static void __maybe_unused fpsimd_dup_sve(struct task_struct *dst,
> +					  struct task_struct const *src) { }
> +#endif /* ! CONFIG_ARM64_SVE */
> +
>  /* For use by EFI runtime services calls only */
>  extern void __efi_fpsimd_begin(void);
>  extern void __efi_fpsimd_end(void);
> diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
> index b7334f1..969feed 100644
> --- a/arch/arm64/include/asm/processor.h
> +++ b/arch/arm64/include/asm/processor.h
> @@ -85,6 +85,8 @@ struct thread_struct {
>  	unsigned long		tp2_value;
>  #endif
>  	struct fpsimd_state	fpsimd_state;
> +	void			*sve_state;	/* SVE registers, if any */
> +	u16			sve_vl;		/* SVE vector length */

sve_vl is implicitly cast to unsigned int bellow - it should be
consistent.

Given the allocation functions rely on sve_vl being valid it might be
worth noting where this is set/live from?

>  	unsigned long		fault_address;	/* fault info */
>  	unsigned long		fault_code;	/* ESR_EL1 value */
>  	struct debug_info	debug;		/* debugging */
> diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h
> index 46c3b93..1a4b30b 100644
> --- a/arch/arm64/include/asm/thread_info.h
> +++ b/arch/arm64/include/asm/thread_info.h
<snip>

And I see there are other comments from Ard.

Re: [PATCH 11/27] arm64/sve: Core task context handling

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> This patch adds the core support for switching and managing the SVE
> architectural state of user tasks.
>
> Calls to the existing FPSIMD low-level save/restore functions are
> factored out as new functions task_fpsimd_{save,load}(), since SVE
> now dynamically may or may not need to be handled at these points
> depending on the kernel configuration, hardware features discovered
> at boot, and the runtime state of the task.  To make these
> decisions as fast as possible, const cpucaps are used where
> feasible, via the system_supports_sve() helper.
>
> The SVE registers are only tracked for threads that have explicitly
> used SVE, indicated by the new thread flag TIF_SVE.  Otherwise, the
> FPSIMD view of the architectural state is stored in
> thread.fpsimd_state as usual.
>
> When in use, the SVE registers are not stored directly in
> thread_struct due to their potentially large and variable size.
> Because the task_struct slab allocator must be configured very
> early during kernel boot, it is also tricky to configure it
> correctly to match the maximum vector length provided by the
> hardware, since this depends on examining secondary CPUs as well as
> the primary.  Instead, a pointer sve_state in thread_struct points
> to a dynamically allocated buffer containing the SVE register data,
> and code is added to allocate, duplicate and free this buffer at
> appropriate times.
>
> TIF_SVE is set when taking an SVE access trap from userspace, if
> suitable hardware support has been detected.  This enables SVE for
> the thread: a subsequent return to userspace will disable the trap
> accordingly.  If such a trap is taken without sufficient hardware
> support, SIGILL is sent to the thread instead as if an undefined
> instruction had been executed: this may happen if userspace tries
> to use SVE in a system where not all CPUs support it for example.
>
> The kernel may clear TIF_SVE and disable SVE for the thread
> whenever an explicit syscall is made by userspace, though this is
> considered an optimisation opportunity rather than a deterministic
> guarantee: the kernel may not do this on every syscall, but it is
> permitted to do so.  For backwards compatibility reasons and
> conformance with the spirit of the base AArch64 procedure call
> standard, the subset of the SVE register state that aliases the
> FPSIMD registers is still preserved across a syscall even if this
> happens.
>
> TIF_SVE is also cleared, and SVE disabled, on exec: this is an
> obvious slow path and a hint that we are running a new binary that
> may not use SVE.
>
> Code is added to sync data between thread.fpsimd_state and
> thread.sve_state whenever enabling/disabling SVE, in a manner
> consistent with the SVE architectural programmer's model.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/include/asm/fpsimd.h      |  19 +++
>  arch/arm64/include/asm/processor.h   |   2 +
>  arch/arm64/include/asm/thread_info.h |   1 +
>  arch/arm64/include/asm/traps.h       |   2 +
>  arch/arm64/kernel/entry.S            |  14 +-
>  arch/arm64/kernel/fpsimd.c           | 241 ++++++++++++++++++++++++++++++++++-
>  arch/arm64/kernel/process.c          |   6 +-
>  arch/arm64/kernel/traps.c            |   4 +-
>  8 files changed, 279 insertions(+), 10 deletions(-)
>
> diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
> index 026a7c7..72090a1 100644
> --- a/arch/arm64/include/asm/fpsimd.h
> +++ b/arch/arm64/include/asm/fpsimd.h
> @@ -20,6 +20,8 @@
>
>  #ifndef __ASSEMBLY__
>
> +#include <linux/stddef.h>
> +
>  /*
>   * FP/SIMD storage area has:
>   *  - FPSR and FPCR
> @@ -72,6 +74,23 @@ extern void sve_load_state(void const *state, u32 const *pfpsr,
>  			   unsigned long vq_minus_1);
>  extern unsigned int sve_get_vl(void);
>
> +#ifdef CONFIG_ARM64_SVE
> +
> +extern size_t sve_state_size(struct task_struct const *task);
> +
> +extern void sve_alloc(struct task_struct *task);
> +extern void fpsimd_release_thread(struct task_struct *task);
> +extern void fpsimd_dup_sve(struct task_struct *dst,
> +			   struct task_struct const *src);
> +
> +#else /* ! CONFIG_ARM64_SVE */
> +
> +static void __maybe_unused sve_alloc(struct task_struct *task) { }
> +static void __maybe_unused fpsimd_release_thread(struct task_struct *task) { }
> +static void __maybe_unused fpsimd_dup_sve(struct task_struct *dst,
> +					  struct task_struct const *src) { }
> +#endif /* ! CONFIG_ARM64_SVE */
> +
>  /* For use by EFI runtime services calls only */
>  extern void __efi_fpsimd_begin(void);
>  extern void __efi_fpsimd_end(void);
> diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
> index b7334f1..969feed 100644
> --- a/arch/arm64/include/asm/processor.h
> +++ b/arch/arm64/include/asm/processor.h
> @@ -85,6 +85,8 @@ struct thread_struct {
>  	unsigned long		tp2_value;
>  #endif
>  	struct fpsimd_state	fpsimd_state;
> +	void			*sve_state;	/* SVE registers, if any */
> +	u16			sve_vl;		/* SVE vector length */

sve_vl is implicitly cast to unsigned int bellow - it should be
consistent.

Given the allocation functions rely on sve_vl being valid it might be
worth noting where this is set/live from?

>  	unsigned long		fault_address;	/* fault info */
>  	unsigned long		fault_code;	/* ESR_EL1 value */
>  	struct debug_info	debug;		/* debugging */
> diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h
> index 46c3b93..1a4b30b 100644
> --- a/arch/arm64/include/asm/thread_info.h
> +++ b/arch/arm64/include/asm/thread_info.h
<snip>

And I see there are other comments from Ard.

--
Alex Bennée

Re: [PATCH 12/27] arm64/sve: Support vector length resetting for new processes

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> It's desirable to be able to reset the vector length to some sane
> default for new processes, since the new binary and its libraries
> processes may or may not be SVE-aware.
>
> This patch tracks the desired post-exec vector length (if any) in a
> new thread member sve_vl_onexec, and adds a new thread flag
> TIF_SVE_VL_INHERIT to control whether to inherit or reset the
> vector length.  Currently these are inactive.  Subsequent patches
> will provide the capability to configure them.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/include/asm/processor.h   |  1 +
>  arch/arm64/include/asm/thread_info.h |  1 +
>  arch/arm64/kernel/fpsimd.c           | 13 +++++++++++++
>  3 files changed, 15 insertions(+)
>
> diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
> index 969feed..da8802a 100644
> --- a/arch/arm64/include/asm/processor.h
> +++ b/arch/arm64/include/asm/processor.h
> @@ -87,6 +87,7 @@ struct thread_struct {
>  	struct fpsimd_state	fpsimd_state;
>  	void			*sve_state;	/* SVE registers, if any */
>  	u16			sve_vl;		/* SVE vector length */
> +	u16			sve_vl_onexec;	/* SVE vl after next
> exec */

Inconsistent size usage here as well...

>  	unsigned long		fault_address;	/* fault info */
>  	unsigned long		fault_code;	/* ESR_EL1 value */
>  	struct debug_info	debug;		/* debugging */
> diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h
> index 1a4b30b..bf9c552 100644
> --- a/arch/arm64/include/asm/thread_info.h
> +++ b/arch/arm64/include/asm/thread_info.h
> @@ -97,6 +97,7 @@ struct thread_info {
>  #define TIF_SINGLESTEP		21
>  #define TIF_32BIT		22	/* 32bit process */
>  #define TIF_SVE			23	/* Scalable Vector Extension in use */
> +#define TIF_SVE_VL_INHERIT	24	/* Inherit sve_vl_onexec across exec */
>
>  #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
>  #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
> diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> index 37dd1b2..80ecb2d 100644
> --- a/arch/arm64/kernel/fpsimd.c
> +++ b/arch/arm64/kernel/fpsimd.c
> @@ -108,6 +108,9 @@ static void task_fpsimd_save(void);
>   */
>  static DEFINE_PER_CPU(struct fpsimd_state *, fpsimd_last_state);
>
> +/* Default VL for tasks that don't set it explicitly: */
> +static int sve_default_vl = -1;
> +
>  static void sve_free(struct task_struct *task)
>  {
>  	kfree(task->thread.sve_state);
> @@ -392,6 +395,9 @@ void fpsimd_flush_thread(void)
>  		clear_thread_flag(TIF_SVE);
>  		sve_free(current);
>
> +		current->thread.sve_vl = current->thread.sve_vl_onexec ?
> +			current->thread.sve_vl_onexec : sve_default_vl;
> +
>  		/*
>  		 * User tasks must have a valid vector length set, but tasks
>  		 * forked early (e.g., init) may not initially have one.
> @@ -401,6 +407,13 @@ void fpsimd_flush_thread(void)
>  		 * If not, something went badly wrong.
>  		 */
>  		BUG_ON(!sve_vl_valid(current->thread.sve_vl));
> +
> +		/*
> +		 * If the task is not set to inherit, ensure that the vector
> +		 * length will be reset by a subsequent exec:
> +		 */
> +		if (!test_thread_flag(TIF_SVE_VL_INHERIT))
> +			current->thread.sve_vl_onexec = 0;
>  	}
>
>  	set_thread_flag(TIF_FOREIGN_FPSTATE);


--
Alex Bennée

Re: [PATCH 12/27] arm64/sve: Support vector length resetting for new processes

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> It's desirable to be able to reset the vector length to some sane
> default for new processes, since the new binary and its libraries
> processes may or may not be SVE-aware.
>
> This patch tracks the desired post-exec vector length (if any) in a
> new thread member sve_vl_onexec, and adds a new thread flag
> TIF_SVE_VL_INHERIT to control whether to inherit or reset the
> vector length.  Currently these are inactive.  Subsequent patches
> will provide the capability to configure them.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/include/asm/processor.h   |  1 +
>  arch/arm64/include/asm/thread_info.h |  1 +
>  arch/arm64/kernel/fpsimd.c           | 13 +++++++++++++
>  3 files changed, 15 insertions(+)
>
> diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
> index 969feed..da8802a 100644
> --- a/arch/arm64/include/asm/processor.h
> +++ b/arch/arm64/include/asm/processor.h
> @@ -87,6 +87,7 @@ struct thread_struct {
>  	struct fpsimd_state	fpsimd_state;
>  	void			*sve_state;	/* SVE registers, if any */
>  	u16			sve_vl;		/* SVE vector length */
> +	u16			sve_vl_onexec;	/* SVE vl after next
> exec */

Inconsistent size usage here as well...

>  	unsigned long		fault_address;	/* fault info */
>  	unsigned long		fault_code;	/* ESR_EL1 value */
>  	struct debug_info	debug;		/* debugging */
> diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h
> index 1a4b30b..bf9c552 100644
> --- a/arch/arm64/include/asm/thread_info.h
> +++ b/arch/arm64/include/asm/thread_info.h
> @@ -97,6 +97,7 @@ struct thread_info {
>  #define TIF_SINGLESTEP		21
>  #define TIF_32BIT		22	/* 32bit process */
>  #define TIF_SVE			23	/* Scalable Vector Extension in use */
> +#define TIF_SVE_VL_INHERIT	24	/* Inherit sve_vl_onexec across exec */
>
>  #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
>  #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
> diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> index 37dd1b2..80ecb2d 100644
> --- a/arch/arm64/kernel/fpsimd.c
> +++ b/arch/arm64/kernel/fpsimd.c
> @@ -108,6 +108,9 @@ static void task_fpsimd_save(void);
>   */
>  static DEFINE_PER_CPU(struct fpsimd_state *, fpsimd_last_state);
>
> +/* Default VL for tasks that don't set it explicitly: */
> +static int sve_default_vl = -1;
> +
>  static void sve_free(struct task_struct *task)
>  {
>  	kfree(task->thread.sve_state);
> @@ -392,6 +395,9 @@ void fpsimd_flush_thread(void)
>  		clear_thread_flag(TIF_SVE);
>  		sve_free(current);
>
> +		current->thread.sve_vl = current->thread.sve_vl_onexec ?
> +			current->thread.sve_vl_onexec : sve_default_vl;
> +
>  		/*
>  		 * User tasks must have a valid vector length set, but tasks
>  		 * forked early (e.g., init) may not initially have one.
> @@ -401,6 +407,13 @@ void fpsimd_flush_thread(void)
>  		 * If not, something went badly wrong.
>  		 */
>  		BUG_ON(!sve_vl_valid(current->thread.sve_vl));
> +
> +		/*
> +		 * If the task is not set to inherit, ensure that the vector
> +		 * length will be reset by a subsequent exec:
> +		 */
> +		if (!test_thread_flag(TIF_SVE_VL_INHERIT))
> +			current->thread.sve_vl_onexec = 0;
>  	}
>
>  	set_thread_flag(TIF_FOREIGN_FPSTATE);


--
Alex Bennée

Re: [PATCH 11/27] arm64/sve: Core task context handling

[Dave Martin]

On Tue, Aug 22, 2017 at 05:21:19PM +0100, Alex Bennée wrote:
> 
> Dave Martin <Dave.Martin@arm.com> writes:

[...]

> > --- a/arch/arm64/include/asm/processor.h
> > +++ b/arch/arm64/include/asm/processor.h
> > @@ -85,6 +85,8 @@ struct thread_struct {
> >  	unsigned long		tp2_value;
> >  #endif
> >  	struct fpsimd_state	fpsimd_state;
> > +	void			*sve_state;	/* SVE registers, if any */
> > +	u16			sve_vl;		/* SVE vector length */
> 
> sve_vl is implicitly cast to unsigned int bellow - it should be
> consistent.

Agreed -- I think this can just be unsigned int here, which is the type
I use everywhere except when encoding the vector length in the signal
frame and ptrace data.

During development, there was an additional flags field here (now
merged into the thread_info flags).  u16 was big enough for the largest
architectural vector length, so it seemed "tidy" to make both u16s.
Elsewhere, this created a risk of overflow if you try to compute say
the size of the whole register file from a u16, so I generally used
unsigned int for local variables in functions that handle the vl.

> Given the allocation functions rely on sve_vl being valid it might be
> worth noting where this is set/live from?

Agreed, I need to look more closely at exactly how to justify the
soundness of this in order to thin out BUG_ONs.

If you need any pointers, please ping me.  In the meantime, I would
rather not colour your judgement by infecting you with my assumptions ;)

> >  	unsigned long		fault_address;	/* fault info */
> >  	unsigned long		fault_code;	/* ESR_EL1 value */
> >  	struct debug_info	debug;		/* debugging */
> > diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h
> > index 46c3b93..1a4b30b 100644
> > --- a/arch/arm64/include/asm/thread_info.h
> > +++ b/arch/arm64/include/asm/thread_info.h
> <snip>
> 
> And I see there are other comments from Ard.

Yup, I've already picked those up.

I'll be posting a v2 with feedback applied once I've reviewed the
existing BUG_ON()s -- should happen sometime this week.

Cheers
---Dave

Re: [PATCH 12/27] arm64/sve: Support vector length resetting for new processes

[Dave Martin]

On Tue, Aug 22, 2017 at 05:22:11PM +0100, Alex Bennée wrote:
> 
> Dave Martin <Dave.Martin@arm.com> writes:
> 
> > It's desirable to be able to reset the vector length to some sane
> > default for new processes, since the new binary and its libraries
> > processes may or may not be SVE-aware.
> >
> > This patch tracks the desired post-exec vector length (if any) in a
> > new thread member sve_vl_onexec, and adds a new thread flag
> > TIF_SVE_VL_INHERIT to control whether to inherit or reset the
> > vector length.  Currently these are inactive.  Subsequent patches
> > will provide the capability to configure them.
> >
> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> > ---
> >  arch/arm64/include/asm/processor.h   |  1 +
> >  arch/arm64/include/asm/thread_info.h |  1 +
> >  arch/arm64/kernel/fpsimd.c           | 13 +++++++++++++
> >  3 files changed, 15 insertions(+)
> >
> > diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
> > index 969feed..da8802a 100644
> > --- a/arch/arm64/include/asm/processor.h
> > +++ b/arch/arm64/include/asm/processor.h
> > @@ -87,6 +87,7 @@ struct thread_struct {
> >  	struct fpsimd_state	fpsimd_state;
> >  	void			*sve_state;	/* SVE registers, if any */
> >  	u16			sve_vl;		/* SVE vector length */
> > +	u16			sve_vl_onexec;	/* SVE vl after next
> > exec */
> 
> Inconsistent size usage here as well...

Agreed, the same history applies as for sve_vl.  I'll replace this with
an unsigned int.

[...]

Cheers
---Dave

Re: [PATCH 12/27] arm64/sve: Support vector length resetting for new processes

[Dave Martin]

On Tue, Aug 22, 2017 at 05:22:11PM +0100, Alex Bennée wrote:
> 
> Dave Martin <Dave.Martin@arm.com> writes:
> 
> > It's desirable to be able to reset the vector length to some sane
> > default for new processes, since the new binary and its libraries
> > processes may or may not be SVE-aware.
> >
> > This patch tracks the desired post-exec vector length (if any) in a
> > new thread member sve_vl_onexec, and adds a new thread flag
> > TIF_SVE_VL_INHERIT to control whether to inherit or reset the
> > vector length.  Currently these are inactive.  Subsequent patches
> > will provide the capability to configure them.
> >
> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> > ---
> >  arch/arm64/include/asm/processor.h   |  1 +
> >  arch/arm64/include/asm/thread_info.h |  1 +
> >  arch/arm64/kernel/fpsimd.c           | 13 +++++++++++++
> >  3 files changed, 15 insertions(+)
> >
> > diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
> > index 969feed..da8802a 100644
> > --- a/arch/arm64/include/asm/processor.h
> > +++ b/arch/arm64/include/asm/processor.h
> > @@ -87,6 +87,7 @@ struct thread_struct {
> >  	struct fpsimd_state	fpsimd_state;
> >  	void			*sve_state;	/* SVE registers, if any */
> >  	u16			sve_vl;		/* SVE vector length */
> > +	u16			sve_vl_onexec;	/* SVE vl after next
> > exec */
> 
> Inconsistent size usage here as well...

Agreed, the same history applies as for sve_vl.  I'll replace this with
an unsigned int.

[...]

Cheers
---Dave

Re: [PATCH 11/27] arm64/sve: Core task context handling

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> On Tue, Aug 22, 2017 at 05:21:19PM +0100, Alex Bennée wrote:
>>
>> Dave Martin <Dave.Martin@arm.com> writes:
>
> [...]
>
>> > --- a/arch/arm64/include/asm/processor.h
>> > +++ b/arch/arm64/include/asm/processor.h
>> > @@ -85,6 +85,8 @@ struct thread_struct {
>> >  	unsigned long		tp2_value;
>> >  #endif
>> >  	struct fpsimd_state	fpsimd_state;
>> > +	void			*sve_state;	/* SVE registers, if any */
>> > +	u16			sve_vl;		/* SVE vector length */
>>
>> sve_vl is implicitly cast to unsigned int bellow - it should be
>> consistent.
>
> Agreed -- I think this can just be unsigned int here, which is the type
> I use everywhere except when encoding the vector length in the signal
> frame and ptrace data.
>
> During development, there was an additional flags field here (now
> merged into the thread_info flags).  u16 was big enough for the largest
> architectural vector length, so it seemed "tidy" to make both u16s.
> Elsewhere, this created a risk of overflow if you try to compute say
> the size of the whole register file from a u16, so I generally used
> unsigned int for local variables in functions that handle the vl.
>
>> Given the allocation functions rely on sve_vl being valid it might be
>> worth noting where this is set/live from?
>
> Agreed, I need to look more closely at exactly how to justify the
> soundness of this in order to thin out BUG_ONs.
>
> If you need any pointers, please ping me.  In the meantime, I would
> rather not colour your judgement by infecting you with my assumptions ;)
>
>> >  	unsigned long		fault_address;	/* fault info */
>> >  	unsigned long		fault_code;	/* ESR_EL1 value */
>> >  	struct debug_info	debug;		/* debugging */
>> > diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h
>> > index 46c3b93..1a4b30b 100644
>> > --- a/arch/arm64/include/asm/thread_info.h
>> > +++ b/arch/arm64/include/asm/thread_info.h
>> <snip>
>>
>> And I see there are other comments from Ard.
>
> Yup, I've already picked those up.
>
> I'll be posting a v2 with feedback applied once I've reviewed the
> existing BUG_ON()s -- should happen sometime this week.

We shall see how far I get tomorrow - you won't get any more from me
after that until I get back from holiday so don't delay v2 on my account
;-)

>
> Cheers
> ---Dave

Re: [PATCH 11/27] arm64/sve: Core task context handling

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> On Tue, Aug 22, 2017 at 05:21:19PM +0100, Alex Bennée wrote:
>>
>> Dave Martin <Dave.Martin@arm.com> writes:
>
> [...]
>
>> > --- a/arch/arm64/include/asm/processor.h
>> > +++ b/arch/arm64/include/asm/processor.h
>> > @@ -85,6 +85,8 @@ struct thread_struct {
>> >  	unsigned long		tp2_value;
>> >  #endif
>> >  	struct fpsimd_state	fpsimd_state;
>> > +	void			*sve_state;	/* SVE registers, if any */
>> > +	u16			sve_vl;		/* SVE vector length */
>>
>> sve_vl is implicitly cast to unsigned int bellow - it should be
>> consistent.
>
> Agreed -- I think this can just be unsigned int here, which is the type
> I use everywhere except when encoding the vector length in the signal
> frame and ptrace data.
>
> During development, there was an additional flags field here (now
> merged into the thread_info flags).  u16 was big enough for the largest
> architectural vector length, so it seemed "tidy" to make both u16s.
> Elsewhere, this created a risk of overflow if you try to compute say
> the size of the whole register file from a u16, so I generally used
> unsigned int for local variables in functions that handle the vl.
>
>> Given the allocation functions rely on sve_vl being valid it might be
>> worth noting where this is set/live from?
>
> Agreed, I need to look more closely at exactly how to justify the
> soundness of this in order to thin out BUG_ONs.
>
> If you need any pointers, please ping me.  In the meantime, I would
> rather not colour your judgement by infecting you with my assumptions ;)
>
>> >  	unsigned long		fault_address;	/* fault info */
>> >  	unsigned long		fault_code;	/* ESR_EL1 value */
>> >  	struct debug_info	debug;		/* debugging */
>> > diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h
>> > index 46c3b93..1a4b30b 100644
>> > --- a/arch/arm64/include/asm/thread_info.h
>> > +++ b/arch/arm64/include/asm/thread_info.h
>> <snip>
>>
>> And I see there are other comments from Ard.
>
> Yup, I've already picked those up.
>
> I'll be posting a v2 with feedback applied once I've reviewed the
> existing BUG_ON()s -- should happen sometime this week.

We shall see how far I get tomorrow - you won't get any more from me
after that until I get back from holiday so don't delay v2 on my account
;-)

>
> Cheers
> ---Dave


--
Alex Bennée

Re: [PATCH 13/27] arm64/sve: Signal handling support

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> This patch implements support for saving and restoring the SVE
> registers around signals.
>
> A fixed-size header struct sve_context is always included in the
> signal frame encoding the thread's vector length at the time of
> signal delivery, optionally followed by a variable-layout structure
> encoding the SVE registers.
>
> Because of the need to preserve backwards compatibility, the FPSIMD
> view of the SVE registers is always dumped as a struct
> fpsimd_context in the usual way, in addition to any sve_context.
>
> The SVE vector registers are dumped in full, including bits 127:0
> of each register which alias the corresponding FPSIMD vector
> registers in the hardware.  To avoid any ambiguity about which
> alias to restore during sigreturn, the kernel always restores bits
> 127:0 of each SVE vector register from the fpsimd_context in the
> signal frame (which must be present): userspace needs to take this
> into account if it wants to modify the SVE vector register contents
> on return from a signal.
>
> FPSR and FPCR, which are used by both FPSIMD and SVE, are not
> included in sve_context because they are always present in
> fpsimd_context anyway.
>
> For signal delivery, a new helper
> fpsimd_signal_preserve_current_state() is added to update _both_
> the FPSIMD and SVE views in the task struct, to make it easier to
> populate this information into the signal frame.  Because of the
> redundancy between the two views of the state, only one is updated
> otherwise.  In order to avoid racing with a pending discard of the
> SVE state, this flush is hoisted before the sigframe layout phase,
> so that the layout and population phases see a consistent view of
> the thread.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/include/asm/fpsimd.h |   1 +
>  arch/arm64/kernel/fpsimd.c      |  23 ++++--
>  arch/arm64/kernel/signal.c      | 169 ++++++++++++++++++++++++++++++++++++++--
>  arch/arm64/kernel/signal32.c    |   2 +-
>  4 files changed, 179 insertions(+), 16 deletions(-)
>
> diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
> index 72090a1..7efd04e 100644
> --- a/arch/arm64/include/asm/fpsimd.h
> +++ b/arch/arm64/include/asm/fpsimd.h
> @@ -63,6 +63,7 @@ extern void fpsimd_load_state(struct fpsimd_state *state);
>  extern void fpsimd_thread_switch(struct task_struct *next);
>  extern void fpsimd_flush_thread(void);
>
> +extern void fpsimd_signal_preserve_current_state(void);
>  extern void fpsimd_preserve_current_state(void);
>  extern void fpsimd_restore_current_state(void);
>  extern void fpsimd_update_current_state(struct fpsimd_state *state);
> diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> index 80ecb2d..e8674f6 100644
> --- a/arch/arm64/kernel/fpsimd.c
> +++ b/arch/arm64/kernel/fpsimd.c
> @@ -148,8 +148,6 @@ static void change_cpacr(u64 old, u64 new)
>  		write_sysreg(new, CPACR_EL1);
>  }
>
> -#ifdef CONFIG_ARM64_SVE
> -
>  #define ZREG(sve_state, vq, n) ((char *)(sve_state) +		\
>  	(SVE_SIG_ZREG_OFFSET(vq, n) - SVE_SIG_REGS_OFFSET))
>
> @@ -191,6 +189,8 @@ static void fpsimd_to_sve(struct task_struct *task)
>  		       sizeof(fst->vregs[i]));
>  }
>
> +#ifdef CONFIG_ARM64_SVE
> +

Hmm have sve_to_fpsimd and fpsimd_to_sve only just started being used by
the generic code here?

>  size_t sve_state_size(struct task_struct const *task)
>  {
>  	unsigned int vl = task->thread.sve_vl;
> @@ -431,13 +431,17 @@ void fpsimd_preserve_current_state(void)
>  		return;
>
>  	local_bh_disable();
> -
> -	if (!test_thread_flag(TIF_FOREIGN_FPSTATE))
> -		fpsimd_save_state(&current->thread.fpsimd_state);
> -
> +	task_fpsimd_save();
>  	local_bh_enable();
>  }
>
> +void fpsimd_signal_preserve_current_state(void)
> +{
> +	fpsimd_preserve_current_state();
> +	if (system_supports_sve() && test_thread_flag(TIF_SVE))
> +		sve_to_fpsimd(current);
> +}
> +
>  /*
>   * Load the userland FPSIMD state of 'current' from memory, but only if the
>   * FPSIMD state already held in the registers is /not/ the most recent FPSIMD
> @@ -473,7 +477,12 @@ void fpsimd_update_current_state(struct fpsimd_state *state)
>
>  	local_bh_disable();
>
> -	fpsimd_load_state(state);
> +	if (system_supports_sve() && test_thread_flag(TIF_SVE)) {
> +		current->thread.fpsimd_state = *state;
> +		fpsimd_to_sve(current);
> +	}
> +	task_fpsimd_load();
> +
>  	if (test_and_clear_thread_flag(TIF_FOREIGN_FPSTATE)) {
>  		struct fpsimd_state *st = &current->thread.fpsimd_state;
>
> diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c
> index 4991e87..2694143 100644
> --- a/arch/arm64/kernel/signal.c
> +++ b/arch/arm64/kernel/signal.c
> @@ -62,6 +62,7 @@ struct rt_sigframe_user_layout {
>
>  	unsigned long fpsimd_offset;
>  	unsigned long esr_offset;
> +	unsigned long sve_offset;
>  	unsigned long extra_offset;
>  	unsigned long end_offset;
>  };
> @@ -178,9 +179,6 @@ static int preserve_fpsimd_context(struct fpsimd_context __user *ctx)
>  	struct fpsimd_state *fpsimd = &current->thread.fpsimd_state;
>  	int err;
>
> -	/* dump the hardware registers to the fpsimd_state structure */
> -	fpsimd_preserve_current_state();
> -
>  	/* copy the FP and status/control registers */
>  	err = __copy_to_user(ctx->vregs, fpsimd->vregs, sizeof(fpsimd->vregs));
>  	__put_user_error(fpsimd->fpsr, &ctx->fpsr, err);
> @@ -213,6 +211,8 @@ static int restore_fpsimd_context(struct fpsimd_context __user *ctx)
>  	__get_user_error(fpsimd.fpsr, &ctx->fpsr, err);
>  	__get_user_error(fpsimd.fpcr, &ctx->fpcr, err);
>
> +	clear_thread_flag(TIF_SVE);
> +
>  	/* load the hardware registers from the fpsimd_state structure */
>  	if (!err)
>  		fpsimd_update_current_state(&fpsimd);
> @@ -220,10 +220,113 @@ static int restore_fpsimd_context(struct fpsimd_context __user *ctx)
>  	return err ? -EFAULT : 0;
>  }
>
> +
>  struct user_ctxs {
>  	struct fpsimd_context __user *fpsimd;
> +	struct sve_context __user *sve;
>  };
>
> +#ifdef CONFIG_ARM64_SVE
> +
> +static int preserve_sve_context(struct sve_context __user *ctx)
> +{
> +	int err = 0;
> +	u16 reserved[ARRAY_SIZE(ctx->__reserved)];
> +	unsigned int vl = current->thread.sve_vl;
> +	unsigned int vq = 0;
> +
> +	BUG_ON(!sve_vl_valid(vl));
> +	if (test_thread_flag(TIF_SVE))
> +		vq = sve_vq_from_vl(vl);
> +
> +	memset(reserved, 0, sizeof(reserved));
> +
> +	__put_user_error(SVE_MAGIC, &ctx->head.magic, err);
> +	__put_user_error(round_up(SVE_SIG_CONTEXT_SIZE(vq), 16),
> +			 &ctx->head.size, err);
> +	__put_user_error(vl, &ctx->vl, err);
> +	BUILD_BUG_ON(sizeof(ctx->__reserved) != sizeof(reserved));
> +	err |= copy_to_user(&ctx->__reserved, reserved, sizeof(reserved));
> +
> +	if (vq) {
> +		/*
> +		 * This assumes that the SVE state has already been saved to
> +		 * the task struct by calling preserve_fpsimd_context().
> +		 */
> +		BUG_ON(SVE_SIG_REGS_SIZE(vq) !=
> sve_state_size(current));

I think others have mentioned the excessive BUG_ON()s here but I think
you are planning on cleaning some up on the next version. Assuming
sve_vq_from_vl() can't give you an invalid answer from a
sve_vl_valid(vl) then I wouldn't expect this test to add much.

> +		err |= copy_to_user((char __user *)ctx + SVE_SIG_REGS_OFFSET,
> +				    current->thread.sve_state,
> +				    SVE_SIG_REGS_SIZE(vq));
> +	}
> +
> +	return err ? -EFAULT : 0;
> +}
> +
> +static int restore_sve_fpsimd_context(struct user_ctxs *user)
> +{
> +	int err;
> +	unsigned int vq;
> +	struct fpsimd_state fpsimd;
> +	struct sve_context sve;
> +
> +	if (__copy_from_user(&sve, user->sve, sizeof(sve)))
> +		return -EFAULT;
> +
> +	if (sve.vl != current->thread.sve_vl)
> +		return -EINVAL;
> +
> +	if (sve.head.size <= sizeof(*user->sve)) {
> +		clear_thread_flag(TIF_SVE);
> +		goto fpsimd_only;
> +	}
> +
> +	BUG_ON(!sve_vl_valid(sve.vl));
> +	vq = sve_vq_from_vl(sve.vl);
> +
> +	if (sve.head.size < SVE_SIG_CONTEXT_SIZE(vq))
> +		return -EINVAL;
> +
> +	fpsimd_flush_task_state(current);
> +	barrier();
> +	set_thread_flag(TIF_FOREIGN_FPSTATE);
> +	barrier();

What are you trying to achieve with barriers here? Is there a potential
interaction between flushing the state and setting the flag that the
compiler can't see? A comment should be added at least.

> +
> +	sve_alloc(current);
> +	BUG_ON(SVE_SIG_REGS_SIZE(vq) != sve_state_size(current));
> +	err = __copy_from_user(current->thread.sve_state,
> +			       (char __user const *)user->sve +
> +					SVE_SIG_REGS_OFFSET,
> +			       SVE_SIG_REGS_SIZE(vq));
> +	if (err)
> +		return err;
> +
> +	barrier();
> +	set_thread_flag(TIF_SVE);

Hmm and again. If this is about visibility of context when the thread
flag is read by other CPUs a barrier() on it's own is not enough as it
only stop local code re-organisation - do you actually mean smp_mb()?

Either way you need to document the potential race in a comment so the
reason can be understood.

> +
> +fpsimd_only:
> +	/* copy the FP and status/control registers */
> +	/* restore_sigframe() already checked that user->fpsimd != NULL. */
> +	err = __copy_from_user(fpsimd.vregs, user->fpsimd->vregs,
> +			       sizeof(fpsimd.vregs));
> +	__get_user_error(fpsimd.fpsr, &user->fpsimd->fpsr, err);
> +	__get_user_error(fpsimd.fpcr, &user->fpsimd->fpcr, err);
> +
> +	/* load the hardware registers from the fpsimd_state structure */
> +	if (!err)
> +		fpsimd_update_current_state(&fpsimd);
> +
> +	return err;
> +}
> +
> +#else /* ! CONFIG_ARM64_SVE */
> +
> +/* Turn any non-optimised out attempts to use these into a link error: */
> +extern int preserve_sve_context(void __user *ctx);
> +extern int restore_sve_fpsimd_context(struct user_ctxs *user);
> +
> +#endif /* ! CONFIG_ARM64_SVE */
> +
> +
>  static int parse_user_sigframe(struct user_ctxs *user,
>  			       struct rt_sigframe __user *sf)
>  {
> @@ -236,6 +339,7 @@ static int parse_user_sigframe(struct user_ctxs *user,
>  	char const __user *const sfp = (char const __user *)sf;
>
>  	user->fpsimd = NULL;
> +	user->sve = NULL;
>
>  	if (!IS_ALIGNED((unsigned long)base, 16))
>  		goto invalid;
> @@ -286,6 +390,19 @@ static int parse_user_sigframe(struct user_ctxs *user,
>  			/* ignore */
>  			break;
>
> +		case SVE_MAGIC:
> +			if (!system_supports_sve())
> +				goto invalid;
> +
> +			if (user->sve)
> +				goto invalid;
> +
> +			if (size < sizeof(*user->sve))
> +				goto invalid;
> +
> +			user->sve = (struct sve_context __user *)head;
> +			break;
> +
>  		case EXTRA_MAGIC:
>  			if (have_extra_context)
>  				goto invalid;
> @@ -358,9 +475,6 @@ static int parse_user_sigframe(struct user_ctxs *user,
>  	}
>
>  done:
> -	if (!user->fpsimd)
> -		goto invalid;
> -
>  	return 0;
>
>  invalid:
> @@ -394,8 +508,18 @@ static int restore_sigframe(struct pt_regs *regs,
>  	if (err == 0)
>  		err = parse_user_sigframe(&user, sf);
>
> -	if (err == 0)
> -		err = restore_fpsimd_context(user.fpsimd);
> +	if (err == 0) {
> +		if (!user.fpsimd)
> +			return -EINVAL;
> +
> +		if (user.sve) {
> +			if (!system_supports_sve())
> +				return -EINVAL;
> +
> +			err = restore_sve_fpsimd_context(&user);
> +		} else
> +			err = restore_fpsimd_context(user.fpsimd);
> +	}
>
>  	return err;
>  }
> @@ -454,6 +578,20 @@ static int setup_sigframe_layout(struct rt_sigframe_user_layout *user)
>  			return err;
>  	}
>
> +	if (system_supports_sve()) {
> +		unsigned int vq = 0;
> +
> +		if (test_thread_flag(TIF_SVE)) {
> +			BUG_ON(!sve_vl_valid(current->thread.sve_vl));
> +			vq = sve_vq_from_vl(current->thread.sve_vl);
> +		}
> +
> +		err = sigframe_alloc(user, &user->sve_offset,
> +				     SVE_SIG_CONTEXT_SIZE(vq));
> +		if (err)
> +			return err;
> +	}
> +
>  	return sigframe_alloc_end(user);
>  }
>
> @@ -495,6 +633,13 @@ static int setup_sigframe(struct rt_sigframe_user_layout *user,
>  		__put_user_error(current->thread.fault_code, &esr_ctx->esr, err);
>  	}
>
> +	/* Scalable Vector Extension state, if present */
> +	if (system_supports_sve() && err == 0 && user->sve_offset) {
> +		struct sve_context __user *sve_ctx =
> +			apply_user_offset(user, user->sve_offset);
> +		err |= preserve_sve_context(sve_ctx);
> +	}
> +
>  	if (err == 0 && user->extra_offset) {
>  		char __user *sfp = (char __user *)user->sigframe;
>  		char __user *userp =
> @@ -594,6 +739,14 @@ static int setup_rt_frame(int usig, struct ksignal *ksig, sigset_t *set,
>  	struct rt_sigframe __user *frame;
>  	int err = 0;
>
> +	/*
> +	 * Ensure FPSIMD/SVE state in task_struct is up-to-date.
> +	 * This is needed here in order to complete any pending SVE discard:
> +	 * otherwise, discard may occur between deciding on the sigframe
> +	 * layout and dumping the register data.
> +	 */
> +	fpsimd_signal_preserve_current_state();
> +
>  	if (get_sigframe(&user, ksig, regs))
>  		return 1;
>
> diff --git a/arch/arm64/kernel/signal32.c b/arch/arm64/kernel/signal32.c
> index 4e5a664..202337d 100644
> --- a/arch/arm64/kernel/signal32.c
> +++ b/arch/arm64/kernel/signal32.c
> @@ -244,7 +244,7 @@ static int compat_preserve_vfp_context(struct compat_vfp_sigframe __user *frame)
>  	 * Note that this also saves V16-31, which aren't visible
>  	 * in AArch32.
>  	 */
> -	fpsimd_preserve_current_state();
> +	fpsimd_signal_preserve_current_state();
>
>  	/* Place structure header on the stack */
>  	__put_user_error(magic, &frame->magic, err);

Re: [PATCH 13/27] arm64/sve: Signal handling support

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> This patch implements support for saving and restoring the SVE
> registers around signals.
>
> A fixed-size header struct sve_context is always included in the
> signal frame encoding the thread's vector length at the time of
> signal delivery, optionally followed by a variable-layout structure
> encoding the SVE registers.
>
> Because of the need to preserve backwards compatibility, the FPSIMD
> view of the SVE registers is always dumped as a struct
> fpsimd_context in the usual way, in addition to any sve_context.
>
> The SVE vector registers are dumped in full, including bits 127:0
> of each register which alias the corresponding FPSIMD vector
> registers in the hardware.  To avoid any ambiguity about which
> alias to restore during sigreturn, the kernel always restores bits
> 127:0 of each SVE vector register from the fpsimd_context in the
> signal frame (which must be present): userspace needs to take this
> into account if it wants to modify the SVE vector register contents
> on return from a signal.
>
> FPSR and FPCR, which are used by both FPSIMD and SVE, are not
> included in sve_context because they are always present in
> fpsimd_context anyway.
>
> For signal delivery, a new helper
> fpsimd_signal_preserve_current_state() is added to update _both_
> the FPSIMD and SVE views in the task struct, to make it easier to
> populate this information into the signal frame.  Because of the
> redundancy between the two views of the state, only one is updated
> otherwise.  In order to avoid racing with a pending discard of the
> SVE state, this flush is hoisted before the sigframe layout phase,
> so that the layout and population phases see a consistent view of
> the thread.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/include/asm/fpsimd.h |   1 +
>  arch/arm64/kernel/fpsimd.c      |  23 ++++--
>  arch/arm64/kernel/signal.c      | 169 ++++++++++++++++++++++++++++++++++++++--
>  arch/arm64/kernel/signal32.c    |   2 +-
>  4 files changed, 179 insertions(+), 16 deletions(-)
>
> diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
> index 72090a1..7efd04e 100644
> --- a/arch/arm64/include/asm/fpsimd.h
> +++ b/arch/arm64/include/asm/fpsimd.h
> @@ -63,6 +63,7 @@ extern void fpsimd_load_state(struct fpsimd_state *state);
>  extern void fpsimd_thread_switch(struct task_struct *next);
>  extern void fpsimd_flush_thread(void);
>
> +extern void fpsimd_signal_preserve_current_state(void);
>  extern void fpsimd_preserve_current_state(void);
>  extern void fpsimd_restore_current_state(void);
>  extern void fpsimd_update_current_state(struct fpsimd_state *state);
> diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> index 80ecb2d..e8674f6 100644
> --- a/arch/arm64/kernel/fpsimd.c
> +++ b/arch/arm64/kernel/fpsimd.c
> @@ -148,8 +148,6 @@ static void change_cpacr(u64 old, u64 new)
>  		write_sysreg(new, CPACR_EL1);
>  }
>
> -#ifdef CONFIG_ARM64_SVE
> -
>  #define ZREG(sve_state, vq, n) ((char *)(sve_state) +		\
>  	(SVE_SIG_ZREG_OFFSET(vq, n) - SVE_SIG_REGS_OFFSET))
>
> @@ -191,6 +189,8 @@ static void fpsimd_to_sve(struct task_struct *task)
>  		       sizeof(fst->vregs[i]));
>  }
>
> +#ifdef CONFIG_ARM64_SVE
> +

Hmm have sve_to_fpsimd and fpsimd_to_sve only just started being used by
the generic code here?

>  size_t sve_state_size(struct task_struct const *task)
>  {
>  	unsigned int vl = task->thread.sve_vl;
> @@ -431,13 +431,17 @@ void fpsimd_preserve_current_state(void)
>  		return;
>
>  	local_bh_disable();
> -
> -	if (!test_thread_flag(TIF_FOREIGN_FPSTATE))
> -		fpsimd_save_state(&current->thread.fpsimd_state);
> -
> +	task_fpsimd_save();
>  	local_bh_enable();
>  }
>
> +void fpsimd_signal_preserve_current_state(void)
> +{
> +	fpsimd_preserve_current_state();
> +	if (system_supports_sve() && test_thread_flag(TIF_SVE))
> +		sve_to_fpsimd(current);
> +}
> +
>  /*
>   * Load the userland FPSIMD state of 'current' from memory, but only if the
>   * FPSIMD state already held in the registers is /not/ the most recent FPSIMD
> @@ -473,7 +477,12 @@ void fpsimd_update_current_state(struct fpsimd_state *state)
>
>  	local_bh_disable();
>
> -	fpsimd_load_state(state);
> +	if (system_supports_sve() && test_thread_flag(TIF_SVE)) {
> +		current->thread.fpsimd_state = *state;
> +		fpsimd_to_sve(current);
> +	}
> +	task_fpsimd_load();
> +
>  	if (test_and_clear_thread_flag(TIF_FOREIGN_FPSTATE)) {
>  		struct fpsimd_state *st = &current->thread.fpsimd_state;
>
> diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c
> index 4991e87..2694143 100644
> --- a/arch/arm64/kernel/signal.c
> +++ b/arch/arm64/kernel/signal.c
> @@ -62,6 +62,7 @@ struct rt_sigframe_user_layout {
>
>  	unsigned long fpsimd_offset;
>  	unsigned long esr_offset;
> +	unsigned long sve_offset;
>  	unsigned long extra_offset;
>  	unsigned long end_offset;
>  };
> @@ -178,9 +179,6 @@ static int preserve_fpsimd_context(struct fpsimd_context __user *ctx)
>  	struct fpsimd_state *fpsimd = &current->thread.fpsimd_state;
>  	int err;
>
> -	/* dump the hardware registers to the fpsimd_state structure */
> -	fpsimd_preserve_current_state();
> -
>  	/* copy the FP and status/control registers */
>  	err = __copy_to_user(ctx->vregs, fpsimd->vregs, sizeof(fpsimd->vregs));
>  	__put_user_error(fpsimd->fpsr, &ctx->fpsr, err);
> @@ -213,6 +211,8 @@ static int restore_fpsimd_context(struct fpsimd_context __user *ctx)
>  	__get_user_error(fpsimd.fpsr, &ctx->fpsr, err);
>  	__get_user_error(fpsimd.fpcr, &ctx->fpcr, err);
>
> +	clear_thread_flag(TIF_SVE);
> +
>  	/* load the hardware registers from the fpsimd_state structure */
>  	if (!err)
>  		fpsimd_update_current_state(&fpsimd);
> @@ -220,10 +220,113 @@ static int restore_fpsimd_context(struct fpsimd_context __user *ctx)
>  	return err ? -EFAULT : 0;
>  }
>
> +
>  struct user_ctxs {
>  	struct fpsimd_context __user *fpsimd;
> +	struct sve_context __user *sve;
>  };
>
> +#ifdef CONFIG_ARM64_SVE
> +
> +static int preserve_sve_context(struct sve_context __user *ctx)
> +{
> +	int err = 0;
> +	u16 reserved[ARRAY_SIZE(ctx->__reserved)];
> +	unsigned int vl = current->thread.sve_vl;
> +	unsigned int vq = 0;
> +
> +	BUG_ON(!sve_vl_valid(vl));
> +	if (test_thread_flag(TIF_SVE))
> +		vq = sve_vq_from_vl(vl);
> +
> +	memset(reserved, 0, sizeof(reserved));
> +
> +	__put_user_error(SVE_MAGIC, &ctx->head.magic, err);
> +	__put_user_error(round_up(SVE_SIG_CONTEXT_SIZE(vq), 16),
> +			 &ctx->head.size, err);
> +	__put_user_error(vl, &ctx->vl, err);
> +	BUILD_BUG_ON(sizeof(ctx->__reserved) != sizeof(reserved));
> +	err |= copy_to_user(&ctx->__reserved, reserved, sizeof(reserved));
> +
> +	if (vq) {
> +		/*
> +		 * This assumes that the SVE state has already been saved to
> +		 * the task struct by calling preserve_fpsimd_context().
> +		 */
> +		BUG_ON(SVE_SIG_REGS_SIZE(vq) !=
> sve_state_size(current));

I think others have mentioned the excessive BUG_ON()s here but I think
you are planning on cleaning some up on the next version. Assuming
sve_vq_from_vl() can't give you an invalid answer from a
sve_vl_valid(vl) then I wouldn't expect this test to add much.

> +		err |= copy_to_user((char __user *)ctx + SVE_SIG_REGS_OFFSET,
> +				    current->thread.sve_state,
> +				    SVE_SIG_REGS_SIZE(vq));
> +	}
> +
> +	return err ? -EFAULT : 0;
> +}
> +
> +static int restore_sve_fpsimd_context(struct user_ctxs *user)
> +{
> +	int err;
> +	unsigned int vq;
> +	struct fpsimd_state fpsimd;
> +	struct sve_context sve;
> +
> +	if (__copy_from_user(&sve, user->sve, sizeof(sve)))
> +		return -EFAULT;
> +
> +	if (sve.vl != current->thread.sve_vl)
> +		return -EINVAL;
> +
> +	if (sve.head.size <= sizeof(*user->sve)) {
> +		clear_thread_flag(TIF_SVE);
> +		goto fpsimd_only;
> +	}
> +
> +	BUG_ON(!sve_vl_valid(sve.vl));
> +	vq = sve_vq_from_vl(sve.vl);
> +
> +	if (sve.head.size < SVE_SIG_CONTEXT_SIZE(vq))
> +		return -EINVAL;
> +
> +	fpsimd_flush_task_state(current);
> +	barrier();
> +	set_thread_flag(TIF_FOREIGN_FPSTATE);
> +	barrier();

What are you trying to achieve with barriers here? Is there a potential
interaction between flushing the state and setting the flag that the
compiler can't see? A comment should be added at least.

> +
> +	sve_alloc(current);
> +	BUG_ON(SVE_SIG_REGS_SIZE(vq) != sve_state_size(current));
> +	err = __copy_from_user(current->thread.sve_state,
> +			       (char __user const *)user->sve +
> +					SVE_SIG_REGS_OFFSET,
> +			       SVE_SIG_REGS_SIZE(vq));
> +	if (err)
> +		return err;
> +
> +	barrier();
> +	set_thread_flag(TIF_SVE);

Hmm and again. If this is about visibility of context when the thread
flag is read by other CPUs a barrier() on it's own is not enough as it
only stop local code re-organisation - do you actually mean smp_mb()?

Either way you need to document the potential race in a comment so the
reason can be understood.

> +
> +fpsimd_only:
> +	/* copy the FP and status/control registers */
> +	/* restore_sigframe() already checked that user->fpsimd != NULL. */
> +	err = __copy_from_user(fpsimd.vregs, user->fpsimd->vregs,
> +			       sizeof(fpsimd.vregs));
> +	__get_user_error(fpsimd.fpsr, &user->fpsimd->fpsr, err);
> +	__get_user_error(fpsimd.fpcr, &user->fpsimd->fpcr, err);
> +
> +	/* load the hardware registers from the fpsimd_state structure */
> +	if (!err)
> +		fpsimd_update_current_state(&fpsimd);
> +
> +	return err;
> +}
> +
> +#else /* ! CONFIG_ARM64_SVE */
> +
> +/* Turn any non-optimised out attempts to use these into a link error: */
> +extern int preserve_sve_context(void __user *ctx);
> +extern int restore_sve_fpsimd_context(struct user_ctxs *user);
> +
> +#endif /* ! CONFIG_ARM64_SVE */
> +
> +
>  static int parse_user_sigframe(struct user_ctxs *user,
>  			       struct rt_sigframe __user *sf)
>  {
> @@ -236,6 +339,7 @@ static int parse_user_sigframe(struct user_ctxs *user,
>  	char const __user *const sfp = (char const __user *)sf;
>
>  	user->fpsimd = NULL;
> +	user->sve = NULL;
>
>  	if (!IS_ALIGNED((unsigned long)base, 16))
>  		goto invalid;
> @@ -286,6 +390,19 @@ static int parse_user_sigframe(struct user_ctxs *user,
>  			/* ignore */
>  			break;
>
> +		case SVE_MAGIC:
> +			if (!system_supports_sve())
> +				goto invalid;
> +
> +			if (user->sve)
> +				goto invalid;
> +
> +			if (size < sizeof(*user->sve))
> +				goto invalid;
> +
> +			user->sve = (struct sve_context __user *)head;
> +			break;
> +
>  		case EXTRA_MAGIC:
>  			if (have_extra_context)
>  				goto invalid;
> @@ -358,9 +475,6 @@ static int parse_user_sigframe(struct user_ctxs *user,
>  	}
>
>  done:
> -	if (!user->fpsimd)
> -		goto invalid;
> -
>  	return 0;
>
>  invalid:
> @@ -394,8 +508,18 @@ static int restore_sigframe(struct pt_regs *regs,
>  	if (err == 0)
>  		err = parse_user_sigframe(&user, sf);
>
> -	if (err == 0)
> -		err = restore_fpsimd_context(user.fpsimd);
> +	if (err == 0) {
> +		if (!user.fpsimd)
> +			return -EINVAL;
> +
> +		if (user.sve) {
> +			if (!system_supports_sve())
> +				return -EINVAL;
> +
> +			err = restore_sve_fpsimd_context(&user);
> +		} else
> +			err = restore_fpsimd_context(user.fpsimd);
> +	}
>
>  	return err;
>  }
> @@ -454,6 +578,20 @@ static int setup_sigframe_layout(struct rt_sigframe_user_layout *user)
>  			return err;
>  	}
>
> +	if (system_supports_sve()) {
> +		unsigned int vq = 0;
> +
> +		if (test_thread_flag(TIF_SVE)) {
> +			BUG_ON(!sve_vl_valid(current->thread.sve_vl));
> +			vq = sve_vq_from_vl(current->thread.sve_vl);
> +		}
> +
> +		err = sigframe_alloc(user, &user->sve_offset,
> +				     SVE_SIG_CONTEXT_SIZE(vq));
> +		if (err)
> +			return err;
> +	}
> +
>  	return sigframe_alloc_end(user);
>  }
>
> @@ -495,6 +633,13 @@ static int setup_sigframe(struct rt_sigframe_user_layout *user,
>  		__put_user_error(current->thread.fault_code, &esr_ctx->esr, err);
>  	}
>
> +	/* Scalable Vector Extension state, if present */
> +	if (system_supports_sve() && err == 0 && user->sve_offset) {
> +		struct sve_context __user *sve_ctx =
> +			apply_user_offset(user, user->sve_offset);
> +		err |= preserve_sve_context(sve_ctx);
> +	}
> +
>  	if (err == 0 && user->extra_offset) {
>  		char __user *sfp = (char __user *)user->sigframe;
>  		char __user *userp =
> @@ -594,6 +739,14 @@ static int setup_rt_frame(int usig, struct ksignal *ksig, sigset_t *set,
>  	struct rt_sigframe __user *frame;
>  	int err = 0;
>
> +	/*
> +	 * Ensure FPSIMD/SVE state in task_struct is up-to-date.
> +	 * This is needed here in order to complete any pending SVE discard:
> +	 * otherwise, discard may occur between deciding on the sigframe
> +	 * layout and dumping the register data.
> +	 */
> +	fpsimd_signal_preserve_current_state();
> +
>  	if (get_sigframe(&user, ksig, regs))
>  		return 1;
>
> diff --git a/arch/arm64/kernel/signal32.c b/arch/arm64/kernel/signal32.c
> index 4e5a664..202337d 100644
> --- a/arch/arm64/kernel/signal32.c
> +++ b/arch/arm64/kernel/signal32.c
> @@ -244,7 +244,7 @@ static int compat_preserve_vfp_context(struct compat_vfp_sigframe __user *frame)
>  	 * Note that this also saves V16-31, which aren't visible
>  	 * in AArch32.
>  	 */
> -	fpsimd_preserve_current_state();
> +	fpsimd_signal_preserve_current_state();
>
>  	/* Place structure header on the stack */
>  	__put_user_error(magic, &frame->magic, err);


--
Alex Bennée

Re: [PATCH 13/27] arm64/sve: Signal handling support

[Dave Martin]

On Wed, Aug 23, 2017 at 10:38:51AM +0100, Alex Bennée wrote:
> 
> Dave Martin <Dave.Martin@arm.com> writes:
> 
> > This patch implements support for saving and restoring the SVE
> > registers around signals.
> >
> > A fixed-size header struct sve_context is always included in the
> > signal frame encoding the thread's vector length at the time of
> > signal delivery, optionally followed by a variable-layout structure
> > encoding the SVE registers.
> >
> > Because of the need to preserve backwards compatibility, the FPSIMD
> > view of the SVE registers is always dumped as a struct
> > fpsimd_context in the usual way, in addition to any sve_context.
> >
> > The SVE vector registers are dumped in full, including bits 127:0
> > of each register which alias the corresponding FPSIMD vector
> > registers in the hardware.  To avoid any ambiguity about which
> > alias to restore during sigreturn, the kernel always restores bits
> > 127:0 of each SVE vector register from the fpsimd_context in the
> > signal frame (which must be present): userspace needs to take this
> > into account if it wants to modify the SVE vector register contents
> > on return from a signal.
> >
> > FPSR and FPCR, which are used by both FPSIMD and SVE, are not
> > included in sve_context because they are always present in
> > fpsimd_context anyway.
> >
> > For signal delivery, a new helper
> > fpsimd_signal_preserve_current_state() is added to update _both_
> > the FPSIMD and SVE views in the task struct, to make it easier to
> > populate this information into the signal frame.  Because of the
> > redundancy between the two views of the state, only one is updated
> > otherwise.  In order to avoid racing with a pending discard of the
> > SVE state, this flush is hoisted before the sigframe layout phase,
> > so that the layout and population phases see a consistent view of
> > the thread.
> >
> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> > ---

[...]

> > diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> > index 80ecb2d..e8674f6 100644
> > --- a/arch/arm64/kernel/fpsimd.c
> > +++ b/arch/arm64/kernel/fpsimd.c
> > @@ -148,8 +148,6 @@ static void change_cpacr(u64 old, u64 new)
> >  		write_sysreg(new, CPACR_EL1);
> >  }
> >
> > -#ifdef CONFIG_ARM64_SVE
> > -
> >  #define ZREG(sve_state, vq, n) ((char *)(sve_state) +		\
> >  	(SVE_SIG_ZREG_OFFSET(vq, n) - SVE_SIG_REGS_OFFSET))
> >
> > @@ -191,6 +189,8 @@ static void fpsimd_to_sve(struct task_struct *task)
> >  		       sizeof(fst->vregs[i]));
> >  }
> >
> > +#ifdef CONFIG_ARM64_SVE
> > +
> 
> Hmm have sve_to_fpsimd and fpsimd_to_sve only just started being used by
> the generic code here?

Yes, the signal code now makes use of these via
fpsimd_signal_preserve_current_state() and
fpsimd_update_current_state(), with this patch.

[...]

> > diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c
> > index 4991e87..2694143 100644
> > --- a/arch/arm64/kernel/signal.c
> > +++ b/arch/arm64/kernel/signal.c

[...]

> > +#ifdef CONFIG_ARM64_SVE
> > +
> > +static int preserve_sve_context(struct sve_context __user *ctx)
> > +{
> > +	int err = 0;
> > +	u16 reserved[ARRAY_SIZE(ctx->__reserved)];
> > +	unsigned int vl = current->thread.sve_vl;
> > +	unsigned int vq = 0;
> > +
> > +	BUG_ON(!sve_vl_valid(vl));
> > +	if (test_thread_flag(TIF_SVE))
> > +		vq = sve_vq_from_vl(vl);
> > +
> > +	memset(reserved, 0, sizeof(reserved));
> > +
> > +	__put_user_error(SVE_MAGIC, &ctx->head.magic, err);
> > +	__put_user_error(round_up(SVE_SIG_CONTEXT_SIZE(vq), 16),
> > +			 &ctx->head.size, err);
> > +	__put_user_error(vl, &ctx->vl, err);
> > +	BUILD_BUG_ON(sizeof(ctx->__reserved) != sizeof(reserved));
> > +	err |= copy_to_user(&ctx->__reserved, reserved, sizeof(reserved));
> > +
> > +	if (vq) {
> > +		/*
> > +		 * This assumes that the SVE state has already been saved to
> > +		 * the task struct by calling preserve_fpsimd_context().
> > +		 */
> > +		BUG_ON(SVE_SIG_REGS_SIZE(vq) !=
> > sve_state_size(current));
> 
> I think others have mentioned the excessive BUG_ON()s here but I think
> you are planning on cleaning some up on the next version. Assuming
> sve_vq_from_vl() can't give you an invalid answer from a
> sve_vl_valid(vl) then I wouldn't expect this test to add much.

Agreed.  Some BUG_ON()s test for things that are now obviously true
by construction, which was not initially the case during development.
This one can go away.

I'm very interested in any BUG_ON() that you think can actually fire.
There _shouldn't_ be any of those, but the reasons are non-trivial in
some cases.

> > +		err |= copy_to_user((char __user *)ctx + SVE_SIG_REGS_OFFSET,
> > +				    current->thread.sve_state,
> > +				    SVE_SIG_REGS_SIZE(vq));
> > +	}
> > +
> > +	return err ? -EFAULT : 0;
> > +}
> > +
> > +static int restore_sve_fpsimd_context(struct user_ctxs *user)
> > +{
> > +	int err;
> > +	unsigned int vq;
> > +	struct fpsimd_state fpsimd;
> > +	struct sve_context sve;
> > +
> > +	if (__copy_from_user(&sve, user->sve, sizeof(sve)))
> > +		return -EFAULT;
> > +
> > +	if (sve.vl != current->thread.sve_vl)
> > +		return -EINVAL;
> > +
> > +	if (sve.head.size <= sizeof(*user->sve)) {
> > +		clear_thread_flag(TIF_SVE);
> > +		goto fpsimd_only;
> > +	}
> > +
> > +	BUG_ON(!sve_vl_valid(sve.vl));
> > +	vq = sve_vq_from_vl(sve.vl);
> > +
> > +	if (sve.head.size < SVE_SIG_CONTEXT_SIZE(vq))
> > +		return -EINVAL;
> > +
> > +	fpsimd_flush_task_state(current);
> > +	barrier();
> > +	set_thread_flag(TIF_FOREIGN_FPSTATE);
> > +	barrier();
> 
> What are you trying to achieve with barriers here? Is there a potential
> interaction between flushing the state and setting the flag that the
> compiler can't see? A comment should be added at least.

This is a tricky one -- I suppose I should definitely add comments,
since it took me 2-3 weeks to debug this and get it right the first time
around...

I'm trying to protect against a context switch during the copy_from_user
rewriting stale register data over the data we're copying from the user
signal frame.

In most places, I now disable preemption when writing into thread_struct,
to keep this can of worms firmly closed.

Here though, the code writes directly into thread_struct with
preemption enabled: in this one case the copy is a copy_from_user(),
and we can't disable preemption for that.  Other than this, the reasons
for this code being the way it is derive from Ard's deferred fpsimd
reload machinery (the TIF_FOREIGN_FPSTATE stuff), which is described in
more detail in fpsimd.c.


The reason for the difference from the old fpsimd code here is that the
SVE state may be unreasonably large to bounce via a buffer on the stack,
and the copy-in occurs while the thread is running.  In the ptrace case
(later on) the thread is guaranteed stopped, whereas for fpsimd the
state is small enough that we _can_ bounce it via the stack.

TIF_FOREIGN_FPSTATE may be cleared by the context switch code, but only
if fpsimd_flush_task_state() wasn't called since the last kernel entry
for current.  (See fpsimd_thread_switch().)

The context switch code may write the CPU registers over thread_struct,
but only if TIF_FOREIGN_FPSTATE is not set at sched-out time.
(fpsimd_thread_switch() again).

These dependencies are strictly intra-thread, since context-switch is
part of the same thread, so

	fpsimd_flush_task_state();
	barrier();
	set_thread_flag(TIF_FOREIGN_FPSTATE);
	barrier();
	copy_from_user(current->thread.sve_state, ...);

should be sufficient.

I prefer to avoid spuriously saving the SVE regs into thread_struct
when they may not already be live, so I don't set TIF_SVE until after
TIF_FOREIGN_FPSTATE is set.  (This is a cost saving rather than a
security issue -- only current's state can be in the registers, but the
extra SVE bits are definitely not worth saving.  In any case, it only
makes a difference to context switches in this narrow window.)

> > +
> > +	sve_alloc(current);
> > +	BUG_ON(SVE_SIG_REGS_SIZE(vq) != sve_state_size(current));
> > +	err = __copy_from_user(current->thread.sve_state,
> > +			       (char __user const *)user->sve +
> > +					SVE_SIG_REGS_OFFSET,
> > +			       SVE_SIG_REGS_SIZE(vq));
> > +	if (err)
> > +		return err;
> > +
> > +	barrier();
> > +	set_thread_flag(TIF_SVE);
> 
> Hmm and again. If this is about visibility of context when the thread
> flag is read by other CPUs a barrier() on it's own is not enough as it
> only stop local code re-organisation - do you actually mean smp_mb()?
> 
> Either way you need to document the potential race in a comment so the
> reason can be understood.

The final barrier() isn't really needed, and the ordering of TIF_SVE
with the copy_from_user is unimportant unless I've missed something ...
so I may get rid of that.  TIF_SVE would affect context switch, but
only if TIF_FOREIGN_FPSTATE is clear or a ret_to_user occurs, neither
of which is possible after the second barrier().


Re smp_mb(), the only cross-thread ordering guarantees needed are for
ptrace (later in the series), since that runs in some thread operating
on a stopped thread that may have last run on some other CPU.  But this
is a general ptrace issue, not specifically related to SVE.  IIRC some
interaction between the ptrace core code and the scheduler's
manipulation of the task state and runqueue locks ensures safety in
that case without need for extra smp_mb() all over the place.

> > +
> > +fpsimd_only:
> > +	/* copy the FP and status/control registers */
> > +	/* restore_sigframe() already checked that user->fpsimd != NULL. */
> > +	err = __copy_from_user(fpsimd.vregs, user->fpsimd->vregs,
> > +			       sizeof(fpsimd.vregs));
> > +	__get_user_error(fpsimd.fpsr, &user->fpsimd->fpsr, err);
> > +	__get_user_error(fpsimd.fpcr, &user->fpsimd->fpcr, err);
> > +
> > +	/* load the hardware registers from the fpsimd_state structure */
> > +	if (!err)
> > +		fpsimd_update_current_state(&fpsimd);
> > +
> > +	return err;
> > +}

[...]

Cheers
---Dave

Re: [PATCH 14/27] arm64/sve: Backend logic for setting the vector length

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> This patch implements the core logic for changing a task's vector
> length on request from userspace.  This will be used by the ptrace
> and prctl frontends that are implemented in later patches.
>
> The SVE architecture permits, but does not require, implementations
> to support vector lengths that are not a power of two.  To handle
> this, logic is added to check a requested vector length against a
> possibly sparse bitmap of available vector lengths at runtime, so
> that the best supported value can be chosen.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/include/asm/fpsimd.h |   6 +++
>  arch/arm64/kernel/fpsimd.c      | 116 ++++++++++++++++++++++++++++++++++++++++
>  include/uapi/linux/prctl.h      |   5 ++
>  3 files changed, 127 insertions(+)
>
> diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
> index 7efd04e..39b26d2 100644
> --- a/arch/arm64/include/asm/fpsimd.h
> +++ b/arch/arm64/include/asm/fpsimd.h
> @@ -70,11 +70,15 @@ extern void fpsimd_update_current_state(struct fpsimd_state *state);
>
>  extern void fpsimd_flush_task_state(struct task_struct *target);
>
> +#define SVE_VL_ARCH_MAX 0x100
> +

Hmm this isn't the same as SVE_VL_MAX. Why aren't we using that?

>  extern void sve_save_state(void *state, u32 *pfpsr);
>  extern void sve_load_state(void const *state, u32 const *pfpsr,
>  			   unsigned long vq_minus_1);
>  extern unsigned int sve_get_vl(void);
>
> +extern int sve_max_vl;
> +
>  #ifdef CONFIG_ARM64_SVE
>
>  extern size_t sve_state_size(struct task_struct const *task);
> @@ -83,6 +87,8 @@ extern void sve_alloc(struct task_struct *task);
>  extern void fpsimd_release_thread(struct task_struct *task);
>  extern void fpsimd_dup_sve(struct task_struct *dst,
>  			   struct task_struct const *src);
> +extern int sve_set_vector_length(struct task_struct *task,
> +				 unsigned long vl, unsigned long flags);
>
>  #else /* ! CONFIG_ARM64_SVE */
>
> diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> index e8674f6..bce95de 100644
> --- a/arch/arm64/kernel/fpsimd.c
> +++ b/arch/arm64/kernel/fpsimd.c
> @@ -18,12 +18,14 @@
>   */
>
>  #include <linux/bottom_half.h>
> +#include <linux/bitmap.h>
>  #include <linux/cpu.h>
>  #include <linux/cpu_pm.h>
>  #include <linux/kernel.h>
>  #include <linux/init.h>
>  #include <linux/percpu.h>
>  #include <linux/preempt.h>
> +#include <linux/prctl.h>
>  #include <linux/ptrace.h>
>  #include <linux/sched/signal.h>
>  #include <linux/signal.h>
> @@ -111,6 +113,20 @@ static DEFINE_PER_CPU(struct fpsimd_state *, fpsimd_last_state);
>  /* Default VL for tasks that don't set it explicitly: */
>  static int sve_default_vl = -1;
>
> +#ifdef CONFIG_ARM64_SVE
> +
> +/* Maximum supported vector length across all CPUs (initially poisoned) */
> +int sve_max_vl = -1;
> +/* Set of available vector lengths, as vq_to_bit(vq): */
> +static DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX);
> +
> +#else /* ! CONFIG_ARM64_SVE */
> +
> +/* Dummy declaration for code that will be optimised out: */
> +extern DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX);
> +
> +#endif /* ! CONFIG_ARM64_SVE */
> +
>  static void sve_free(struct task_struct *task)
>  {
>  	kfree(task->thread.sve_state);
> @@ -148,6 +164,37 @@ static void change_cpacr(u64 old, u64 new)
>  		write_sysreg(new, CPACR_EL1);
>  }
>
> +static unsigned int vq_to_bit(unsigned int vq)
> +{
> +	BUILD_BUG_ON(vq < 1 || vq > SVE_VQ_MAX);
> +
> +	return SVE_VQ_MAX - vq;
> +}
> +
> +static unsigned int bit_to_vq(unsigned int bit)
> +{
> +	BUILD_BUG_ON(bit >= SVE_VQ_MAX);
> +
> +	return SVE_VQ_MAX - bit;
> +}
> +
> +static unsigned int find_supported_vector_length(unsigned int vl)
> +{
> +	int bit;
> +
> +	BUG_ON(!sve_vl_valid(vl));
> +
> +	BUG_ON(!sve_vl_valid(sve_max_vl));
> +	if (vl > sve_max_vl)
> +		vl = sve_max_vl;
> +
> +	bit = find_next_bit(sve_vq_map, SVE_VQ_MAX,
> +			    vq_to_bit(sve_vq_from_vl(vl)));
> +	BUG_ON(bit < 0 || bit >= SVE_VQ_MAX);
> +
> +	return 16 * bit_to_vq(bit);
> +}
> +
>  #define ZREG(sve_state, vq, n) ((char *)(sve_state) +		\
>  	(SVE_SIG_ZREG_OFFSET(vq, n) - SVE_SIG_REGS_OFFSET))
>
> @@ -235,6 +282,73 @@ void fpsimd_dup_sve(struct task_struct *dst, struct task_struct const *src)
>  	}
>  }
>
> +int sve_set_vector_length(struct task_struct *task,
> +			  unsigned long vl, unsigned long flags)
> +{
> +	BUG_ON(task == current && preemptible());
> +
> +	if (flags & ~(unsigned long)(PR_SVE_VL_INHERIT |
> +				     PR_SVE_SET_VL_ONEXEC))
> +		return -EINVAL;
> +
> +	if (!sve_vl_valid(vl))
> +		return -EINVAL;
> +
> +	/*
> +	 * Clamp to the maximum vector length that VL-agnostic SVE code can
> +	 * work with.  A flag may be assigned in the future to allow setting
> +	 * of larger vector lengths without confusing older software.
> +	 */
> +	if (vl > SVE_VL_ARCH_MAX)
> +		vl = SVE_VL_ARCH_MAX;
> +
> +	vl = find_supported_vector_length(vl);
> +
> +	if (flags & (PR_SVE_VL_INHERIT |
> +		     PR_SVE_SET_VL_ONEXEC))
> +		task->thread.sve_vl_onexec = vl;
> +	else
> +		/* Reset VL to system default on next exec: */
> +		task->thread.sve_vl_onexec = 0;
> +
> +	/* Only actually set the VL if not deferred: */
> +	if (flags & PR_SVE_SET_VL_ONEXEC)
> +		goto out;
> +
> +	/*
> +	 * To ensure the FPSIMD bits of the SVE vector registers are preserved,
> +	 * write any live register state back to task_struct, and convert to a
> +	 * non-SVE thread.
> +	 */
> +	if (vl != task->thread.sve_vl) {
> +		if (task == current) {
> +			task_fpsimd_save();
> +			set_thread_flag(TIF_FOREIGN_FPSTATE);
> +		}
> +
> +		if (test_and_clear_tsk_thread_flag(task, TIF_SVE))
> +			sve_to_fpsimd(task);
> +
> +		/*
> +		 * Force reallocation of task SVE state to the correct size
> +		 * on next use:
> +		 */
> +		sve_free(task);
> +	}
> +
> +	task->thread.sve_vl = vl;
> +
> +	fpsimd_flush_task_state(task);
> +
> +out:
> +	if (flags & PR_SVE_VL_INHERIT)
> +		set_thread_flag(TIF_SVE_VL_INHERIT);
> +	else
> +		clear_thread_flag(TIF_SVE_VL_INHERIT);
> +
> +	return 0;
> +}
> +
>  void fpsimd_release_thread(struct task_struct *dead_task)
>  {
>  	sve_free(dead_task);
> @@ -407,6 +521,8 @@ void fpsimd_flush_thread(void)
>  		 * If not, something went badly wrong.
>  		 */
>  		BUG_ON(!sve_vl_valid(current->thread.sve_vl));
> +		BUG_ON(find_supported_vector_length(current->thread.sve_vl) !=
> +		       current->thread.sve_vl);
>
>  		/*
>  		 * If the task is not set to inherit, ensure that the vector
> diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h
> index a8d0759..1b64901 100644
> --- a/include/uapi/linux/prctl.h
> +++ b/include/uapi/linux/prctl.h
> @@ -197,4 +197,9 @@ struct prctl_mm_map {
>  # define PR_CAP_AMBIENT_LOWER		3
>  # define PR_CAP_AMBIENT_CLEAR_ALL	4
>
> +/* arm64 Scalable Vector Extension controls */
> +# define PR_SVE_SET_VL_ONEXEC		(1 << 18) /* defer effect until exec */
> +# define PR_SVE_VL_LEN_MASK		0xffff
> +# define PR_SVE_VL_INHERIT		(1 << 17) /* inherit across exec */
> +
>  #endif /* _LINUX_PRCTL_H */


--
Alex Bennée
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [PATCH 14/27] arm64/sve: Backend logic for setting the vector length

[Alex Bennée]

Dave Martin <Dave.Martin@arm.com> writes:

> This patch implements the core logic for changing a task's vector
> length on request from userspace.  This will be used by the ptrace
> and prctl frontends that are implemented in later patches.
>
> The SVE architecture permits, but does not require, implementations
> to support vector lengths that are not a power of two.  To handle
> this, logic is added to check a requested vector length against a
> possibly sparse bitmap of available vector lengths at runtime, so
> that the best supported value can be chosen.
>
> Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> ---
>  arch/arm64/include/asm/fpsimd.h |   6 +++
>  arch/arm64/kernel/fpsimd.c      | 116 ++++++++++++++++++++++++++++++++++++++++
>  include/uapi/linux/prctl.h      |   5 ++
>  3 files changed, 127 insertions(+)
>
> diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
> index 7efd04e..39b26d2 100644
> --- a/arch/arm64/include/asm/fpsimd.h
> +++ b/arch/arm64/include/asm/fpsimd.h
> @@ -70,11 +70,15 @@ extern void fpsimd_update_current_state(struct fpsimd_state *state);
>
>  extern void fpsimd_flush_task_state(struct task_struct *target);
>
> +#define SVE_VL_ARCH_MAX 0x100
> +

Hmm this isn't the same as SVE_VL_MAX. Why aren't we using that?

>  extern void sve_save_state(void *state, u32 *pfpsr);
>  extern void sve_load_state(void const *state, u32 const *pfpsr,
>  			   unsigned long vq_minus_1);
>  extern unsigned int sve_get_vl(void);
>
> +extern int sve_max_vl;
> +
>  #ifdef CONFIG_ARM64_SVE
>
>  extern size_t sve_state_size(struct task_struct const *task);
> @@ -83,6 +87,8 @@ extern void sve_alloc(struct task_struct *task);
>  extern void fpsimd_release_thread(struct task_struct *task);
>  extern void fpsimd_dup_sve(struct task_struct *dst,
>  			   struct task_struct const *src);
> +extern int sve_set_vector_length(struct task_struct *task,
> +				 unsigned long vl, unsigned long flags);
>
>  #else /* ! CONFIG_ARM64_SVE */
>
> diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
> index e8674f6..bce95de 100644
> --- a/arch/arm64/kernel/fpsimd.c
> +++ b/arch/arm64/kernel/fpsimd.c
> @@ -18,12 +18,14 @@
>   */
>
>  #include <linux/bottom_half.h>
> +#include <linux/bitmap.h>
>  #include <linux/cpu.h>
>  #include <linux/cpu_pm.h>
>  #include <linux/kernel.h>
>  #include <linux/init.h>
>  #include <linux/percpu.h>
>  #include <linux/preempt.h>
> +#include <linux/prctl.h>
>  #include <linux/ptrace.h>
>  #include <linux/sched/signal.h>
>  #include <linux/signal.h>
> @@ -111,6 +113,20 @@ static DEFINE_PER_CPU(struct fpsimd_state *, fpsimd_last_state);
>  /* Default VL for tasks that don't set it explicitly: */
>  static int sve_default_vl = -1;
>
> +#ifdef CONFIG_ARM64_SVE
> +
> +/* Maximum supported vector length across all CPUs (initially poisoned) */
> +int sve_max_vl = -1;
> +/* Set of available vector lengths, as vq_to_bit(vq): */
> +static DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX);
> +
> +#else /* ! CONFIG_ARM64_SVE */
> +
> +/* Dummy declaration for code that will be optimised out: */
> +extern DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX);
> +
> +#endif /* ! CONFIG_ARM64_SVE */
> +
>  static void sve_free(struct task_struct *task)
>  {
>  	kfree(task->thread.sve_state);
> @@ -148,6 +164,37 @@ static void change_cpacr(u64 old, u64 new)
>  		write_sysreg(new, CPACR_EL1);
>  }
>
> +static unsigned int vq_to_bit(unsigned int vq)
> +{
> +	BUILD_BUG_ON(vq < 1 || vq > SVE_VQ_MAX);
> +
> +	return SVE_VQ_MAX - vq;
> +}
> +
> +static unsigned int bit_to_vq(unsigned int bit)
> +{
> +	BUILD_BUG_ON(bit >= SVE_VQ_MAX);
> +
> +	return SVE_VQ_MAX - bit;
> +}
> +
> +static unsigned int find_supported_vector_length(unsigned int vl)
> +{
> +	int bit;
> +
> +	BUG_ON(!sve_vl_valid(vl));
> +
> +	BUG_ON(!sve_vl_valid(sve_max_vl));
> +	if (vl > sve_max_vl)
> +		vl = sve_max_vl;
> +
> +	bit = find_next_bit(sve_vq_map, SVE_VQ_MAX,
> +			    vq_to_bit(sve_vq_from_vl(vl)));
> +	BUG_ON(bit < 0 || bit >= SVE_VQ_MAX);
> +
> +	return 16 * bit_to_vq(bit);
> +}
> +
>  #define ZREG(sve_state, vq, n) ((char *)(sve_state) +		\
>  	(SVE_SIG_ZREG_OFFSET(vq, n) - SVE_SIG_REGS_OFFSET))
>
> @@ -235,6 +282,73 @@ void fpsimd_dup_sve(struct task_struct *dst, struct task_struct const *src)
>  	}
>  }
>
> +int sve_set_vector_length(struct task_struct *task,
> +			  unsigned long vl, unsigned long flags)
> +{
> +	BUG_ON(task == current && preemptible());
> +
> +	if (flags & ~(unsigned long)(PR_SVE_VL_INHERIT |
> +				     PR_SVE_SET_VL_ONEXEC))
> +		return -EINVAL;
> +
> +	if (!sve_vl_valid(vl))
> +		return -EINVAL;
> +
> +	/*
> +	 * Clamp to the maximum vector length that VL-agnostic SVE code can
> +	 * work with.  A flag may be assigned in the future to allow setting
> +	 * of larger vector lengths without confusing older software.
> +	 */
> +	if (vl > SVE_VL_ARCH_MAX)
> +		vl = SVE_VL_ARCH_MAX;
> +
> +	vl = find_supported_vector_length(vl);
> +
> +	if (flags & (PR_SVE_VL_INHERIT |
> +		     PR_SVE_SET_VL_ONEXEC))
> +		task->thread.sve_vl_onexec = vl;
> +	else
> +		/* Reset VL to system default on next exec: */
> +		task->thread.sve_vl_onexec = 0;
> +
> +	/* Only actually set the VL if not deferred: */
> +	if (flags & PR_SVE_SET_VL_ONEXEC)
> +		goto out;
> +
> +	/*
> +	 * To ensure the FPSIMD bits of the SVE vector registers are preserved,
> +	 * write any live register state back to task_struct, and convert to a
> +	 * non-SVE thread.
> +	 */
> +	if (vl != task->thread.sve_vl) {
> +		if (task == current) {
> +			task_fpsimd_save();
> +			set_thread_flag(TIF_FOREIGN_FPSTATE);
> +		}
> +
> +		if (test_and_clear_tsk_thread_flag(task, TIF_SVE))
> +			sve_to_fpsimd(task);
> +
> +		/*
> +		 * Force reallocation of task SVE state to the correct size
> +		 * on next use:
> +		 */
> +		sve_free(task);
> +	}
> +
> +	task->thread.sve_vl = vl;
> +
> +	fpsimd_flush_task_state(task);
> +
> +out:
> +	if (flags & PR_SVE_VL_INHERIT)
> +		set_thread_flag(TIF_SVE_VL_INHERIT);
> +	else
> +		clear_thread_flag(TIF_SVE_VL_INHERIT);
> +
> +	return 0;
> +}
> +
>  void fpsimd_release_thread(struct task_struct *dead_task)
>  {
>  	sve_free(dead_task);
> @@ -407,6 +521,8 @@ void fpsimd_flush_thread(void)
>  		 * If not, something went badly wrong.
>  		 */
>  		BUG_ON(!sve_vl_valid(current->thread.sve_vl));
> +		BUG_ON(find_supported_vector_length(current->thread.sve_vl) !=
> +		       current->thread.sve_vl);
>
>  		/*
>  		 * If the task is not set to inherit, ensure that the vector
> diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h
> index a8d0759..1b64901 100644
> --- a/include/uapi/linux/prctl.h
> +++ b/include/uapi/linux/prctl.h
> @@ -197,4 +197,9 @@ struct prctl_mm_map {
>  # define PR_CAP_AMBIENT_LOWER		3
>  # define PR_CAP_AMBIENT_CLEAR_ALL	4
>
> +/* arm64 Scalable Vector Extension controls */
> +# define PR_SVE_SET_VL_ONEXEC		(1 << 18) /* defer effect until exec */
> +# define PR_SVE_VL_LEN_MASK		0xffff
> +# define PR_SVE_VL_INHERIT		(1 << 17) /* inherit across exec */
> +
>  #endif /* _LINUX_PRCTL_H */


--
Alex Bennée

Re: [PATCH 14/27] arm64/sve: Backend logic for setting the vector length

[Dave Martin]

On Wed, Aug 23, 2017 at 04:33:18PM +0100, Alex Bennée wrote:
> 
> Dave Martin <Dave.Martin@arm.com> writes:
> 
> > This patch implements the core logic for changing a task's vector
> > length on request from userspace.  This will be used by the ptrace
> > and prctl frontends that are implemented in later patches.
> >
> > The SVE architecture permits, but does not require, implementations
> > to support vector lengths that are not a power of two.  To handle
> > this, logic is added to check a requested vector length against a
> > possibly sparse bitmap of available vector lengths at runtime, so
> > that the best supported value can be chosen.
> >
> > Signed-off-by: Dave Martin <Dave.Martin@arm.com>
> > ---
> >  arch/arm64/include/asm/fpsimd.h |   6 +++
> >  arch/arm64/kernel/fpsimd.c      | 116 ++++++++++++++++++++++++++++++++++++++++
> >  include/uapi/linux/prctl.h      |   5 ++
> >  3 files changed, 127 insertions(+)
> >
> > diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h
> > index 7efd04e..39b26d2 100644
> > --- a/arch/arm64/include/asm/fpsimd.h
> > +++ b/arch/arm64/include/asm/fpsimd.h
> > @@ -70,11 +70,15 @@ extern void fpsimd_update_current_state(struct fpsimd_state *state);
> >
> >  extern void fpsimd_flush_task_state(struct task_struct *target);
> >
> > +#define SVE_VL_ARCH_MAX 0x100
> > +
> 
> Hmm this isn't the same as SVE_VL_MAX. Why aren't we using that?

There should probably be a comment on this.

SVE vector-length agnostic software needs to be able to assume that
VL <= 256 (the SVE maximum), because some instructions won't work as
expected if this is not the case, particularly TBL, where in a larger
vector there's no guarantee that a vector index (which might be as small
as a byte) can index a vector element (of which there could be >256).

I really don't want to expose SVE_VL_ARCH_MAX to userspace, so that
people are not tempted to design data structures and protocols that
can't handle up to SVE_VL_MAX.  Exposing both together seemed to carry
a high risk of confusion and misuse.

Instead, I silently clamp to an SVE-compatible length (i.e.,
SVE_VL_ARCH_MAX) in sve_set_vector_length().  If future architecture
revisions support larger vectors later on, my plan is to require an
extra "force flag" to be passed to override the clamp.

[...]

Cheers
---Dave