[PATCH 0/6] s390: improve speculative execution handling v3

[PATCH 0/6] s390: improve speculative execution handling v3

[Martin Schwidefsky]

Version 3 of the speculative execution improvements for s390.

Changes to v2:

* Dropped the prctl to introduce the PR_ISOLATE_BP control and simply
  added two exported functions s390_isolate_bp and s390_isolate_bp_guest.
  There is currently no caller for these functions, for now an out-of-tree
  module can be used until an acceptable upstream solution for the user
  space interface is found.

* Added an optimized version for the the array_index_mask_nospec
  function based on subtract with borrow for the spectre v1 defense.

* Introduce "expoline", the s390 version of a retpoline. As s390 does
  not have a return instruction and the associate return stack we use
  an execute-type instruction on an indirect branch to get unpredicatable
  branches. This requires gcc support for -mindirect-branch=thunk /
  -mfunction-return=thunk.  To be able to disable expolines there is
  another gcc option -mindirect-branch-table to keep a list of PC relative
  locations of calls to the execute thunks. With spectre_v2=off the call
  will be replaced with the original indirect branch and a nop.

Martin Schwidefsky (6):
  s390: scrub registers on kernel entry and KVM exit
  s390: add optimized array_index_mask_nospec
  s390/alternative: use a copy of the facility bit mask
  s390: add options to change branch prediction behaviour for the kernel
  s390: run user space and KVM guests with modified branch prediction
  s390: introduce execute-trampolines for branches

 arch/s390/Kconfig                     |  45 ++++++
 arch/s390/Makefile                    |  12 ++
 arch/s390/include/asm/barrier.h       |  24 ++++
 arch/s390/include/asm/facility.h      |  18 +++
 arch/s390/include/asm/lowcore.h       |   9 +-
 arch/s390/include/asm/nospec-branch.h |  18 +++
 arch/s390/include/asm/processor.h     |   4 +
 arch/s390/include/asm/thread_info.h   |   4 +
 arch/s390/kernel/Makefile             |   4 +
 arch/s390/kernel/alternative.c        |  26 +++-
 arch/s390/kernel/early.c              |   5 +
 arch/s390/kernel/entry.S              | 249 ++++++++++++++++++++++++++++++----
 arch/s390/kernel/ipl.c                |   1 +
 arch/s390/kernel/module.c             |  62 +++++++--
 arch/s390/kernel/nospec-branch.c      | 100 ++++++++++++++
 arch/s390/kernel/processor.c          |  18 +++
 arch/s390/kernel/setup.c              |   8 +-
 arch/s390/kernel/smp.c                |   7 +-
 arch/s390/kernel/vmlinux.lds.S        |  14 ++
 drivers/s390/char/Makefile            |   2 +
 20 files changed, 591 insertions(+), 39 deletions(-)
 create mode 100644 arch/s390/include/asm/nospec-branch.h
 create mode 100644 arch/s390/kernel/nospec-branch.c

-- 
2.7.4

[PATCH 1/6] s390: scrub registers on kernel entry and KVM exit

[Martin Schwidefsky]

Clear all user space registers on entry to the kernel and all KVM guest
registers on KVM guest exit if the register does not contain either a
parameter or a result value.

Reviewed-by: Christian Borntraeger <borntraeger@de.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
---
 arch/s390/kernel/entry.S | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/arch/s390/kernel/entry.S b/arch/s390/kernel/entry.S
index 6cd444d..5d87eda 100644
--- a/arch/s390/kernel/entry.S
+++ b/arch/s390/kernel/entry.S
@@ -248,6 +248,12 @@ ENTRY(sie64a)
 sie_exit:
 	lg	%r14,__SF_EMPTY+8(%r15)		# load guest register save area
 	stmg	%r0,%r13,0(%r14)		# save guest gprs 0-13
+	xgr	%r0,%r0				# clear guest registers to
+	xgr	%r1,%r1				# prevent speculative use
+	xgr	%r2,%r2
+	xgr	%r3,%r3
+	xgr	%r4,%r4
+	xgr	%r5,%r5
 	lmg	%r6,%r14,__SF_GPRS(%r15)	# restore kernel registers
 	lg	%r2,__SF_EMPTY+16(%r15)		# return exit reason code
 	br	%r14
@@ -282,6 +288,8 @@ ENTRY(system_call)
 .Lsysc_vtime:
 	UPDATE_VTIME %r8,%r9,__LC_SYNC_ENTER_TIMER
 	stmg	%r0,%r7,__PT_R0(%r11)
+	# clear user controlled register to prevent speculative use
+	xgr	%r0,%r0
 	mvc	__PT_R8(64,%r11),__LC_SAVE_AREA_SYNC
 	mvc	__PT_PSW(16,%r11),__LC_SVC_OLD_PSW
 	mvc	__PT_INT_CODE(4,%r11),__LC_SVC_ILC
@@ -561,6 +569,15 @@ ENTRY(pgm_check_handler)
 4:	lgr	%r13,%r11
 	la	%r11,STACK_FRAME_OVERHEAD(%r15)
 	stmg	%r0,%r7,__PT_R0(%r11)
+	# clear user controlled registers to prevent speculative use
+	xgr	%r0,%r0
+	xgr	%r1,%r1
+	xgr	%r2,%r2
+	xgr	%r3,%r3
+	xgr	%r4,%r4
+	xgr	%r5,%r5
+	xgr	%r6,%r6
+	xgr	%r7,%r7
 	mvc	__PT_R8(64,%r11),__LC_SAVE_AREA_SYNC
 	stmg	%r8,%r9,__PT_PSW(%r11)
 	mvc	__PT_INT_CODE(4,%r11),__LC_PGM_ILC
@@ -626,6 +643,16 @@ ENTRY(io_int_handler)
 	lmg	%r8,%r9,__LC_IO_OLD_PSW
 	SWITCH_ASYNC __LC_SAVE_AREA_ASYNC,__LC_ASYNC_ENTER_TIMER
 	stmg	%r0,%r7,__PT_R0(%r11)
+	# clear user controlled registers to prevent speculative use
+	xgr	%r0,%r0
+	xgr	%r1,%r1
+	xgr	%r2,%r2
+	xgr	%r3,%r3
+	xgr	%r4,%r4
+	xgr	%r5,%r5
+	xgr	%r6,%r6
+	xgr	%r7,%r7
+	xgr	%r10,%r10
 	mvc	__PT_R8(64,%r11),__LC_SAVE_AREA_ASYNC
 	stmg	%r8,%r9,__PT_PSW(%r11)
 	mvc	__PT_INT_CODE(12,%r11),__LC_SUBCHANNEL_ID
@@ -839,6 +866,16 @@ ENTRY(ext_int_handler)
 	lmg	%r8,%r9,__LC_EXT_OLD_PSW
 	SWITCH_ASYNC __LC_SAVE_AREA_ASYNC,__LC_ASYNC_ENTER_TIMER
 	stmg	%r0,%r7,__PT_R0(%r11)
+	# clear user controlled registers to prevent speculative use
+	xgr	%r0,%r0
+	xgr	%r1,%r1
+	xgr	%r2,%r2
+	xgr	%r3,%r3
+	xgr	%r4,%r4
+	xgr	%r5,%r5
+	xgr	%r6,%r6
+	xgr	%r7,%r7
+	xgr	%r10,%r10
 	mvc	__PT_R8(64,%r11),__LC_SAVE_AREA_ASYNC
 	stmg	%r8,%r9,__PT_PSW(%r11)
 	lghi	%r1,__LC_EXT_PARAMS2
@@ -1046,6 +1083,16 @@ ENTRY(mcck_int_handler)
 .Lmcck_skip:
 	lghi	%r14,__LC_GPREGS_SAVE_AREA+64
 	stmg	%r0,%r7,__PT_R0(%r11)
+	# clear user controlled registers to prevent speculative use
+	xgr	%r0,%r0
+	xgr	%r1,%r1
+	xgr	%r2,%r2
+	xgr	%r3,%r3
+	xgr	%r4,%r4
+	xgr	%r5,%r5
+	xgr	%r6,%r6
+	xgr	%r7,%r7
+	xgr	%r10,%r10
 	mvc	__PT_R8(64,%r11),0(%r14)
 	stmg	%r8,%r9,__PT_PSW(%r11)
 	xc	__PT_FLAGS(8,%r11),__PT_FLAGS(%r11)
-- 
2.7.4

[PATCH 2/6] s390: add optimized array_index_mask_nospec

[Martin Schwidefsky]

Add an optimized version of the array_index_mask_nospec function for
s390 based on a compare and a subtract with borrow.

Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
---
 arch/s390/include/asm/barrier.h | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/arch/s390/include/asm/barrier.h b/arch/s390/include/asm/barrier.h
index 1043260..f9eddbc 100644
--- a/arch/s390/include/asm/barrier.h
+++ b/arch/s390/include/asm/barrier.h
@@ -49,6 +49,30 @@ do {									\
 #define __smp_mb__before_atomic()	barrier()
 #define __smp_mb__after_atomic()	barrier()
 
+/**
+ * array_index_mask_nospec - generate a mask for array_idx() that is
+ * ~0UL when the bounds check succeeds and 0 otherwise
+ * @index: array element index
+ * @size: number of elements in array
+ */
+#define array_index_mask_nospec array_index_mask_nospec
+static inline unsigned long array_index_mask_nospec(unsigned long index,
+						    unsigned long size)
+{
+	unsigned long mask;
+
+	if (__builtin_constant_p(size) && size > 0) {
+		asm("	clgr	%2,%1\n"
+		    "	slbgr	%0,%0\n"
+		    :"=d" (mask) : "d" (size-1), "d" (index) :"cc");
+		return mask;
+	}
+	asm("	clgr	%1,%2\n"
+	    "	slbgr	%0,%0\n"
+	    :"=d" (mask) : "d" (size), "d" (index) :"cc");
+	return ~mask;
+}
+
 #include <asm-generic/barrier.h>
 
 #endif /* __ASM_BARRIER_H */
-- 
2.7.4

[PATCH 3/6] s390/alternative: use a copy of the facility bit mask

[Martin Schwidefsky]

To be able to switch off specific CPU alternatives with kernel parameters
make a copy of the facility bit mask provided by STFLE and use the copy
for the decision to apply an alternative.

Reviewed-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
---
 arch/s390/include/asm/facility.h | 18 ++++++++++++++++++
 arch/s390/include/asm/lowcore.h  |  3 ++-
 arch/s390/kernel/alternative.c   |  3 ++-
 arch/s390/kernel/early.c         |  3 +++
 arch/s390/kernel/setup.c         |  4 +++-
 arch/s390/kernel/smp.c           |  4 +++-
 6 files changed, 31 insertions(+), 4 deletions(-)

diff --git a/arch/s390/include/asm/facility.h b/arch/s390/include/asm/facility.h
index fbe0c4b..99c8ce3 100644
--- a/arch/s390/include/asm/facility.h
+++ b/arch/s390/include/asm/facility.h
@@ -15,6 +15,24 @@
 
 #define MAX_FACILITY_BIT (sizeof(((struct lowcore *)0)->stfle_fac_list) * 8)
 
+static inline void __set_facility(unsigned long nr, void *facilities)
+{
+	unsigned char *ptr = (unsigned char *) facilities;
+
+	if (nr >= MAX_FACILITY_BIT)
+		return;
+	ptr[nr >> 3] |= 0x80 >> (nr & 7);
+}
+
+static inline void __clear_facility(unsigned long nr, void *facilities)
+{
+	unsigned char *ptr = (unsigned char *) facilities;
+
+	if (nr >= MAX_FACILITY_BIT)
+		return;
+	ptr[nr >> 3] &= ~(0x80 >> (nr & 7));
+}
+
 static inline int __test_facility(unsigned long nr, void *facilities)
 {
 	unsigned char *ptr;
diff --git a/arch/s390/include/asm/lowcore.h b/arch/s390/include/asm/lowcore.h
index ec6592e..c63986a 100644
--- a/arch/s390/include/asm/lowcore.h
+++ b/arch/s390/include/asm/lowcore.h
@@ -151,7 +151,8 @@ struct lowcore {
 	__u8	pad_0x0e20[0x0f00-0x0e20];	/* 0x0e20 */
 
 	/* Extended facility list */
-	__u64	stfle_fac_list[32];		/* 0x0f00 */
+	__u64	stfle_fac_list[16];		/* 0x0f00 */
+	__u64	alt_stfle_fac_list[16];		/* 0x0f80 */
 	__u8	pad_0x1000[0x11b0-0x1000];	/* 0x1000 */
 
 	/* Pointer to the machine check extended save area */
diff --git a/arch/s390/kernel/alternative.c b/arch/s390/kernel/alternative.c
index 574e776..1abf4f3 100644
--- a/arch/s390/kernel/alternative.c
+++ b/arch/s390/kernel/alternative.c
@@ -75,7 +75,8 @@ static void __init_or_module __apply_alternatives(struct alt_instr *start,
 		instr = (u8 *)&a->instr_offset + a->instr_offset;
 		replacement = (u8 *)&a->repl_offset + a->repl_offset;
 
-		if (!test_facility(a->facility))
+		if (!__test_facility(a->facility,
+				     S390_lowcore.alt_stfle_fac_list))
 			continue;
 
 		if (unlikely(a->instrlen % 2 || a->replacementlen % 2)) {
diff --git a/arch/s390/kernel/early.c b/arch/s390/kernel/early.c
index 497a920..510f218 100644
--- a/arch/s390/kernel/early.c
+++ b/arch/s390/kernel/early.c
@@ -193,6 +193,9 @@ static noinline __init void setup_facility_list(void)
 {
 	stfle(S390_lowcore.stfle_fac_list,
 	      ARRAY_SIZE(S390_lowcore.stfle_fac_list));
+	memcpy(S390_lowcore.alt_stfle_fac_list,
+	       S390_lowcore.stfle_fac_list,
+	       sizeof(S390_lowcore.alt_stfle_fac_list));
 }
 
 static __init void detect_diag9c(void)
diff --git a/arch/s390/kernel/setup.c b/arch/s390/kernel/setup.c
index 793da97..bcd2a4a 100644
--- a/arch/s390/kernel/setup.c
+++ b/arch/s390/kernel/setup.c
@@ -340,7 +340,9 @@ static void __init setup_lowcore(void)
 	lc->preempt_count = S390_lowcore.preempt_count;
 	lc->stfl_fac_list = S390_lowcore.stfl_fac_list;
 	memcpy(lc->stfle_fac_list, S390_lowcore.stfle_fac_list,
-	       MAX_FACILITY_BIT/8);
+	       sizeof(lc->stfle_fac_list));
+	memcpy(lc->alt_stfle_fac_list, S390_lowcore.alt_stfle_fac_list,
+	       sizeof(lc->alt_stfle_fac_list));
 	nmi_alloc_boot_cpu(lc);
 	vdso_alloc_boot_cpu(lc);
 	lc->sync_enter_timer = S390_lowcore.sync_enter_timer;
diff --git a/arch/s390/kernel/smp.c b/arch/s390/kernel/smp.c
index a919b2f..fc28c95 100644
--- a/arch/s390/kernel/smp.c
+++ b/arch/s390/kernel/smp.c
@@ -266,7 +266,9 @@ static void pcpu_prepare_secondary(struct pcpu *pcpu, int cpu)
 	__ctl_store(lc->cregs_save_area, 0, 15);
 	save_access_regs((unsigned int *) lc->access_regs_save_area);
 	memcpy(lc->stfle_fac_list, S390_lowcore.stfle_fac_list,
-	       MAX_FACILITY_BIT/8);
+	       sizeof(lc->stfle_fac_list));
+	memcpy(lc->alt_stfle_fac_list, S390_lowcore.alt_stfle_fac_list,
+	       sizeof(lc->alt_stfle_fac_list));
 	arch_spin_lock_setup(cpu);
 }
 
-- 
2.7.4

[PATCH 4/6] s390: add options to change branch prediction behaviour for the kernel

[Martin Schwidefsky]

Add the PPA instruction to the system entry and exit path to switch
the kernel to a different branch prediction behaviour. The instructions
are added via CPU alternatives and can be disabled with the "nospec"
or the "nobp=0" kernel parameter. If the default behaviour selected
with CONFIG_KERNEL_NOBP is set to "n" then the "nobp=1" parameter can be
used to enable the changed kernel branch prediction.

Acked-by: Cornelia Huck <cohuck@redhat.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
---
 arch/s390/Kconfig                 | 17 ++++++++++++++
 arch/s390/include/asm/processor.h |  1 +
 arch/s390/kernel/alternative.c    | 23 +++++++++++++++++++
 arch/s390/kernel/early.c          |  2 ++
 arch/s390/kernel/entry.S          | 48 +++++++++++++++++++++++++++++++++++++++
 arch/s390/kernel/ipl.c            |  1 +
 arch/s390/kernel/smp.c            |  2 ++
 7 files changed, 94 insertions(+)

diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index 0105ce2..d514e25 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -540,6 +540,23 @@ config ARCH_RANDOM
 
 	  If unsure, say Y.
 
+config KERNEL_NOBP
+	def_bool n
+	prompt "Enable modified branch prediction for the kernel by default"
+	help
+	  If this option is selected the kernel will switch to a modified
+	  branch prediction mode if the firmware interface is available.
+	  The modified branch prediction mode improves the behaviour in
+	  regard to speculative execution.
+
+	  With the option enabled the kernel parameter "nobp=0" or "nospec"
+	  can be used to run the kernel in the normal branch prediction mode.
+
+	  With the option disabled the modified branch prediction mode is
+	  enabled with the "nobp=1" kernel parameter.
+
+	  If unsure, say N.
+
 endmenu
 
 menu "Memory setup"
diff --git a/arch/s390/include/asm/processor.h b/arch/s390/include/asm/processor.h
index bfbfad4..5f37f9c 100644
--- a/arch/s390/include/asm/processor.h
+++ b/arch/s390/include/asm/processor.h
@@ -91,6 +91,7 @@ void cpu_detect_mhz_feature(void);
 extern const struct seq_operations cpuinfo_op;
 extern int sysctl_ieee_emulation_warnings;
 extern void execve_tail(void);
+extern void __bpon(void);
 
 /*
  * User space process size: 2GB for 31 bit, 4TB or 8PT for 64 bit.
diff --git a/arch/s390/kernel/alternative.c b/arch/s390/kernel/alternative.c
index 1abf4f3..2247613 100644
--- a/arch/s390/kernel/alternative.c
+++ b/arch/s390/kernel/alternative.c
@@ -15,6 +15,29 @@ static int __init disable_alternative_instructions(char *str)
 
 early_param("noaltinstr", disable_alternative_instructions);
 
+static int __init nobp_setup_early(char *str)
+{
+	bool enabled;
+	int rc;
+
+	rc = kstrtobool(str, &enabled);
+	if (rc)
+		return rc;
+	if (enabled && test_facility(82))
+		__set_facility(82, S390_lowcore.alt_stfle_fac_list);
+	else
+		__clear_facility(82, S390_lowcore.alt_stfle_fac_list);
+	return 0;
+}
+early_param("nobp", nobp_setup_early);
+
+static int __init nospec_setup_early(char *str)
+{
+	__clear_facility(82, S390_lowcore.alt_stfle_fac_list);
+	return 0;
+}
+early_param("nospec", nospec_setup_early);
+
 struct brcl_insn {
 	u16 opc;
 	s32 disp;
diff --git a/arch/s390/kernel/early.c b/arch/s390/kernel/early.c
index 510f218..ac707a9 100644
--- a/arch/s390/kernel/early.c
+++ b/arch/s390/kernel/early.c
@@ -196,6 +196,8 @@ static noinline __init void setup_facility_list(void)
 	memcpy(S390_lowcore.alt_stfle_fac_list,
 	       S390_lowcore.stfle_fac_list,
 	       sizeof(S390_lowcore.alt_stfle_fac_list));
+	if (!IS_ENABLED(CONFIG_KERNEL_NOBP))
+		__clear_facility(82, S390_lowcore.alt_stfle_fac_list);
 }
 
 static __init void detect_diag9c(void)
diff --git a/arch/s390/kernel/entry.S b/arch/s390/kernel/entry.S
index 5d87eda..e6d7550 100644
--- a/arch/s390/kernel/entry.S
+++ b/arch/s390/kernel/entry.S
@@ -159,6 +159,34 @@ _PIF_WORK	= (_PIF_PER_TRAP | _PIF_SYSCALL_RESTART)
 		tm	off+\addr, \mask
 	.endm
 
+	.macro BPOFF
+	.pushsection .altinstr_replacement, "ax"
+660:	.long	0xb2e8c000
+	.popsection
+661:	.long	0x47000000
+	.pushsection .altinstructions, "a"
+	.long 661b - .
+	.long 660b - .
+	.word 82
+	.byte 4
+	.byte 4
+	.popsection
+	.endm
+
+	.macro BPON
+	.pushsection .altinstr_replacement, "ax"
+662:	.long	0xb2e8d000
+	.popsection
+663:	.long	0x47000000
+	.pushsection .altinstructions, "a"
+	.long 663b - .
+	.long 662b - .
+	.word 82
+	.byte 4
+	.byte 4
+	.popsection
+	.endm
+
 	.section .kprobes.text, "ax"
 .Ldummy:
 	/*
@@ -171,6 +199,11 @@ _PIF_WORK	= (_PIF_PER_TRAP | _PIF_SYSCALL_RESTART)
 	 */
 	nop	0
 
+ENTRY(__bpon)
+	.globl __bpon
+	BPON
+	br	%r14
+
 /*
  * Scheduler resume function, called by switch_to
  *  gpr2 = (task_struct *) prev
@@ -226,8 +259,11 @@ ENTRY(sie64a)
 	jnz	.Lsie_skip
 	TSTMSK	__LC_CPU_FLAGS,_CIF_FPU
 	jo	.Lsie_skip			# exit if fp/vx regs changed
+	BPON
 .Lsie_entry:
 	sie	0(%r14)
+.Lsie_exit:
+	BPOFF
 .Lsie_skip:
 	ni	__SIE_PROG0C+3(%r14),0xfe	# no longer in SIE
 	lctlg	%c1,%c1,__LC_USER_ASCE		# load primary asce
@@ -279,6 +315,7 @@ ENTRY(system_call)
 	stpt	__LC_SYNC_ENTER_TIMER
 .Lsysc_stmg:
 	stmg	%r8,%r15,__LC_SAVE_AREA_SYNC
+	BPOFF
 	lg	%r12,__LC_CURRENT
 	lghi	%r13,__TASK_thread
 	lghi	%r14,_PIF_SYSCALL
@@ -325,6 +362,7 @@ ENTRY(system_call)
 	jnz	.Lsysc_work			# check for work
 	TSTMSK	__LC_CPU_FLAGS,_CIF_WORK
 	jnz	.Lsysc_work
+	BPON
 .Lsysc_restore:
 	lg	%r14,__LC_VDSO_PER_CPU
 	lmg	%r0,%r10,__PT_R0(%r11)
@@ -530,6 +568,7 @@ ENTRY(kernel_thread_starter)
 
 ENTRY(pgm_check_handler)
 	stpt	__LC_SYNC_ENTER_TIMER
+	BPOFF
 	stmg	%r8,%r15,__LC_SAVE_AREA_SYNC
 	lg	%r10,__LC_LAST_BREAK
 	lg	%r12,__LC_CURRENT
@@ -637,6 +676,7 @@ ENTRY(pgm_check_handler)
 ENTRY(io_int_handler)
 	STCK	__LC_INT_CLOCK
 	stpt	__LC_ASYNC_ENTER_TIMER
+	BPOFF
 	stmg	%r8,%r15,__LC_SAVE_AREA_ASYNC
 	lg	%r12,__LC_CURRENT
 	larl	%r13,cleanup_critical
@@ -687,9 +727,13 @@ ENTRY(io_int_handler)
 	lg	%r14,__LC_VDSO_PER_CPU
 	lmg	%r0,%r10,__PT_R0(%r11)
 	mvc	__LC_RETURN_PSW(16),__PT_PSW(%r11)
+	tm	__PT_PSW+1(%r11),0x01	# returning to user ?
+	jno	.Lio_exit_kernel
+	BPON
 .Lio_exit_timer:
 	stpt	__LC_EXIT_TIMER
 	mvc	__VDSO_ECTG_BASE(16,%r14),__LC_EXIT_TIMER
+.Lio_exit_kernel:
 	lmg	%r11,%r15,__PT_R11(%r11)
 	lpswe	__LC_RETURN_PSW
 .Lio_done:
@@ -860,6 +904,7 @@ ENTRY(io_int_handler)
 ENTRY(ext_int_handler)
 	STCK	__LC_INT_CLOCK
 	stpt	__LC_ASYNC_ENTER_TIMER
+	BPOFF
 	stmg	%r8,%r15,__LC_SAVE_AREA_ASYNC
 	lg	%r12,__LC_CURRENT
 	larl	%r13,cleanup_critical
@@ -908,6 +953,7 @@ ENTRY(psw_idle)
 .Lpsw_idle_stcctm:
 #endif
 	oi	__LC_CPU_FLAGS+7,_CIF_ENABLED_WAIT
+	BPON
 	STCK	__CLOCK_IDLE_ENTER(%r2)
 	stpt	__TIMER_IDLE_ENTER(%r2)
 .Lpsw_idle_lpsw:
@@ -1008,6 +1054,7 @@ load_fpu_regs:
  */
 ENTRY(mcck_int_handler)
 	STCK	__LC_MCCK_CLOCK
+	BPOFF
 	la	%r1,4095		# validate r1
 	spt	__LC_CPU_TIMER_SAVE_AREA-4095(%r1)	# validate cpu timer
 	sckc	__LC_CLOCK_COMPARATOR			# validate comparator
@@ -1118,6 +1165,7 @@ ENTRY(mcck_int_handler)
 	mvc	__LC_RETURN_MCCK_PSW(16),__PT_PSW(%r11) # move return PSW
 	tm	__LC_RETURN_MCCK_PSW+1,0x01 # returning to user ?
 	jno	0f
+	BPON
 	stpt	__LC_EXIT_TIMER
 	mvc	__VDSO_ECTG_BASE(16,%r14),__LC_EXIT_TIMER
 0:	lmg	%r11,%r15,__PT_R11(%r11)
diff --git a/arch/s390/kernel/ipl.c b/arch/s390/kernel/ipl.c
index da5cc3b..34477c1 100644
--- a/arch/s390/kernel/ipl.c
+++ b/arch/s390/kernel/ipl.c
@@ -543,6 +543,7 @@ static struct kset *ipl_kset;
 
 static void __ipl_run(void *unused)
 {
+	__bpon();
 	diag308(DIAG308_LOAD_CLEAR, NULL);
 	if (MACHINE_IS_VM)
 		__cpcmd("IPL", NULL, 0, NULL);
diff --git a/arch/s390/kernel/smp.c b/arch/s390/kernel/smp.c
index fc28c95..2fd7d60 100644
--- a/arch/s390/kernel/smp.c
+++ b/arch/s390/kernel/smp.c
@@ -319,6 +319,7 @@ static void pcpu_delegate(struct pcpu *pcpu, void (*func)(void *),
 	mem_assign_absolute(lc->restart_fn, (unsigned long) func);
 	mem_assign_absolute(lc->restart_data, (unsigned long) data);
 	mem_assign_absolute(lc->restart_source, source_cpu);
+	__bpon();
 	asm volatile(
 		"0:	sigp	0,%0,%2	# sigp restart to target cpu\n"
 		"	brc	2,0b	# busy, try again\n"
@@ -903,6 +904,7 @@ void __cpu_die(unsigned int cpu)
 void __noreturn cpu_die(void)
 {
 	idle_task_exit();
+	__bpon();
 	pcpu_sigp_retry(pcpu_devices + smp_processor_id(), SIGP_STOP, 0);
 	for (;;) ;
 }
-- 
2.7.4

[PATCH 5/6] s390: run user space and KVM guests with modified branch prediction

[Martin Schwidefsky]

Define TIF_ISOLATE_BP and TIF_ISOLATE_BP_GUEST and add the necessary
plumbing in entry.S to be able to run user space and KVM guests with
limited branch prediction.

To switch a user space process to limited branch prediction the
s390_isolate_bp() function has to be call, and to run a vCPU of a KVM
guest associated with the current task with limited branch prediction
call s390_isolate_bp_guest().

Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
---
 arch/s390/include/asm/processor.h   |  3 +++
 arch/s390/include/asm/thread_info.h |  4 +++
 arch/s390/kernel/entry.S            | 51 +++++++++++++++++++++++++++++++++----
 arch/s390/kernel/processor.c        | 18 +++++++++++++
 4 files changed, 71 insertions(+), 5 deletions(-)

diff --git a/arch/s390/include/asm/processor.h b/arch/s390/include/asm/processor.h
index 5f37f9c..7f2953c 100644
--- a/arch/s390/include/asm/processor.h
+++ b/arch/s390/include/asm/processor.h
@@ -378,6 +378,9 @@ extern void memcpy_absolute(void *, void *, size_t);
 	memcpy_absolute(&(dest), &__tmp, sizeof(__tmp));	\
 } while (0)
 
+extern int s390_isolate_bp(void);
+extern int s390_isolate_bp_guest(void);
+
 #endif /* __ASSEMBLY__ */
 
 #endif /* __ASM_S390_PROCESSOR_H */
diff --git a/arch/s390/include/asm/thread_info.h b/arch/s390/include/asm/thread_info.h
index 25d6ec3..83ba575 100644
--- a/arch/s390/include/asm/thread_info.h
+++ b/arch/s390/include/asm/thread_info.h
@@ -58,6 +58,8 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
 #define TIF_GUARDED_STORAGE	4	/* load guarded storage control block */
 #define TIF_PATCH_PENDING	5	/* pending live patching update */
 #define TIF_PGSTE		6	/* New mm's will use 4K page tables */
+#define TIF_ISOLATE_BP		8	/* Run process with isolated BP */
+#define TIF_ISOLATE_BP_GUEST	9	/* Run KVM guests with isolated BP */
 
 #define TIF_31BIT		16	/* 32bit process */
 #define TIF_MEMDIE		17	/* is terminating due to OOM killer */
@@ -78,6 +80,8 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
 #define _TIF_UPROBE		_BITUL(TIF_UPROBE)
 #define _TIF_GUARDED_STORAGE	_BITUL(TIF_GUARDED_STORAGE)
 #define _TIF_PATCH_PENDING	_BITUL(TIF_PATCH_PENDING)
+#define _TIF_ISOLATE_BP		_BITUL(TIF_ISOLATE_BP)
+#define _TIF_ISOLATE_BP_GUEST	_BITUL(TIF_ISOLATE_BP_GUEST)
 
 #define _TIF_31BIT		_BITUL(TIF_31BIT)
 #define _TIF_SINGLE_STEP	_BITUL(TIF_SINGLE_STEP)
diff --git a/arch/s390/kernel/entry.S b/arch/s390/kernel/entry.S
index e6d7550..53145b5 100644
--- a/arch/s390/kernel/entry.S
+++ b/arch/s390/kernel/entry.S
@@ -107,6 +107,7 @@ _PIF_WORK	= (_PIF_PER_TRAP | _PIF_SYSCALL_RESTART)
 	aghi	%r15,-(STACK_FRAME_OVERHEAD + __PT_SIZE)
 	j	3f
 1:	UPDATE_VTIME %r14,%r15,\timer
+	BPENTER __TI_flags(%r12),_TIF_ISOLATE_BP
 2:	lg	%r15,__LC_ASYNC_STACK	# load async stack
 3:	la	%r11,STACK_FRAME_OVERHEAD(%r15)
 	.endm
@@ -187,6 +188,40 @@ _PIF_WORK	= (_PIF_PER_TRAP | _PIF_SYSCALL_RESTART)
 	.popsection
 	.endm
 
+	.macro BPENTER tif_ptr,tif_mask
+	.pushsection .altinstr_replacement, "ax"
+662:	.word	0xc004, 0x0000, 0x0000	# 6 byte nop
+	.word	0xc004, 0x0000, 0x0000	# 6 byte nop
+	.popsection
+664:	TSTMSK	\tif_ptr,\tif_mask
+	jz	. + 8
+	.long	0xb2e8d000
+	.pushsection .altinstructions, "a"
+	.long 664b - .
+	.long 662b - .
+	.word 82
+	.byte 12
+	.byte 12
+	.popsection
+	.endm
+
+	.macro BPEXIT tif_ptr,tif_mask
+	TSTMSK	\tif_ptr,\tif_mask
+	.pushsection .altinstr_replacement, "ax"
+662:	jnz	. + 8
+	.long	0xb2e8d000
+	.popsection
+664:	jz	. + 8
+	.long	0xb2e8c000
+	.pushsection .altinstructions, "a"
+	.long 664b - .
+	.long 662b - .
+	.word 82
+	.byte 8
+	.byte 8
+	.popsection
+	.endm
+
 	.section .kprobes.text, "ax"
 .Ldummy:
 	/*
@@ -240,9 +275,11 @@ ENTRY(__switch_to)
  */
 ENTRY(sie64a)
 	stmg	%r6,%r14,__SF_GPRS(%r15)	# save kernel registers
+	lg	%r12,__LC_CURRENT
 	stg	%r2,__SF_EMPTY(%r15)		# save control block pointer
 	stg	%r3,__SF_EMPTY+8(%r15)		# save guest register save area
 	xc	__SF_EMPTY+16(8,%r15),__SF_EMPTY+16(%r15) # reason code = 0
+	mvc	__SF_EMPTY+24(8,%r15),__TI_flags(%r12) # copy thread flags
 	TSTMSK	__LC_CPU_FLAGS,_CIF_FPU		# load guest fp/vx registers ?
 	jno	.Lsie_load_guest_gprs
 	brasl	%r14,load_fpu_regs		# load guest fp/vx regs
@@ -259,11 +296,12 @@ ENTRY(sie64a)
 	jnz	.Lsie_skip
 	TSTMSK	__LC_CPU_FLAGS,_CIF_FPU
 	jo	.Lsie_skip			# exit if fp/vx regs changed
-	BPON
+	BPEXIT	__SF_EMPTY+24(%r15),(_TIF_ISOLATE_BP|_TIF_ISOLATE_BP_GUEST)
 .Lsie_entry:
 	sie	0(%r14)
 .Lsie_exit:
 	BPOFF
+	BPENTER	__SF_EMPTY+24(%r15),(_TIF_ISOLATE_BP|_TIF_ISOLATE_BP_GUEST)
 .Lsie_skip:
 	ni	__SIE_PROG0C+3(%r14),0xfe	# no longer in SIE
 	lctlg	%c1,%c1,__LC_USER_ASCE		# load primary asce
@@ -324,6 +362,7 @@ ENTRY(system_call)
 	la	%r11,STACK_FRAME_OVERHEAD(%r15)	# pointer to pt_regs
 .Lsysc_vtime:
 	UPDATE_VTIME %r8,%r9,__LC_SYNC_ENTER_TIMER
+	BPENTER __TI_flags(%r12),_TIF_ISOLATE_BP
 	stmg	%r0,%r7,__PT_R0(%r11)
 	# clear user controlled register to prevent speculative use
 	xgr	%r0,%r0
@@ -362,7 +401,7 @@ ENTRY(system_call)
 	jnz	.Lsysc_work			# check for work
 	TSTMSK	__LC_CPU_FLAGS,_CIF_WORK
 	jnz	.Lsysc_work
-	BPON
+	BPEXIT	__TI_flags(%r12),_TIF_ISOLATE_BP
 .Lsysc_restore:
 	lg	%r14,__LC_VDSO_PER_CPU
 	lmg	%r0,%r10,__PT_R0(%r11)
@@ -597,6 +636,7 @@ ENTRY(pgm_check_handler)
 	aghi	%r15,-(STACK_FRAME_OVERHEAD + __PT_SIZE)
 	j	4f
 2:	UPDATE_VTIME %r14,%r15,__LC_SYNC_ENTER_TIMER
+	BPENTER __TI_flags(%r12),_TIF_ISOLATE_BP
 	lg	%r15,__LC_KERNEL_STACK
 	lgr	%r14,%r12
 	aghi	%r14,__TASK_thread	# pointer to thread_struct
@@ -729,7 +769,7 @@ ENTRY(io_int_handler)
 	mvc	__LC_RETURN_PSW(16),__PT_PSW(%r11)
 	tm	__PT_PSW+1(%r11),0x01	# returning to user ?
 	jno	.Lio_exit_kernel
-	BPON
+	BPEXIT	__TI_flags(%r12),_TIF_ISOLATE_BP
 .Lio_exit_timer:
 	stpt	__LC_EXIT_TIMER
 	mvc	__VDSO_ECTG_BASE(16,%r14),__LC_EXIT_TIMER
@@ -1165,7 +1205,7 @@ ENTRY(mcck_int_handler)
 	mvc	__LC_RETURN_MCCK_PSW(16),__PT_PSW(%r11) # move return PSW
 	tm	__LC_RETURN_MCCK_PSW+1,0x01 # returning to user ?
 	jno	0f
-	BPON
+	BPEXIT	__TI_flags(%r12),_TIF_ISOLATE_BP
 	stpt	__LC_EXIT_TIMER
 	mvc	__VDSO_ECTG_BASE(16,%r14),__LC_EXIT_TIMER
 0:	lmg	%r11,%r15,__PT_R11(%r11)
@@ -1292,7 +1332,8 @@ cleanup_critical:
 	clg     %r9,BASED(.Lsie_crit_mcck_length)
 	jh      1f
 	oi      __LC_CPU_FLAGS+7, _CIF_MCCK_GUEST
-1:	lg	%r9,__SF_EMPTY(%r15)		# get control block pointer
+1:	BPENTER __SF_EMPTY+24(%r15),(_TIF_ISOLATE_BP|_TIF_ISOLATE_BP_GUEST)
+	lg	%r9,__SF_EMPTY(%r15)		# get control block pointer
 	ni	__SIE_PROG0C+3(%r9),0xfe	# no longer in SIE
 	lctlg	%c1,%c1,__LC_USER_ASCE		# load primary asce
 	larl	%r9,sie_exit			# skip forward to sie_exit
diff --git a/arch/s390/kernel/processor.c b/arch/s390/kernel/processor.c
index 5362fd8..6fe2e18 100644
--- a/arch/s390/kernel/processor.c
+++ b/arch/s390/kernel/processor.c
@@ -197,3 +197,21 @@ const struct seq_operations cpuinfo_op = {
 	.stop	= c_stop,
 	.show	= show_cpuinfo,
 };
+
+int s390_isolate_bp(void)
+{
+	if (!test_facility(82))
+		return -EOPNOTSUPP;
+	set_thread_flag(TIF_ISOLATE_BP);
+	return 0;
+}
+EXPORT_SYMBOL(s390_isolate_bp);
+
+int s390_isolate_bp_guest(void)
+{
+	if (!test_facility(82))
+		return -EOPNOTSUPP;
+	set_thread_flag(TIF_ISOLATE_BP_GUEST);
+	return 0;
+}
+EXPORT_SYMBOL(s390_isolate_bp_guest);
-- 
2.7.4

[PATCH 6/6] s390: introduce execute-trampolines for branches

[Martin Schwidefsky]

Add CONFIG_EXPOLINE to enable the use of the new -mindirect-branch= and
-mfunction_return= compiler options to create a kernel fortified against
the specte v2 attack.

With CONFIG_EXPOLINE=y all indirect branches will be issued with an
execute type instruction. For z10 or newer the EXRL instruction will
be used, for older machines the EX instruction. The typical indirect
call

	basr	%r14,%r1

is replaced with a PC relative call to a new thunk

	brasl	%r14,__s390x_indirect_jump_r1

The thunk contains the EXRL/EX instruction to the indirect branch

__s390x_indirect_jump_r1:
	exrl	0,0f
	j	.
0:	br	%r1

The detour via the execute type instruction has a performance impact.
To get rid of the detour the new kernel parameter "nospectre_v2" and
"spectre_v2=[on,off,auto]" can be used. If the parameter is specified
the kernel and module code will be patched at runtime.

Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
---
 arch/s390/Kconfig                     |  28 +++++++++
 arch/s390/Makefile                    |  12 ++++
 arch/s390/include/asm/lowcore.h       |   6 +-
 arch/s390/include/asm/nospec-branch.h |  18 ++++++
 arch/s390/kernel/Makefile             |   4 ++
 arch/s390/kernel/entry.S              | 113 ++++++++++++++++++++++++++--------
 arch/s390/kernel/module.c             |  62 ++++++++++++++++---
 arch/s390/kernel/nospec-branch.c      | 100 ++++++++++++++++++++++++++++++
 arch/s390/kernel/setup.c              |   4 ++
 arch/s390/kernel/smp.c                |   1 +
 arch/s390/kernel/vmlinux.lds.S        |  14 +++++
 drivers/s390/char/Makefile            |   2 +
 12 files changed, 329 insertions(+), 35 deletions(-)
 create mode 100644 arch/s390/include/asm/nospec-branch.h
 create mode 100644 arch/s390/kernel/nospec-branch.c

diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index d514e25..d4a65bf 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -557,6 +557,34 @@ config KERNEL_NOBP
 
 	  If unsure, say N.
 
+config EXPOLINE
+	def_bool n
+	prompt "Avoid speculative indirect branches in the kernel"
+	help
+	  Compile the kernel with the expoline compiler options to guard
+	  against kernel-to-user data leaks by avoiding speculative indirect
+	  branches.
+	  Requires a compiler with -mindirect-branch=thunk support for full
+	  protection. The kernel may run slower.
+
+	  If unsure, say N.
+
+choice
+	prompt "Expoline default"
+	depends on EXPOLINE
+	default EXPOLINE_FULL
+
+config EXPOLINE_OFF
+	bool "spectre_v2=off"
+
+config EXPOLINE_MEDIUM
+	bool "spectre_v2=auto"
+
+config EXPOLINE_FULL
+	bool "spectre_v2=on"
+
+endchoice
+
 endmenu
 
 menu "Memory setup"
diff --git a/arch/s390/Makefile b/arch/s390/Makefile
index fd691c4..2f925ef 100644
--- a/arch/s390/Makefile
+++ b/arch/s390/Makefile
@@ -78,6 +78,18 @@ ifeq ($(call cc-option-yn,-mwarn-dynamicstack),y)
 cflags-$(CONFIG_WARN_DYNAMIC_STACK) += -mwarn-dynamicstack
 endif
 
+ifdef CONFIG_EXPOLINE
+  ifeq ($(call cc-option-yn,$(CC_FLAGS_MARCH) -mindirect-branch=thunk),y)
+    CC_FLAGS_EXPOLINE := -mindirect-branch=thunk
+    CC_FLAGS_EXPOLINE += -mfunction-return=thunk
+    CC_FLAGS_EXPOLINE += -mindirect-branch-table
+    export CC_FLAGS_EXPOLINE
+    cflags-y += $(CC_FLAGS_EXPOLINE)
+  else
+    $(warning "Your gcc lacks the -mindirect-branch= option")
+  endif
+endif
+
 ifdef CONFIG_FUNCTION_TRACER
 # make use of hotpatch feature if the compiler supports it
 cc_hotpatch	:= -mhotpatch=0,3
diff --git a/arch/s390/include/asm/lowcore.h b/arch/s390/include/asm/lowcore.h
index c63986a..5bc8888 100644
--- a/arch/s390/include/asm/lowcore.h
+++ b/arch/s390/include/asm/lowcore.h
@@ -136,7 +136,11 @@ struct lowcore {
 	__u64	vdso_per_cpu_data;		/* 0x03b8 */
 	__u64	machine_flags;			/* 0x03c0 */
 	__u64	gmap;				/* 0x03c8 */
-	__u8	pad_0x03d0[0x0e00-0x03d0];	/* 0x03d0 */
+	__u8	pad_0x03d0[0x0400-0x03d0];	/* 0x03d0 */
+
+	/* br %r1 trampoline */
+	__u16	br_r1_trampoline;		/* 0x0400 */
+	__u8	pad_0x0402[0x0e00-0x0402];	/* 0x0402 */
 
 	/*
 	 * 0xe00 contains the address of the IPL Parameter Information
diff --git a/arch/s390/include/asm/nospec-branch.h b/arch/s390/include/asm/nospec-branch.h
new file mode 100644
index 0000000..7df48e5
--- /dev/null
+++ b/arch/s390/include/asm/nospec-branch.h
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_S390_EXPOLINE_H
+#define _ASM_S390_EXPOLINE_H
+
+#ifndef __ASSEMBLY__
+
+#include <linux/types.h>
+
+extern int nospec_call_disable;
+extern int nospec_return_disable;
+
+void nospec_init_branches(void);
+void nospec_call_revert(s32 *start, s32 *end);
+void nospec_return_revert(s32 *start, s32 *end);
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* _ASM_S390_EXPOLINE_H */
diff --git a/arch/s390/kernel/Makefile b/arch/s390/kernel/Makefile
index 909bce6..7f27e3d 100644
--- a/arch/s390/kernel/Makefile
+++ b/arch/s390/kernel/Makefile
@@ -29,6 +29,7 @@ UBSAN_SANITIZE_early.o	:= n
 #
 ifneq ($(CC_FLAGS_MARCH),-march=z900)
 CFLAGS_REMOVE_als.o	+= $(CC_FLAGS_MARCH)
+CFLAGS_REMOVE_als.o	+= $(CC_FLAGS_EXPOLINE)
 CFLAGS_als.o		+= -march=z900
 AFLAGS_REMOVE_head.o	+= $(CC_FLAGS_MARCH)
 AFLAGS_head.o		+= -march=z900
@@ -63,6 +64,9 @@ obj-y	+= entry.o reipl.o relocate_kernel.o kdebugfs.o alternative.o
 
 extra-y				+= head.o head64.o vmlinux.lds
 
+obj-$(CONFIG_EXPOLINE)		+= nospec-branch.o
+CFLAGS_REMOVE_expoline.o	+= $(CC_FLAGS_EXPOLINE)
+
 obj-$(CONFIG_MODULES)		+= module.o
 obj-$(CONFIG_SMP)		+= smp.o
 obj-$(CONFIG_SCHED_TOPOLOGY)	+= topology.o
diff --git a/arch/s390/kernel/entry.S b/arch/s390/kernel/entry.S
index 53145b5..13a133a 100644
--- a/arch/s390/kernel/entry.S
+++ b/arch/s390/kernel/entry.S
@@ -222,6 +222,68 @@ _PIF_WORK	= (_PIF_PER_TRAP | _PIF_SYSCALL_RESTART)
 	.popsection
 	.endm
 
+#ifdef CONFIG_EXPOLINE
+
+	.macro GEN_BR_THUNK name,reg,tmp
+	.section .text.\name,"axG",@progbits,\name,comdat
+	.globl \name
+	.hidden \name
+	.type \name,@function
+\name:
+	.cfi_startproc
+#ifdef CONFIG_HAVE_MARCH_Z10_FEATURES
+	exrl	0,0f
+#else
+	larl	\tmp,0f
+	ex	0,0(\tmp)
+#endif
+	j	.
+0:	br	\reg
+	.cfi_endproc
+	.endm
+
+	GEN_BR_THUNK __s390x_indirect_jump_r1use_r9,%r9,%r1
+	GEN_BR_THUNK __s390x_indirect_jump_r1use_r14,%r14,%r1
+	GEN_BR_THUNK __s390x_indirect_jump_r11use_r14,%r14,%r11
+
+	.macro BASR_R14_R9
+0:	brasl	%r14,__s390x_indirect_jump_r1use_r9
+	.pushsection .s390_indirect_branches,"a",@progbits
+	.long	0b-.
+	.popsection
+	.endm
+
+	.macro BR_R1USE_R14
+0:	jg	__s390x_indirect_jump_r1use_r14
+	.pushsection .s390_indirect_branches,"a",@progbits
+	.long	0b-.
+	.popsection
+	.endm
+
+	.macro BR_R11USE_R14
+0:	jg	__s390x_indirect_jump_r11use_r14
+	.pushsection .s390_indirect_branches,"a",@progbits
+	.long	0b-.
+	.popsection
+	.endm
+
+#else	/* CONFIG_EXPOLINE */
+
+	.macro BASR_R14_R9
+	basr	%r14,%r9
+	.endm
+
+	.macro BR_R1USE_R14
+	br	%r14
+	.endm
+
+	.macro BR_R11USE_R14
+	br	%r14
+	.endm
+
+#endif /* CONFIG_EXPOLINE */
+
+
 	.section .kprobes.text, "ax"
 .Ldummy:
 	/*
@@ -237,7 +299,7 @@ _PIF_WORK	= (_PIF_PER_TRAP | _PIF_SYSCALL_RESTART)
 ENTRY(__bpon)
 	.globl __bpon
 	BPON
-	br	%r14
+	BR_R1USE_R14
 
 /*
  * Scheduler resume function, called by switch_to
@@ -261,9 +323,9 @@ ENTRY(__switch_to)
 	mvc	__LC_CURRENT_PID(4,%r0),0(%r3)	# store pid of next
 	lmg	%r6,%r15,__SF_GPRS(%r15)	# load gprs of next task
 	TSTMSK	__LC_MACHINE_FLAGS,MACHINE_FLAG_LPP
-	bzr	%r14
+	jz	0f
 	.insn	s,0xb2800000,__LC_LPP		# set program parameter
-	br	%r14
+0:	BR_R1USE_R14
 
 .L__critical_start:
 
@@ -330,7 +392,7 @@ sie_exit:
 	xgr	%r5,%r5
 	lmg	%r6,%r14,__SF_GPRS(%r15)	# restore kernel registers
 	lg	%r2,__SF_EMPTY+16(%r15)		# return exit reason code
-	br	%r14
+	BR_R1USE_R14
 .Lsie_fault:
 	lghi	%r14,-EFAULT
 	stg	%r14,__SF_EMPTY+16(%r15)	# set exit reason code
@@ -389,7 +451,7 @@ ENTRY(system_call)
 	lgf	%r9,0(%r8,%r10)			# get system call add.
 	TSTMSK	__TI_flags(%r12),_TIF_TRACE
 	jnz	.Lsysc_tracesys
-	basr	%r14,%r9			# call sys_xxxx
+	BASR_R14_R9				# call sys_xxxx
 	stg	%r2,__PT_R2(%r11)		# store return value
 
 .Lsysc_return:
@@ -574,7 +636,7 @@ ENTRY(system_call)
 	lmg	%r3,%r7,__PT_R3(%r11)
 	stg	%r7,STACK_FRAME_OVERHEAD(%r15)
 	lg	%r2,__PT_ORIG_GPR2(%r11)
-	basr	%r14,%r9		# call sys_xxx
+	BASR_R14_R9			# call sys_xxx
 	stg	%r2,__PT_R2(%r11)	# store return value
 .Lsysc_tracenogo:
 	TSTMSK	__TI_flags(%r12),_TIF_TRACE
@@ -598,7 +660,7 @@ ENTRY(ret_from_fork)
 	lmg	%r9,%r10,__PT_R9(%r11)	# load gprs
 ENTRY(kernel_thread_starter)
 	la	%r2,0(%r10)
-	basr	%r14,%r9
+	BASR_R14_R9
 	j	.Lsysc_tracenogo
 
 /*
@@ -678,9 +740,9 @@ ENTRY(pgm_check_handler)
 	nill	%r10,0x007f
 	sll	%r10,2
 	je	.Lpgm_return
-	lgf	%r1,0(%r10,%r1)		# load address of handler routine
+	lgf	%r9,0(%r10,%r1)		# load address of handler routine
 	lgr	%r2,%r11		# pass pointer to pt_regs
-	basr	%r14,%r1		# branch to interrupt-handler
+	BASR_R14_R9			# branch to interrupt-handler
 .Lpgm_return:
 	LOCKDEP_SYS_EXIT
 	tm	__PT_PSW+1(%r11),0x01	# returning to user ?
@@ -998,7 +1060,7 @@ ENTRY(psw_idle)
 	stpt	__TIMER_IDLE_ENTER(%r2)
 .Lpsw_idle_lpsw:
 	lpswe	__SF_EMPTY(%r15)
-	br	%r14
+	BR_R1USE_R14
 .Lpsw_idle_end:
 
 /*
@@ -1012,7 +1074,7 @@ ENTRY(save_fpu_regs)
 	lg	%r2,__LC_CURRENT
 	aghi	%r2,__TASK_thread
 	TSTMSK	__LC_CPU_FLAGS,_CIF_FPU
-	bor	%r14
+	jo	.Lsave_fpu_regs_exit
 	stfpc	__THREAD_FPU_fpc(%r2)
 	lg	%r3,__THREAD_FPU_regs(%r2)
 	TSTMSK	__LC_MACHINE_FLAGS,MACHINE_FLAG_VX
@@ -1039,7 +1101,8 @@ ENTRY(save_fpu_regs)
 	std	15,120(%r3)
 .Lsave_fpu_regs_done:
 	oi	__LC_CPU_FLAGS+7,_CIF_FPU
-	br	%r14
+.Lsave_fpu_regs_exit:
+	BR_R1USE_R14
 .Lsave_fpu_regs_end:
 EXPORT_SYMBOL(save_fpu_regs)
 
@@ -1057,7 +1120,7 @@ load_fpu_regs:
 	lg	%r4,__LC_CURRENT
 	aghi	%r4,__TASK_thread
 	TSTMSK	__LC_CPU_FLAGS,_CIF_FPU
-	bnor	%r14
+	jno	.Lload_fpu_regs_exit
 	lfpc	__THREAD_FPU_fpc(%r4)
 	TSTMSK	__LC_MACHINE_FLAGS,MACHINE_FLAG_VX
 	lg	%r4,__THREAD_FPU_regs(%r4)	# %r4 <- reg save area
@@ -1084,7 +1147,8 @@ load_fpu_regs:
 	ld	15,120(%r4)
 .Lload_fpu_regs_done:
 	ni	__LC_CPU_FLAGS+7,255-_CIF_FPU
-	br	%r14
+.Lload_fpu_regs_exit:
+	BR_R1USE_R14
 .Lload_fpu_regs_end:
 
 .L__critical_end:
@@ -1301,7 +1365,7 @@ cleanup_critical:
 	jl	0f
 	clg	%r9,BASED(.Lcleanup_table+104)	# .Lload_fpu_regs_end
 	jl	.Lcleanup_load_fpu_regs
-0:	br	%r14
+0:	BR_R11USE_R14
 
 	.align	8
 .Lcleanup_table:
@@ -1337,7 +1401,7 @@ cleanup_critical:
 	ni	__SIE_PROG0C+3(%r9),0xfe	# no longer in SIE
 	lctlg	%c1,%c1,__LC_USER_ASCE		# load primary asce
 	larl	%r9,sie_exit			# skip forward to sie_exit
-	br	%r14
+	BR_R11USE_R14
 #endif
 
 .Lcleanup_system_call:
@@ -1390,7 +1454,7 @@ cleanup_critical:
 	stg	%r15,56(%r11)		# r15 stack pointer
 	# set new psw address and exit
 	larl	%r9,.Lsysc_do_svc
-	br	%r14
+	BR_R11USE_R14
 .Lcleanup_system_call_insn:
 	.quad	system_call
 	.quad	.Lsysc_stmg
@@ -1402,7 +1466,7 @@ cleanup_critical:
 
 .Lcleanup_sysc_tif:
 	larl	%r9,.Lsysc_tif
-	br	%r14
+	BR_R11USE_R14
 
 .Lcleanup_sysc_restore:
 	# check if stpt has been executed
@@ -1419,14 +1483,14 @@ cleanup_critical:
 	mvc	0(64,%r11),__PT_R8(%r9)
 	lmg	%r0,%r7,__PT_R0(%r9)
 1:	lmg	%r8,%r9,__LC_RETURN_PSW
-	br	%r14
+	BR_R11USE_R14
 .Lcleanup_sysc_restore_insn:
 	.quad	.Lsysc_exit_timer
 	.quad	.Lsysc_done - 4
 
 .Lcleanup_io_tif:
 	larl	%r9,.Lio_tif
-	br	%r14
+	BR_R11USE_R14
 
 .Lcleanup_io_restore:
 	# check if stpt has been executed
@@ -1440,7 +1504,7 @@ cleanup_critical:
 	mvc	0(64,%r11),__PT_R8(%r9)
 	lmg	%r0,%r7,__PT_R0(%r9)
 1:	lmg	%r8,%r9,__LC_RETURN_PSW
-	br	%r14
+	BR_R11USE_R14
 .Lcleanup_io_restore_insn:
 	.quad	.Lio_exit_timer
 	.quad	.Lio_done - 4
@@ -1493,17 +1557,17 @@ cleanup_critical:
 	# prepare return psw
 	nihh	%r8,0xfcfd		# clear irq & wait state bits
 	lg	%r9,48(%r11)		# return from psw_idle
-	br	%r14
+	BR_R11USE_R14
 .Lcleanup_idle_insn:
 	.quad	.Lpsw_idle_lpsw
 
 .Lcleanup_save_fpu_regs:
 	larl	%r9,save_fpu_regs
-	br	%r14
+	BR_R11USE_R14
 
 .Lcleanup_load_fpu_regs:
 	larl	%r9,load_fpu_regs
-	br	%r14
+	BR_R11USE_R14
 
 /*
  * Integer constants
@@ -1523,7 +1587,6 @@ cleanup_critical:
 .Lsie_crit_mcck_length:
 	.quad   .Lsie_skip - .Lsie_entry
 #endif
-
 	.section .rodata, "a"
 #define SYSCALL(esame,emu)	.long esame
 	.globl	sys_call_table
diff --git a/arch/s390/kernel/module.c b/arch/s390/kernel/module.c
index b7abfad..1fc6d1f 100644
--- a/arch/s390/kernel/module.c
+++ b/arch/s390/kernel/module.c
@@ -19,6 +19,8 @@
 #include <linux/moduleloader.h>
 #include <linux/bug.h>
 #include <asm/alternative.h>
+#include <asm/nospec-branch.h>
+#include <asm/facility.h>
 
 #if 0
 #define DEBUGP printk
@@ -156,7 +158,11 @@ int module_frob_arch_sections(Elf_Ehdr *hdr, Elf_Shdr *sechdrs,
 	me->arch.got_offset = me->core_layout.size;
 	me->core_layout.size += me->arch.got_size;
 	me->arch.plt_offset = me->core_layout.size;
-	me->core_layout.size += me->arch.plt_size;
+	if (me->arch.plt_size) {
+		if (IS_ENABLED(CONFIG_EXPOLINE) && !nospec_call_disable)
+			me->arch.plt_size += PLT_ENTRY_SIZE;
+		me->core_layout.size += me->arch.plt_size;
+	}
 	return 0;
 }
 
@@ -310,9 +316,21 @@ static int apply_rela(Elf_Rela *rela, Elf_Addr base, Elf_Sym *symtab,
 			unsigned int *ip;
 			ip = me->core_layout.base + me->arch.plt_offset +
 				info->plt_offset;
-			ip[0] = 0x0d10e310; /* basr 1,0; lg 1,10(1); br 1 */
-			ip[1] = 0x100a0004;
-			ip[2] = 0x07f10000;
+			ip[0] = 0x0d10e310;	/* basr 1,0  */
+			ip[1] = 0x100a0004;	/* lg	1,10(1) */
+			if (IS_ENABLED(CONFIG_EXPOLINE) &&
+			    !nospec_call_disable) {
+				unsigned int *ij;
+				ij = me->core_layout.base +
+					me->arch.plt_offset +
+					me->arch.plt_size - PLT_ENTRY_SIZE;
+				ip[2] = 0xa7f40000 +	/* j __jump_r1 */
+					(unsigned int)(u16)
+					(((unsigned long) ij - 8 -
+					  (unsigned long) ip) / 2);
+			} else {
+				ip[2] = 0x07f10000;	/* br %r1 */
+			}
 			ip[3] = (unsigned int) (val >> 32);
 			ip[4] = (unsigned int) val;
 			info->plt_initialized = 1;
@@ -418,16 +436,42 @@ int module_finalize(const Elf_Ehdr *hdr,
 		    struct module *me)
 {
 	const Elf_Shdr *s;
-	char *secstrings;
+	char *secstrings, *secname;
+	void *aseg;
+
+	if (IS_ENABLED(CONFIG_EXPOLINE) &&
+	    !nospec_call_disable && me->arch.plt_size) {
+		unsigned int *ij;
+
+		ij = me->core_layout.base + me->arch.plt_offset +
+			me->arch.plt_size - PLT_ENTRY_SIZE;
+		if (test_facility(35)) {
+			ij[0] = 0xc6000000;	/* exrl	%r0,.+10	*/
+			ij[1] = 0x0005a7f4;	/* j	.		*/
+			ij[2] = 0x000007f1;	/* br	%r1		*/
+		} else {
+			ij[0] = 0x44000000 | (unsigned int)
+				offsetof(struct lowcore, br_r1_trampoline);
+			ij[1] = 0xa7f40000;	/* j	.		*/
+		}
+	}
 
 	secstrings = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset;
 	for (s = sechdrs; s < sechdrs + hdr->e_shnum; s++) {
-		if (!strcmp(".altinstructions", secstrings + s->sh_name)) {
-			/* patch .altinstructions */
-			void *aseg = (void *)s->sh_addr;
+		aseg = (void *) s->sh_addr;
+		secname = secstrings + s->sh_name;
 
+		if (!strcmp(".altinstructions", secname))
+			/* patch .altinstructions */
 			apply_alternatives(aseg, aseg + s->sh_size);
-		}
+
+		if (IS_ENABLED(CONFIG_EXPOLINE) &&
+		    (!strcmp(".nospec_call_table", secname)))
+			nospec_call_revert(aseg, aseg + s->sh_size);
+
+		if (IS_ENABLED(CONFIG_EXPOLINE) &&
+		    (!strcmp(".nospec_return_table", secname)))
+			nospec_return_revert(aseg, aseg + s->sh_size);
 	}
 
 	jump_label_apply_nops(me);
diff --git a/arch/s390/kernel/nospec-branch.c b/arch/s390/kernel/nospec-branch.c
new file mode 100644
index 0000000..69d7fcf
--- /dev/null
+++ b/arch/s390/kernel/nospec-branch.c
@@ -0,0 +1,100 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/module.h>
+#include <asm/nospec-branch.h>
+
+int nospec_call_disable = IS_ENABLED(EXPOLINE_OFF);
+int nospec_return_disable = !IS_ENABLED(EXPOLINE_FULL);
+
+static int __init nospectre_v2_setup_early(char *str)
+{
+	nospec_call_disable = 1;
+	nospec_return_disable = 1;
+	return 0;
+}
+early_param("nospectre_v2", nospectre_v2_setup_early);
+
+static int __init spectre_v2_setup_early(char *str)
+{
+	if (str && !strncmp(str, "on", 2)) {
+		nospec_call_disable = 0;
+		nospec_return_disable = 0;
+	}
+	if (str && !strncmp(str, "off", 3)) {
+		nospec_call_disable = 1;
+		nospec_return_disable = 1;
+	}
+	if (str && !strncmp(str, "auto", 4)) {
+		nospec_call_disable = 0;
+		nospec_return_disable = 1;
+	}
+	return 0;
+}
+early_param("spectre_v2", spectre_v2_setup_early);
+
+static void __init_or_module __nospec_revert(s32 *start, s32 *end)
+{
+	enum { BRCL_EXPOLINE, BRASL_EXPOLINE } type;
+	u8 *instr, *thunk, *br;
+	u8 insnbuf[6];
+	s32 *epo;
+
+	/* Second part of the instruction replace is always a nop */
+	memcpy(insnbuf + 2, (char[]) { 0x47, 0x00, 0x00, 0x00 }, 4);
+	for (epo = start; epo < end; epo++) {
+		instr = (u8 *) epo + *epo;
+		if (instr[0] == 0xc0 && (instr[1] & 0x0f) == 0x04)
+			type = BRCL_EXPOLINE;	/* brcl instruction */
+		else if (instr[0] == 0xc0 && (instr[1] & 0x0f) == 0x05)
+			type = BRASL_EXPOLINE;	/* brasl instruction */
+		else
+			continue;
+		thunk = instr + (*(int *)(instr + 2)) * 2;
+		if (thunk[0] == 0xc6 && thunk[1] == 0x00)
+			/* exrl %r0,<target-br> */
+			br = thunk + (*(int *)(thunk + 2)) * 2;
+		else if (thunk[0] == 0xc0 && (thunk[1] & 0x0f) == 0x00 &&
+			 thunk[6] == 0x44 && thunk[7] == 0x00 &&
+			 (thunk[8] & 0x0f) == 0x00 && thunk[9] == 0x00 &&
+			 (thunk[1] & 0xf0) == (thunk[8] & 0xf0))
+			/* larl %rx,<target br> + ex %r0,0(%rx) */
+			br = thunk + (*(int *)(thunk + 2)) * 2;
+		else
+			continue;
+		if (br[0] != 0x07 || (br[1] & 0xf0) != 0xf0)
+			continue;
+		switch (type) {
+		case BRCL_EXPOLINE:
+			/* brcl to thunk, replace with br + nop */
+			insnbuf[0] = br[0];
+			insnbuf[1] = (instr[1] & 0xf0) | (br[1] & 0x0f);
+			break;
+		case BRASL_EXPOLINE:
+			/* brasl to thunk, replace with basr + nop */
+			insnbuf[0] = 0x0d;
+			insnbuf[1] = (instr[1] & 0xf0) | (br[1] & 0x0f);
+			break;
+		}
+
+		s390_kernel_write(instr, insnbuf, 6);
+	}
+}
+
+void __init_or_module nospec_call_revert(s32 *start, s32 *end)
+{
+	if (nospec_call_disable)
+		__nospec_revert(start, end);
+}
+
+void __init_or_module nospec_return_revert(s32 *start, s32 *end)
+{
+	if (nospec_return_disable)
+		__nospec_revert(start, end);
+}
+
+extern s32 __nospec_call_start[], __nospec_call_end[];
+extern s32 __nospec_return_start[], __nospec_return_end[];
+void __init nospec_init_branches(void)
+{
+	nospec_call_revert(__nospec_call_start, __nospec_call_end);
+	nospec_return_revert(__nospec_return_start, __nospec_return_end);
+}
diff --git a/arch/s390/kernel/setup.c b/arch/s390/kernel/setup.c
index bcd2a4a..a6a91f0 100644
--- a/arch/s390/kernel/setup.c
+++ b/arch/s390/kernel/setup.c
@@ -68,6 +68,7 @@
 #include <asm/sysinfo.h>
 #include <asm/numa.h>
 #include <asm/alternative.h>
+#include <asm/nospec-branch.h>
 #include "entry.h"
 
 /*
@@ -379,6 +380,7 @@ static void __init setup_lowcore(void)
 	lc->spinlock_index = 0;
 	arch_spin_lock_setup(0);
 #endif
+	lc->br_r1_trampoline = 0x07f1;	/* br %r1 */
 
 	set_prefix((u32)(unsigned long) lc);
 	lowcore_ptr[0] = lc;
@@ -954,6 +956,8 @@ void __init setup_arch(char **cmdline_p)
 	set_preferred_console();
 
 	apply_alternative_instructions();
+	if (IS_ENABLED(CONFIG_EXPOLINE))
+		nospec_init_branches();
 
 	/* Setup zfcpdump support */
 	setup_zfcpdump();
diff --git a/arch/s390/kernel/smp.c b/arch/s390/kernel/smp.c
index 2fd7d60..a4a9fe1 100644
--- a/arch/s390/kernel/smp.c
+++ b/arch/s390/kernel/smp.c
@@ -214,6 +214,7 @@ static int pcpu_alloc_lowcore(struct pcpu *pcpu, int cpu)
 	lc->cpu_nr = cpu;
 	lc->spinlock_lockval = arch_spin_lockval(cpu);
 	lc->spinlock_index = 0;
+	lc->br_r1_trampoline = 0x07f1;	/* br %r1 */
 	if (nmi_alloc_per_cpu(lc))
 		goto out;
 	if (vdso_alloc_per_cpu(lc))
diff --git a/arch/s390/kernel/vmlinux.lds.S b/arch/s390/kernel/vmlinux.lds.S
index 608cf29..08d12cf 100644
--- a/arch/s390/kernel/vmlinux.lds.S
+++ b/arch/s390/kernel/vmlinux.lds.S
@@ -123,6 +123,20 @@ SECTIONS
 		*(.altinstr_replacement)
 	}
 
+	/*
+	 * Table with the patch locations to undo expolines
+	*/
+	.nospec_call_table : {
+		__nospec_call_start = . ;
+		*(.s390_indirect*)
+		__nospec_call_end = . ;
+	}
+	.nospec_return_table : {
+		__nospec_return_start = . ;
+		*(.s390_return*)
+		__nospec_return_end = . ;
+	}
+
 	/* early.c uses stsi, which requires page aligned data. */
 	. = ALIGN(PAGE_SIZE);
 	INIT_DATA_SECTION(0x100)
diff --git a/drivers/s390/char/Makefile b/drivers/s390/char/Makefile
index 614b44e..a2b33a2 100644
--- a/drivers/s390/char/Makefile
+++ b/drivers/s390/char/Makefile
@@ -19,6 +19,8 @@ endif
 
 CFLAGS_sclp_early_core.o		+= -D__NO_FORTIFY
 
+CFLAGS_REMOVE_sclp_early_core.o	+= $(CC_FLAGS_EXPOLINE)
+
 obj-y += ctrlchar.o keyboard.o defkeymap.o sclp.o sclp_rw.o sclp_quiesce.o \
 	 sclp_cmd.o sclp_config.o sclp_cpi_sys.o sclp_ocf.o sclp_ctl.o \
 	 sclp_early.o sclp_early_core.o
-- 
2.7.4

Re: [PATCH 4/6] s390: add options to change branch prediction behaviour for the kernel

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2410 bytes --]

On Wed 2018-02-07 08:00:09, Martin Schwidefsky wrote:
> Add the PPA instruction to the system entry and exit path to switch
> the kernel to a different branch prediction behaviour. The instructions
> are added via CPU alternatives and can be disabled with the "nospec"
> or the "nobp=0" kernel parameter. If the default behaviour selected
> with CONFIG_KERNEL_NOBP is set to "n" then the "nobp=1" parameter can be
> used to enable the changed kernel branch prediction.
> 
> Acked-by: Cornelia Huck <cohuck@redhat.com>
> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
> ---
>  arch/s390/Kconfig                 | 17 ++++++++++++++
>  arch/s390/include/asm/processor.h |  1 +
>  arch/s390/kernel/alternative.c    | 23 +++++++++++++++++++
>  arch/s390/kernel/early.c          |  2 ++
>  arch/s390/kernel/entry.S          | 48 +++++++++++++++++++++++++++++++++++++++
>  arch/s390/kernel/ipl.c            |  1 +
>  arch/s390/kernel/smp.c            |  2 ++
>  7 files changed, 94 insertions(+)
> 
> diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
> index 0105ce2..d514e25 100644
> --- a/arch/s390/Kconfig
> +++ b/arch/s390/Kconfig
> @@ -540,6 +540,23 @@ config ARCH_RANDOM
>  
>  	  If unsure, say Y.
>  
> +config KERNEL_NOBP
> +	def_bool n
> +	prompt "Enable modified branch prediction for the kernel by default"
> +	help
> +	  If this option is selected the kernel will switch to a modified
> +	  branch prediction mode if the firmware interface is available.
> +	  The modified branch prediction mode improves the behaviour in
> +	  regard to speculative execution.
> +
> +	  With the option enabled the kernel parameter "nobp=0" or "nospec"
> +	  can be used to run the kernel in the normal branch prediction mode.
> +
> +	  With the option disabled the modified branch prediction mode is
> +	  enabled with the "nobp=1" kernel parameter.
> +
> +	  If unsure, say N.
> +

This could use some improvement.

Afaict the config option only changes the default behaviour? Do we
need the option in such case? (CONFIG_CMDLINE_APPEND can be useful to
avoid some options).

Plus, it is not clear what "modified branch prediction mode is" and if
"improves behaviour" means "faster" or "safer".

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [PATCH 6/6] s390: introduce execute-trampolines for branches

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 23264 bytes --]

On Wed 2018-02-07 08:00:11, Martin Schwidefsky wrote:
> Add CONFIG_EXPOLINE to enable the use of the new -mindirect-branch= and
> -mfunction_return= compiler options to create a kernel fortified against
> the specte v2 attack.
> 
> With CONFIG_EXPOLINE=y all indirect branches will be issued with an
> execute type instruction. For z10 or newer the EXRL instruction will
> be used, for older machines the EX instruction. The typical indirect
> call
> 
> 	basr	%r14,%r1
> 
> is replaced with a PC relative call to a new thunk
> 
> 	brasl	%r14,__s390x_indirect_jump_r1
> 
> The thunk contains the EXRL/EX instruction to the indirect branch
> 
> __s390x_indirect_jump_r1:
> 	exrl	0,0f
> 	j	.
> 0:	br	%r1
> 
> The detour via the execute type instruction has a performance impact.
> To get rid of the detour the new kernel parameter "nospectre_v2" and
> "spectre_v2=[on,off,auto]" can be used. If the parameter is specified
> the kernel and module code will be patched at runtime.

This is really unfortunate naming of kernel option.

spectre_v2=off sounds like we are turning the "bug" off, but i somehow
suspect you are turning the bug _workaround_ off.

									Pavel

> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
> ---
>  arch/s390/Kconfig                     |  28 +++++++++
>  arch/s390/Makefile                    |  12 ++++
>  arch/s390/include/asm/lowcore.h       |   6 +-
>  arch/s390/include/asm/nospec-branch.h |  18 ++++++
>  arch/s390/kernel/Makefile             |   4 ++
>  arch/s390/kernel/entry.S              | 113 ++++++++++++++++++++++++++--------
>  arch/s390/kernel/module.c             |  62 ++++++++++++++++---
>  arch/s390/kernel/nospec-branch.c      | 100 ++++++++++++++++++++++++++++++
>  arch/s390/kernel/setup.c              |   4 ++
>  arch/s390/kernel/smp.c                |   1 +
>  arch/s390/kernel/vmlinux.lds.S        |  14 +++++
>  drivers/s390/char/Makefile            |   2 +
>  12 files changed, 329 insertions(+), 35 deletions(-)
>  create mode 100644 arch/s390/include/asm/nospec-branch.h
>  create mode 100644 arch/s390/kernel/nospec-branch.c
> 
> diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
> index d514e25..d4a65bf 100644
> --- a/arch/s390/Kconfig
> +++ b/arch/s390/Kconfig
> @@ -557,6 +557,34 @@ config KERNEL_NOBP
>  
>  	  If unsure, say N.
>  
> +config EXPOLINE
> +	def_bool n
> +	prompt "Avoid speculative indirect branches in the kernel"
> +	help
> +	  Compile the kernel with the expoline compiler options to guard
> +	  against kernel-to-user data leaks by avoiding speculative indirect
> +	  branches.
> +	  Requires a compiler with -mindirect-branch=thunk support for full
> +	  protection. The kernel may run slower.
> +
> +	  If unsure, say N.
> +
> +choice
> +	prompt "Expoline default"
> +	depends on EXPOLINE
> +	default EXPOLINE_FULL
> +
> +config EXPOLINE_OFF
> +	bool "spectre_v2=off"
> +
> +config EXPOLINE_MEDIUM
> +	bool "spectre_v2=auto"
> +
> +config EXPOLINE_FULL
> +	bool "spectre_v2=on"
> +
> +endchoice
> +
>  endmenu
>  
>  menu "Memory setup"
> diff --git a/arch/s390/Makefile b/arch/s390/Makefile
> index fd691c4..2f925ef 100644
> --- a/arch/s390/Makefile
> +++ b/arch/s390/Makefile
> @@ -78,6 +78,18 @@ ifeq ($(call cc-option-yn,-mwarn-dynamicstack),y)
>  cflags-$(CONFIG_WARN_DYNAMIC_STACK) += -mwarn-dynamicstack
>  endif
>  
> +ifdef CONFIG_EXPOLINE
> +  ifeq ($(call cc-option-yn,$(CC_FLAGS_MARCH) -mindirect-branch=thunk),y)
> +    CC_FLAGS_EXPOLINE := -mindirect-branch=thunk
> +    CC_FLAGS_EXPOLINE += -mfunction-return=thunk
> +    CC_FLAGS_EXPOLINE += -mindirect-branch-table
> +    export CC_FLAGS_EXPOLINE
> +    cflags-y += $(CC_FLAGS_EXPOLINE)
> +  else
> +    $(warning "Your gcc lacks the -mindirect-branch= option")
> +  endif
> +endif
> +
>  ifdef CONFIG_FUNCTION_TRACER
>  # make use of hotpatch feature if the compiler supports it
>  cc_hotpatch	:= -mhotpatch=0,3
> diff --git a/arch/s390/include/asm/lowcore.h b/arch/s390/include/asm/lowcore.h
> index c63986a..5bc8888 100644
> --- a/arch/s390/include/asm/lowcore.h
> +++ b/arch/s390/include/asm/lowcore.h
> @@ -136,7 +136,11 @@ struct lowcore {
>  	__u64	vdso_per_cpu_data;		/* 0x03b8 */
>  	__u64	machine_flags;			/* 0x03c0 */
>  	__u64	gmap;				/* 0x03c8 */
> -	__u8	pad_0x03d0[0x0e00-0x03d0];	/* 0x03d0 */
> +	__u8	pad_0x03d0[0x0400-0x03d0];	/* 0x03d0 */
> +
> +	/* br %r1 trampoline */
> +	__u16	br_r1_trampoline;		/* 0x0400 */
> +	__u8	pad_0x0402[0x0e00-0x0402];	/* 0x0402 */
>  
>  	/*
>  	 * 0xe00 contains the address of the IPL Parameter Information
> diff --git a/arch/s390/include/asm/nospec-branch.h b/arch/s390/include/asm/nospec-branch.h
> new file mode 100644
> index 0000000..7df48e5
> --- /dev/null
> +++ b/arch/s390/include/asm/nospec-branch.h
> @@ -0,0 +1,18 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +#ifndef _ASM_S390_EXPOLINE_H
> +#define _ASM_S390_EXPOLINE_H
> +
> +#ifndef __ASSEMBLY__
> +
> +#include <linux/types.h>
> +
> +extern int nospec_call_disable;
> +extern int nospec_return_disable;
> +
> +void nospec_init_branches(void);
> +void nospec_call_revert(s32 *start, s32 *end);
> +void nospec_return_revert(s32 *start, s32 *end);
> +
> +#endif /* __ASSEMBLY__ */
> +
> +#endif /* _ASM_S390_EXPOLINE_H */
> diff --git a/arch/s390/kernel/Makefile b/arch/s390/kernel/Makefile
> index 909bce6..7f27e3d 100644
> --- a/arch/s390/kernel/Makefile
> +++ b/arch/s390/kernel/Makefile
> @@ -29,6 +29,7 @@ UBSAN_SANITIZE_early.o	:= n
>  #
>  ifneq ($(CC_FLAGS_MARCH),-march=z900)
>  CFLAGS_REMOVE_als.o	+= $(CC_FLAGS_MARCH)
> +CFLAGS_REMOVE_als.o	+= $(CC_FLAGS_EXPOLINE)
>  CFLAGS_als.o		+= -march=z900
>  AFLAGS_REMOVE_head.o	+= $(CC_FLAGS_MARCH)
>  AFLAGS_head.o		+= -march=z900
> @@ -63,6 +64,9 @@ obj-y	+= entry.o reipl.o relocate_kernel.o kdebugfs.o alternative.o
>  
>  extra-y				+= head.o head64.o vmlinux.lds
>  
> +obj-$(CONFIG_EXPOLINE)		+= nospec-branch.o
> +CFLAGS_REMOVE_expoline.o	+= $(CC_FLAGS_EXPOLINE)
> +
>  obj-$(CONFIG_MODULES)		+= module.o
>  obj-$(CONFIG_SMP)		+= smp.o
>  obj-$(CONFIG_SCHED_TOPOLOGY)	+= topology.o
> diff --git a/arch/s390/kernel/entry.S b/arch/s390/kernel/entry.S
> index 53145b5..13a133a 100644
> --- a/arch/s390/kernel/entry.S
> +++ b/arch/s390/kernel/entry.S
> @@ -222,6 +222,68 @@ _PIF_WORK	= (_PIF_PER_TRAP | _PIF_SYSCALL_RESTART)
>  	.popsection
>  	.endm
>  
> +#ifdef CONFIG_EXPOLINE
> +
> +	.macro GEN_BR_THUNK name,reg,tmp
> +	.section .text.\name,"axG",@progbits,\name,comdat
> +	.globl \name
> +	.hidden \name
> +	.type \name,@function
> +\name:
> +	.cfi_startproc
> +#ifdef CONFIG_HAVE_MARCH_Z10_FEATURES
> +	exrl	0,0f
> +#else
> +	larl	\tmp,0f
> +	ex	0,0(\tmp)
> +#endif
> +	j	.
> +0:	br	\reg
> +	.cfi_endproc
> +	.endm
> +
> +	GEN_BR_THUNK __s390x_indirect_jump_r1use_r9,%r9,%r1
> +	GEN_BR_THUNK __s390x_indirect_jump_r1use_r14,%r14,%r1
> +	GEN_BR_THUNK __s390x_indirect_jump_r11use_r14,%r14,%r11
> +
> +	.macro BASR_R14_R9
> +0:	brasl	%r14,__s390x_indirect_jump_r1use_r9
> +	.pushsection .s390_indirect_branches,"a",@progbits
> +	.long	0b-.
> +	.popsection
> +	.endm
> +
> +	.macro BR_R1USE_R14
> +0:	jg	__s390x_indirect_jump_r1use_r14
> +	.pushsection .s390_indirect_branches,"a",@progbits
> +	.long	0b-.
> +	.popsection
> +	.endm
> +
> +	.macro BR_R11USE_R14
> +0:	jg	__s390x_indirect_jump_r11use_r14
> +	.pushsection .s390_indirect_branches,"a",@progbits
> +	.long	0b-.
> +	.popsection
> +	.endm
> +
> +#else	/* CONFIG_EXPOLINE */
> +
> +	.macro BASR_R14_R9
> +	basr	%r14,%r9
> +	.endm
> +
> +	.macro BR_R1USE_R14
> +	br	%r14
> +	.endm
> +
> +	.macro BR_R11USE_R14
> +	br	%r14
> +	.endm
> +
> +#endif /* CONFIG_EXPOLINE */
> +
> +
>  	.section .kprobes.text, "ax"
>  .Ldummy:
>  	/*
> @@ -237,7 +299,7 @@ _PIF_WORK	= (_PIF_PER_TRAP | _PIF_SYSCALL_RESTART)
>  ENTRY(__bpon)
>  	.globl __bpon
>  	BPON
> -	br	%r14
> +	BR_R1USE_R14
>  
>  /*
>   * Scheduler resume function, called by switch_to
> @@ -261,9 +323,9 @@ ENTRY(__switch_to)
>  	mvc	__LC_CURRENT_PID(4,%r0),0(%r3)	# store pid of next
>  	lmg	%r6,%r15,__SF_GPRS(%r15)	# load gprs of next task
>  	TSTMSK	__LC_MACHINE_FLAGS,MACHINE_FLAG_LPP
> -	bzr	%r14
> +	jz	0f
>  	.insn	s,0xb2800000,__LC_LPP		# set program parameter
> -	br	%r14
> +0:	BR_R1USE_R14
>  
>  .L__critical_start:
>  
> @@ -330,7 +392,7 @@ sie_exit:
>  	xgr	%r5,%r5
>  	lmg	%r6,%r14,__SF_GPRS(%r15)	# restore kernel registers
>  	lg	%r2,__SF_EMPTY+16(%r15)		# return exit reason code
> -	br	%r14
> +	BR_R1USE_R14
>  .Lsie_fault:
>  	lghi	%r14,-EFAULT
>  	stg	%r14,__SF_EMPTY+16(%r15)	# set exit reason code
> @@ -389,7 +451,7 @@ ENTRY(system_call)
>  	lgf	%r9,0(%r8,%r10)			# get system call add.
>  	TSTMSK	__TI_flags(%r12),_TIF_TRACE
>  	jnz	.Lsysc_tracesys
> -	basr	%r14,%r9			# call sys_xxxx
> +	BASR_R14_R9				# call sys_xxxx
>  	stg	%r2,__PT_R2(%r11)		# store return value
>  
>  .Lsysc_return:
> @@ -574,7 +636,7 @@ ENTRY(system_call)
>  	lmg	%r3,%r7,__PT_R3(%r11)
>  	stg	%r7,STACK_FRAME_OVERHEAD(%r15)
>  	lg	%r2,__PT_ORIG_GPR2(%r11)
> -	basr	%r14,%r9		# call sys_xxx
> +	BASR_R14_R9			# call sys_xxx
>  	stg	%r2,__PT_R2(%r11)	# store return value
>  .Lsysc_tracenogo:
>  	TSTMSK	__TI_flags(%r12),_TIF_TRACE
> @@ -598,7 +660,7 @@ ENTRY(ret_from_fork)
>  	lmg	%r9,%r10,__PT_R9(%r11)	# load gprs
>  ENTRY(kernel_thread_starter)
>  	la	%r2,0(%r10)
> -	basr	%r14,%r9
> +	BASR_R14_R9
>  	j	.Lsysc_tracenogo
>  
>  /*
> @@ -678,9 +740,9 @@ ENTRY(pgm_check_handler)
>  	nill	%r10,0x007f
>  	sll	%r10,2
>  	je	.Lpgm_return
> -	lgf	%r1,0(%r10,%r1)		# load address of handler routine
> +	lgf	%r9,0(%r10,%r1)		# load address of handler routine
>  	lgr	%r2,%r11		# pass pointer to pt_regs
> -	basr	%r14,%r1		# branch to interrupt-handler
> +	BASR_R14_R9			# branch to interrupt-handler
>  .Lpgm_return:
>  	LOCKDEP_SYS_EXIT
>  	tm	__PT_PSW+1(%r11),0x01	# returning to user ?
> @@ -998,7 +1060,7 @@ ENTRY(psw_idle)
>  	stpt	__TIMER_IDLE_ENTER(%r2)
>  .Lpsw_idle_lpsw:
>  	lpswe	__SF_EMPTY(%r15)
> -	br	%r14
> +	BR_R1USE_R14
>  .Lpsw_idle_end:
>  
>  /*
> @@ -1012,7 +1074,7 @@ ENTRY(save_fpu_regs)
>  	lg	%r2,__LC_CURRENT
>  	aghi	%r2,__TASK_thread
>  	TSTMSK	__LC_CPU_FLAGS,_CIF_FPU
> -	bor	%r14
> +	jo	.Lsave_fpu_regs_exit
>  	stfpc	__THREAD_FPU_fpc(%r2)
>  	lg	%r3,__THREAD_FPU_regs(%r2)
>  	TSTMSK	__LC_MACHINE_FLAGS,MACHINE_FLAG_VX
> @@ -1039,7 +1101,8 @@ ENTRY(save_fpu_regs)
>  	std	15,120(%r3)
>  .Lsave_fpu_regs_done:
>  	oi	__LC_CPU_FLAGS+7,_CIF_FPU
> -	br	%r14
> +.Lsave_fpu_regs_exit:
> +	BR_R1USE_R14
>  .Lsave_fpu_regs_end:
>  EXPORT_SYMBOL(save_fpu_regs)
>  
> @@ -1057,7 +1120,7 @@ load_fpu_regs:
>  	lg	%r4,__LC_CURRENT
>  	aghi	%r4,__TASK_thread
>  	TSTMSK	__LC_CPU_FLAGS,_CIF_FPU
> -	bnor	%r14
> +	jno	.Lload_fpu_regs_exit
>  	lfpc	__THREAD_FPU_fpc(%r4)
>  	TSTMSK	__LC_MACHINE_FLAGS,MACHINE_FLAG_VX
>  	lg	%r4,__THREAD_FPU_regs(%r4)	# %r4 <- reg save area
> @@ -1084,7 +1147,8 @@ load_fpu_regs:
>  	ld	15,120(%r4)
>  .Lload_fpu_regs_done:
>  	ni	__LC_CPU_FLAGS+7,255-_CIF_FPU
> -	br	%r14
> +.Lload_fpu_regs_exit:
> +	BR_R1USE_R14
>  .Lload_fpu_regs_end:
>  
>  .L__critical_end:
> @@ -1301,7 +1365,7 @@ cleanup_critical:
>  	jl	0f
>  	clg	%r9,BASED(.Lcleanup_table+104)	# .Lload_fpu_regs_end
>  	jl	.Lcleanup_load_fpu_regs
> -0:	br	%r14
> +0:	BR_R11USE_R14
>  
>  	.align	8
>  .Lcleanup_table:
> @@ -1337,7 +1401,7 @@ cleanup_critical:
>  	ni	__SIE_PROG0C+3(%r9),0xfe	# no longer in SIE
>  	lctlg	%c1,%c1,__LC_USER_ASCE		# load primary asce
>  	larl	%r9,sie_exit			# skip forward to sie_exit
> -	br	%r14
> +	BR_R11USE_R14
>  #endif
>  
>  .Lcleanup_system_call:
> @@ -1390,7 +1454,7 @@ cleanup_critical:
>  	stg	%r15,56(%r11)		# r15 stack pointer
>  	# set new psw address and exit
>  	larl	%r9,.Lsysc_do_svc
> -	br	%r14
> +	BR_R11USE_R14
>  .Lcleanup_system_call_insn:
>  	.quad	system_call
>  	.quad	.Lsysc_stmg
> @@ -1402,7 +1466,7 @@ cleanup_critical:
>  
>  .Lcleanup_sysc_tif:
>  	larl	%r9,.Lsysc_tif
> -	br	%r14
> +	BR_R11USE_R14
>  
>  .Lcleanup_sysc_restore:
>  	# check if stpt has been executed
> @@ -1419,14 +1483,14 @@ cleanup_critical:
>  	mvc	0(64,%r11),__PT_R8(%r9)
>  	lmg	%r0,%r7,__PT_R0(%r9)
>  1:	lmg	%r8,%r9,__LC_RETURN_PSW
> -	br	%r14
> +	BR_R11USE_R14
>  .Lcleanup_sysc_restore_insn:
>  	.quad	.Lsysc_exit_timer
>  	.quad	.Lsysc_done - 4
>  
>  .Lcleanup_io_tif:
>  	larl	%r9,.Lio_tif
> -	br	%r14
> +	BR_R11USE_R14
>  
>  .Lcleanup_io_restore:
>  	# check if stpt has been executed
> @@ -1440,7 +1504,7 @@ cleanup_critical:
>  	mvc	0(64,%r11),__PT_R8(%r9)
>  	lmg	%r0,%r7,__PT_R0(%r9)
>  1:	lmg	%r8,%r9,__LC_RETURN_PSW
> -	br	%r14
> +	BR_R11USE_R14
>  .Lcleanup_io_restore_insn:
>  	.quad	.Lio_exit_timer
>  	.quad	.Lio_done - 4
> @@ -1493,17 +1557,17 @@ cleanup_critical:
>  	# prepare return psw
>  	nihh	%r8,0xfcfd		# clear irq & wait state bits
>  	lg	%r9,48(%r11)		# return from psw_idle
> -	br	%r14
> +	BR_R11USE_R14
>  .Lcleanup_idle_insn:
>  	.quad	.Lpsw_idle_lpsw
>  
>  .Lcleanup_save_fpu_regs:
>  	larl	%r9,save_fpu_regs
> -	br	%r14
> +	BR_R11USE_R14
>  
>  .Lcleanup_load_fpu_regs:
>  	larl	%r9,load_fpu_regs
> -	br	%r14
> +	BR_R11USE_R14
>  
>  /*
>   * Integer constants
> @@ -1523,7 +1587,6 @@ cleanup_critical:
>  .Lsie_crit_mcck_length:
>  	.quad   .Lsie_skip - .Lsie_entry
>  #endif
> -
>  	.section .rodata, "a"
>  #define SYSCALL(esame,emu)	.long esame
>  	.globl	sys_call_table
> diff --git a/arch/s390/kernel/module.c b/arch/s390/kernel/module.c
> index b7abfad..1fc6d1f 100644
> --- a/arch/s390/kernel/module.c
> +++ b/arch/s390/kernel/module.c
> @@ -19,6 +19,8 @@
>  #include <linux/moduleloader.h>
>  #include <linux/bug.h>
>  #include <asm/alternative.h>
> +#include <asm/nospec-branch.h>
> +#include <asm/facility.h>
>  
>  #if 0
>  #define DEBUGP printk
> @@ -156,7 +158,11 @@ int module_frob_arch_sections(Elf_Ehdr *hdr, Elf_Shdr *sechdrs,
>  	me->arch.got_offset = me->core_layout.size;
>  	me->core_layout.size += me->arch.got_size;
>  	me->arch.plt_offset = me->core_layout.size;
> -	me->core_layout.size += me->arch.plt_size;
> +	if (me->arch.plt_size) {
> +		if (IS_ENABLED(CONFIG_EXPOLINE) && !nospec_call_disable)
> +			me->arch.plt_size += PLT_ENTRY_SIZE;
> +		me->core_layout.size += me->arch.plt_size;
> +	}
>  	return 0;
>  }
>  
> @@ -310,9 +316,21 @@ static int apply_rela(Elf_Rela *rela, Elf_Addr base, Elf_Sym *symtab,
>  			unsigned int *ip;
>  			ip = me->core_layout.base + me->arch.plt_offset +
>  				info->plt_offset;
> -			ip[0] = 0x0d10e310; /* basr 1,0; lg 1,10(1); br 1 */
> -			ip[1] = 0x100a0004;
> -			ip[2] = 0x07f10000;
> +			ip[0] = 0x0d10e310;	/* basr 1,0  */
> +			ip[1] = 0x100a0004;	/* lg	1,10(1) */
> +			if (IS_ENABLED(CONFIG_EXPOLINE) &&
> +			    !nospec_call_disable) {
> +				unsigned int *ij;
> +				ij = me->core_layout.base +
> +					me->arch.plt_offset +
> +					me->arch.plt_size - PLT_ENTRY_SIZE;
> +				ip[2] = 0xa7f40000 +	/* j __jump_r1 */
> +					(unsigned int)(u16)
> +					(((unsigned long) ij - 8 -
> +					  (unsigned long) ip) / 2);
> +			} else {
> +				ip[2] = 0x07f10000;	/* br %r1 */
> +			}
>  			ip[3] = (unsigned int) (val >> 32);
>  			ip[4] = (unsigned int) val;
>  			info->plt_initialized = 1;
> @@ -418,16 +436,42 @@ int module_finalize(const Elf_Ehdr *hdr,
>  		    struct module *me)
>  {
>  	const Elf_Shdr *s;
> -	char *secstrings;
> +	char *secstrings, *secname;
> +	void *aseg;
> +
> +	if (IS_ENABLED(CONFIG_EXPOLINE) &&
> +	    !nospec_call_disable && me->arch.plt_size) {
> +		unsigned int *ij;
> +
> +		ij = me->core_layout.base + me->arch.plt_offset +
> +			me->arch.plt_size - PLT_ENTRY_SIZE;
> +		if (test_facility(35)) {
> +			ij[0] = 0xc6000000;	/* exrl	%r0,.+10	*/
> +			ij[1] = 0x0005a7f4;	/* j	.		*/
> +			ij[2] = 0x000007f1;	/* br	%r1		*/
> +		} else {
> +			ij[0] = 0x44000000 | (unsigned int)
> +				offsetof(struct lowcore, br_r1_trampoline);
> +			ij[1] = 0xa7f40000;	/* j	.		*/
> +		}
> +	}
>  
>  	secstrings = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset;
>  	for (s = sechdrs; s < sechdrs + hdr->e_shnum; s++) {
> -		if (!strcmp(".altinstructions", secstrings + s->sh_name)) {
> -			/* patch .altinstructions */
> -			void *aseg = (void *)s->sh_addr;
> +		aseg = (void *) s->sh_addr;
> +		secname = secstrings + s->sh_name;
>  
> +		if (!strcmp(".altinstructions", secname))
> +			/* patch .altinstructions */
>  			apply_alternatives(aseg, aseg + s->sh_size);
> -		}
> +
> +		if (IS_ENABLED(CONFIG_EXPOLINE) &&
> +		    (!strcmp(".nospec_call_table", secname)))
> +			nospec_call_revert(aseg, aseg + s->sh_size);
> +
> +		if (IS_ENABLED(CONFIG_EXPOLINE) &&
> +		    (!strcmp(".nospec_return_table", secname)))
> +			nospec_return_revert(aseg, aseg + s->sh_size);
>  	}
>  
>  	jump_label_apply_nops(me);
> diff --git a/arch/s390/kernel/nospec-branch.c b/arch/s390/kernel/nospec-branch.c
> new file mode 100644
> index 0000000..69d7fcf
> --- /dev/null
> +++ b/arch/s390/kernel/nospec-branch.c
> @@ -0,0 +1,100 @@
> +// SPDX-License-Identifier: GPL-2.0
> +#include <linux/module.h>
> +#include <asm/nospec-branch.h>
> +
> +int nospec_call_disable = IS_ENABLED(EXPOLINE_OFF);
> +int nospec_return_disable = !IS_ENABLED(EXPOLINE_FULL);
> +
> +static int __init nospectre_v2_setup_early(char *str)
> +{
> +	nospec_call_disable = 1;
> +	nospec_return_disable = 1;
> +	return 0;
> +}
> +early_param("nospectre_v2", nospectre_v2_setup_early);
> +
> +static int __init spectre_v2_setup_early(char *str)
> +{
> +	if (str && !strncmp(str, "on", 2)) {
> +		nospec_call_disable = 0;
> +		nospec_return_disable = 0;
> +	}
> +	if (str && !strncmp(str, "off", 3)) {
> +		nospec_call_disable = 1;
> +		nospec_return_disable = 1;
> +	}
> +	if (str && !strncmp(str, "auto", 4)) {
> +		nospec_call_disable = 0;
> +		nospec_return_disable = 1;
> +	}
> +	return 0;
> +}
> +early_param("spectre_v2", spectre_v2_setup_early);
> +
> +static void __init_or_module __nospec_revert(s32 *start, s32 *end)
> +{
> +	enum { BRCL_EXPOLINE, BRASL_EXPOLINE } type;
> +	u8 *instr, *thunk, *br;
> +	u8 insnbuf[6];
> +	s32 *epo;
> +
> +	/* Second part of the instruction replace is always a nop */
> +	memcpy(insnbuf + 2, (char[]) { 0x47, 0x00, 0x00, 0x00 }, 4);
> +	for (epo = start; epo < end; epo++) {
> +		instr = (u8 *) epo + *epo;
> +		if (instr[0] == 0xc0 && (instr[1] & 0x0f) == 0x04)
> +			type = BRCL_EXPOLINE;	/* brcl instruction */
> +		else if (instr[0] == 0xc0 && (instr[1] & 0x0f) == 0x05)
> +			type = BRASL_EXPOLINE;	/* brasl instruction */
> +		else
> +			continue;
> +		thunk = instr + (*(int *)(instr + 2)) * 2;
> +		if (thunk[0] == 0xc6 && thunk[1] == 0x00)
> +			/* exrl %r0,<target-br> */
> +			br = thunk + (*(int *)(thunk + 2)) * 2;
> +		else if (thunk[0] == 0xc0 && (thunk[1] & 0x0f) == 0x00 &&
> +			 thunk[6] == 0x44 && thunk[7] == 0x00 &&
> +			 (thunk[8] & 0x0f) == 0x00 && thunk[9] == 0x00 &&
> +			 (thunk[1] & 0xf0) == (thunk[8] & 0xf0))
> +			/* larl %rx,<target br> + ex %r0,0(%rx) */
> +			br = thunk + (*(int *)(thunk + 2)) * 2;
> +		else
> +			continue;
> +		if (br[0] != 0x07 || (br[1] & 0xf0) != 0xf0)
> +			continue;
> +		switch (type) {
> +		case BRCL_EXPOLINE:
> +			/* brcl to thunk, replace with br + nop */
> +			insnbuf[0] = br[0];
> +			insnbuf[1] = (instr[1] & 0xf0) | (br[1] & 0x0f);
> +			break;
> +		case BRASL_EXPOLINE:
> +			/* brasl to thunk, replace with basr + nop */
> +			insnbuf[0] = 0x0d;
> +			insnbuf[1] = (instr[1] & 0xf0) | (br[1] & 0x0f);
> +			break;
> +		}
> +
> +		s390_kernel_write(instr, insnbuf, 6);
> +	}
> +}
> +
> +void __init_or_module nospec_call_revert(s32 *start, s32 *end)
> +{
> +	if (nospec_call_disable)
> +		__nospec_revert(start, end);
> +}
> +
> +void __init_or_module nospec_return_revert(s32 *start, s32 *end)
> +{
> +	if (nospec_return_disable)
> +		__nospec_revert(start, end);
> +}
> +
> +extern s32 __nospec_call_start[], __nospec_call_end[];
> +extern s32 __nospec_return_start[], __nospec_return_end[];
> +void __init nospec_init_branches(void)
> +{
> +	nospec_call_revert(__nospec_call_start, __nospec_call_end);
> +	nospec_return_revert(__nospec_return_start, __nospec_return_end);
> +}
> diff --git a/arch/s390/kernel/setup.c b/arch/s390/kernel/setup.c
> index bcd2a4a..a6a91f0 100644
> --- a/arch/s390/kernel/setup.c
> +++ b/arch/s390/kernel/setup.c
> @@ -68,6 +68,7 @@
>  #include <asm/sysinfo.h>
>  #include <asm/numa.h>
>  #include <asm/alternative.h>
> +#include <asm/nospec-branch.h>
>  #include "entry.h"
>  
>  /*
> @@ -379,6 +380,7 @@ static void __init setup_lowcore(void)
>  	lc->spinlock_index = 0;
>  	arch_spin_lock_setup(0);
>  #endif
> +	lc->br_r1_trampoline = 0x07f1;	/* br %r1 */
>  
>  	set_prefix((u32)(unsigned long) lc);
>  	lowcore_ptr[0] = lc;
> @@ -954,6 +956,8 @@ void __init setup_arch(char **cmdline_p)
>  	set_preferred_console();
>  
>  	apply_alternative_instructions();
> +	if (IS_ENABLED(CONFIG_EXPOLINE))
> +		nospec_init_branches();
>  
>  	/* Setup zfcpdump support */
>  	setup_zfcpdump();
> diff --git a/arch/s390/kernel/smp.c b/arch/s390/kernel/smp.c
> index 2fd7d60..a4a9fe1 100644
> --- a/arch/s390/kernel/smp.c
> +++ b/arch/s390/kernel/smp.c
> @@ -214,6 +214,7 @@ static int pcpu_alloc_lowcore(struct pcpu *pcpu, int cpu)
>  	lc->cpu_nr = cpu;
>  	lc->spinlock_lockval = arch_spin_lockval(cpu);
>  	lc->spinlock_index = 0;
> +	lc->br_r1_trampoline = 0x07f1;	/* br %r1 */
>  	if (nmi_alloc_per_cpu(lc))
>  		goto out;
>  	if (vdso_alloc_per_cpu(lc))
> diff --git a/arch/s390/kernel/vmlinux.lds.S b/arch/s390/kernel/vmlinux.lds.S
> index 608cf29..08d12cf 100644
> --- a/arch/s390/kernel/vmlinux.lds.S
> +++ b/arch/s390/kernel/vmlinux.lds.S
> @@ -123,6 +123,20 @@ SECTIONS
>  		*(.altinstr_replacement)
>  	}
>  
> +	/*
> +	 * Table with the patch locations to undo expolines
> +	*/
> +	.nospec_call_table : {
> +		__nospec_call_start = . ;
> +		*(.s390_indirect*)
> +		__nospec_call_end = . ;
> +	}
> +	.nospec_return_table : {
> +		__nospec_return_start = . ;
> +		*(.s390_return*)
> +		__nospec_return_end = . ;
> +	}
> +
>  	/* early.c uses stsi, which requires page aligned data. */
>  	. = ALIGN(PAGE_SIZE);
>  	INIT_DATA_SECTION(0x100)
> diff --git a/drivers/s390/char/Makefile b/drivers/s390/char/Makefile
> index 614b44e..a2b33a2 100644
> --- a/drivers/s390/char/Makefile
> +++ b/drivers/s390/char/Makefile
> @@ -19,6 +19,8 @@ endif
>  
>  CFLAGS_sclp_early_core.o		+= -D__NO_FORTIFY
>  
> +CFLAGS_REMOVE_sclp_early_core.o	+= $(CC_FLAGS_EXPOLINE)
> +
>  obj-y += ctrlchar.o keyboard.o defkeymap.o sclp.o sclp_rw.o sclp_quiesce.o \
>  	 sclp_cmd.o sclp_config.o sclp_cpi_sys.o sclp_ocf.o sclp_ctl.o \
>  	 sclp_early.o sclp_early_core.o

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [PATCH 4/6] s390: add options to change branch prediction behaviour for the kernel

[Martin Schwidefsky]

On Wed, 7 Feb 2018 11:02:52 +0100
Pavel Machek <pavel@ucw.cz> wrote:

> On Wed 2018-02-07 08:00:09, Martin Schwidefsky wrote:
> > Add the PPA instruction to the system entry and exit path to switch
> > the kernel to a different branch prediction behaviour. The instructions
> > are added via CPU alternatives and can be disabled with the "nospec"
> > or the "nobp=0" kernel parameter. If the default behaviour selected
> > with CONFIG_KERNEL_NOBP is set to "n" then the "nobp=1" parameter can be
> > used to enable the changed kernel branch prediction.
> > 
> > Acked-by: Cornelia Huck <cohuck@redhat.com>
> > Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
> > ---
> >  arch/s390/Kconfig                 | 17 ++++++++++++++
> >  arch/s390/include/asm/processor.h |  1 +
> >  arch/s390/kernel/alternative.c    | 23 +++++++++++++++++++
> >  arch/s390/kernel/early.c          |  2 ++
> >  arch/s390/kernel/entry.S          | 48 +++++++++++++++++++++++++++++++++++++++
> >  arch/s390/kernel/ipl.c            |  1 +
> >  arch/s390/kernel/smp.c            |  2 ++
> >  7 files changed, 94 insertions(+)
> > 
> > diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
> > index 0105ce2..d514e25 100644
> > --- a/arch/s390/Kconfig
> > +++ b/arch/s390/Kconfig
> > @@ -540,6 +540,23 @@ config ARCH_RANDOM
> >  
> >  	  If unsure, say Y.
> >  
> > +config KERNEL_NOBP
> > +	def_bool n
> > +	prompt "Enable modified branch prediction for the kernel by default"
> > +	help
> > +	  If this option is selected the kernel will switch to a modified
> > +	  branch prediction mode if the firmware interface is available.
> > +	  The modified branch prediction mode improves the behaviour in
> > +	  regard to speculative execution.
> > +
> > +	  With the option enabled the kernel parameter "nobp=0" or "nospec"
> > +	  can be used to run the kernel in the normal branch prediction mode.
> > +
> > +	  With the option disabled the modified branch prediction mode is
> > +	  enabled with the "nobp=1" kernel parameter.
> > +
> > +	  If unsure, say N.
> > +  
> 
> This could use some improvement.
> 
> Afaict the config option only changes the default behaviour? Do we
> need the option in such case? (CONFIG_CMDLINE_APPEND can be useful to
> avoid some options).

# git grep CMDLINE_APPEND

returns nothing. What are you referring to?

The idea of this config option is that a distributor can decide what
default behaviour the system should have if no command line parameter
is specified.
 
> Plus, it is not clear what "modified branch prediction mode is" and if
> "improves behaviour" means "faster" or "safer".

Naturally "improves behaviour in regard to speculative execution" is
indicating that the system is safer. In general that means slower as well
but as always it depends. What exactly is done is up to the machine.

-- 
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.

Re: [PATCH 6/6] s390: introduce execute-trampolines for branches

[Martin Schwidefsky]

On Wed, 7 Feb 2018 11:07:26 +0100
Pavel Machek <pavel@ucw.cz> wrote:

> On Wed 2018-02-07 08:00:11, Martin Schwidefsky wrote:
> > Add CONFIG_EXPOLINE to enable the use of the new -mindirect-branch= and
> > -mfunction_return= compiler options to create a kernel fortified against
> > the specte v2 attack.
> > 
> > With CONFIG_EXPOLINE=y all indirect branches will be issued with an
> > execute type instruction. For z10 or newer the EXRL instruction will
> > be used, for older machines the EX instruction. The typical indirect
> > call
> > 
> > 	basr	%r14,%r1
> > 
> > is replaced with a PC relative call to a new thunk
> > 
> > 	brasl	%r14,__s390x_indirect_jump_r1
> > 
> > The thunk contains the EXRL/EX instruction to the indirect branch
> > 
> > __s390x_indirect_jump_r1:
> > 	exrl	0,0f
> > 	j	.
> > 0:	br	%r1
> > 
> > The detour via the execute type instruction has a performance impact.
> > To get rid of the detour the new kernel parameter "nospectre_v2" and
> > "spectre_v2=[on,off,auto]" can be used. If the parameter is specified
> > the kernel and module code will be patched at runtime.  
> 
> This is really unfortunate naming of kernel option.
> 
> spectre_v2=off sounds like we are turning the "bug" off, but i somehow
> suspect you are turning the bug _workaround_ off.
 
Well, that is the 1:1 copy of the x86 option. Do you want to change that
one as well?

-- 
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.

Re: [PATCH 6/6] s390: introduce execute-trampolines for branches

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 974 bytes --]



On Wed, 2018-02-07 at 11:07 +0100, Pavel Machek wrote:
> This is really unfortunate naming of kernel option.
> 
> spectre_v2=off sounds like we are turning the "bug" off, but i somehow
> suspect you are turning the bug _workaround_ off.

That's consistent with what we have on x86.

> > +ifdef CONFIG_EXPOLINE
> > +  ifeq ($(call cc-option-yn,$(CC_FLAGS_MARCH) -mindirect-branch=thunk),y)
> > +    CC_FLAGS_EXPOLINE := -mindirect-branch=thunk
> > +    CC_FLAGS_EXPOLINE += -mfunction-return=thunk
> > +    CC_FLAGS_EXPOLINE += -mindirect-branch-table
> > +    export CC_FLAGS_EXPOLINE
> > +    cflags-y += $(CC_FLAGS_EXPOLINE)
> > +  else
> > +    $(warning "Your gcc lacks the -mindirect-branch= option")
> > +  endif
> > +endif

That isn't, though. Linus asked us to drop the $(warning) part.

... and then spent a week building with a non-retpoline compiler and
not noticing, so he might have changed his mind ;)

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5213 bytes --]

Re: [PATCH 6/6] s390: introduce execute-trampolines for branches

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 974 bytes --]



On Wed, 2018-02-07 at 11:07 +0100, Pavel Machek wrote:
> This is really unfortunate naming of kernel option.
> 
> spectre_v2=off sounds like we are turning the "bug" off, but i somehow
> suspect you are turning the bug _workaround_ off.

That's consistent with what we have on x86.

> > +ifdef CONFIG_EXPOLINE
> > +  ifeq ($(call cc-option-yn,$(CC_FLAGS_MARCH) -mindirect-branch=thunk),y)
> > +    CC_FLAGS_EXPOLINE := -mindirect-branch=thunk
> > +    CC_FLAGS_EXPOLINE += -mfunction-return=thunk
> > +    CC_FLAGS_EXPOLINE += -mindirect-branch-table
> > +    export CC_FLAGS_EXPOLINE
> > +    cflags-y += $(CC_FLAGS_EXPOLINE)
> > +  else
> > +    $(warning "Your gcc lacks the -mindirect-branch= option")
> > +  endif
> > +endif

That isn't, though. Linus asked us to drop the $(warning) part.

... and then spent a week building with a non-retpoline compiler and
not noticing, so he might have changed his mind ;)

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5213 bytes --]

Re: [PATCH 6/6] s390: introduce execute-trampolines for branches

[Martin Schwidefsky]

On Wed, 07 Feb 2018 12:07:55 +0000
David Woodhouse <dwmw2@infradead.org> wrote:

> On Wed, 2018-02-07 at 11:07 +0100, Pavel Machek wrote:
> > This is really unfortunate naming of kernel option.
> > 
> > spectre_v2=off sounds like we are turning the "bug" off, but i somehow
> > suspect you are turning the bug _workaround_ off.  
> 
> That's consistent with what we have on x86.
> 
> > > +ifdef CONFIG_EXPOLINE
> > > +  ifeq ($(call cc-option-yn,$(CC_FLAGS_MARCH) -mindirect-branch=thunk),y)
> > > +    CC_FLAGS_EXPOLINE := -mindirect-branch=thunk
> > > +    CC_FLAGS_EXPOLINE += -mfunction-return=thunk
> > > +    CC_FLAGS_EXPOLINE += -mindirect-branch-table
> > > +    export CC_FLAGS_EXPOLINE
> > > +    cflags-y += $(CC_FLAGS_EXPOLINE)
> > > +  else
> > > +    $(warning "Your gcc lacks the -mindirect-branch= option")
> > > +  endif
> > > +endif  
> 
> That isn't, though. Linus asked us to drop the $(warning) part.
> 
> ... and then spent a week building with a non-retpoline compiler and
> not noticing, so he might have changed his mind ;)

I found the warning to have some value, it helps for the case where my
fingers are faster than my brain and I type "make" instead of "smake"
which uses the alternative compiler with the required support.

@Linus: do you want a warning or prefer not to have one ?

-- 
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.

Re: [PATCH 6/6] s390: introduce execute-trampolines for branches

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1812 bytes --]



On Wed, 2018-02-07 at 13:17 +0100, Martin Schwidefsky wrote:
> On Wed, 07 Feb 2018 12:07:55 +0000
> David Woodhouse <dwmw2@infradead.org> wrote:
> 
> > 
> > On Wed, 2018-02-07 at 11:07 +0100, Pavel Machek wrote:
> > > 
> > > This is really unfortunate naming of kernel option.
> > > 
> > > spectre_v2=off sounds like we are turning the "bug" off, but i somehow
> > > suspect you are turning the bug _workaround_ off.  
> > That's consistent with what we have on x86.
> > 
> > > 
> > > > 
> > > > +ifdef CONFIG_EXPOLINE
> > > > +  ifeq ($(call cc-option-yn,$(CC_FLAGS_MARCH) -mindirect-branch=thunk),y)
> > > > +    CC_FLAGS_EXPOLINE := -mindirect-branch=thunk
> > > > +    CC_FLAGS_EXPOLINE += -mfunction-return=thunk
> > > > +    CC_FLAGS_EXPOLINE += -mindirect-branch-table
> > > > +    export CC_FLAGS_EXPOLINE
> > > > +    cflags-y += $(CC_FLAGS_EXPOLINE)
> > > > +  else
> > > > +    $(warning "Your gcc lacks the -mindirect-branch= option")
> > > > +  endif
> > > > +endif  
> > That isn't, though. Linus asked us to drop the $(warning) part.
> > 
> > ... and then spent a week building with a non-retpoline compiler and
> > not noticing, so he might have changed his mind ;)
>
> I found the warning to have some value, it helps for the case where my
> fingers are faster than my brain and I type "make" instead of "smake"
> which uses the alternative compiler with the required support.
> 
> @Linus: do you want a warning or prefer not to have one ?

FWIW I agreed to drop it when the plan in my head was "we'll just turn
on IBRS instead if the compiler doesn't do full retpoline support".

Now that Linus has expressed a disinclination to take IBRS support in
that form, I might be more inclined to defend the $(warning) too.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5213 bytes --]

Re: [PATCH 6/6] s390: introduce execute-trampolines for branches

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1812 bytes --]



On Wed, 2018-02-07 at 13:17 +0100, Martin Schwidefsky wrote:
> On Wed, 07 Feb 2018 12:07:55 +0000
> David Woodhouse <dwmw2@infradead.org> wrote:
> 
> > 
> > On Wed, 2018-02-07 at 11:07 +0100, Pavel Machek wrote:
> > > 
> > > This is really unfortunate naming of kernel option.
> > > 
> > > spectre_v2=off sounds like we are turning the "bug" off, but i somehow
> > > suspect you are turning the bug _workaround_ off.  
> > That's consistent with what we have on x86.
> > 
> > > 
> > > > 
> > > > +ifdef CONFIG_EXPOLINE
> > > > +  ifeq ($(call cc-option-yn,$(CC_FLAGS_MARCH) -mindirect-branch=thunk),y)
> > > > +    CC_FLAGS_EXPOLINE := -mindirect-branch=thunk
> > > > +    CC_FLAGS_EXPOLINE += -mfunction-return=thunk
> > > > +    CC_FLAGS_EXPOLINE += -mindirect-branch-table
> > > > +    export CC_FLAGS_EXPOLINE
> > > > +    cflags-y += $(CC_FLAGS_EXPOLINE)
> > > > +  else
> > > > +    $(warning "Your gcc lacks the -mindirect-branch= option")
> > > > +  endif
> > > > +endif  
> > That isn't, though. Linus asked us to drop the $(warning) part.
> > 
> > ... and then spent a week building with a non-retpoline compiler and
> > not noticing, so he might have changed his mind ;)
>
> I found the warning to have some value, it helps for the case where my
> fingers are faster than my brain and I type "make" instead of "smake"
> which uses the alternative compiler with the required support.
> 
> @Linus: do you want a warning or prefer not to have one ?

FWIW I agreed to drop it when the plan in my head was "we'll just turn
on IBRS instead if the compiler doesn't do full retpoline support".

Now that Linus has expressed a disinclination to take IBRS support in
that form, I might be more inclined to defend the $(warning) too.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5213 bytes --]

Re: [PATCH 6/6] s390: introduce execute-trampolines for branches

[Linus Torvalds]

On Wed, Feb 7, 2018 at 4:17 AM, Martin Schwidefsky
<schwidefsky@de.ibm.com> wrote:
>> That isn't, though. Linus asked us to drop the $(warning) part.
>>
>> ... and then spent a week building with a non-retpoline compiler and
>> not noticing, so he might have changed his mind ;)
>
> I found the warning to have some value, it helps for the case where my
> fingers are faster than my brain and I type "make" instead of "smake"
> which uses the alternative compiler with the required support.
>
> @Linus: do you want a warning or prefer not to have one ?

Honestly, I think I'd be much happier with the warning as part of the
"make config" phase.

What really annoyed me was that it showed up at every build.

What I would really want - and this is entirely unrelated to this
particular case - is to have those damn compiler option tests as part
of the config phase in general. We now have about a million of these
crazy things, where we have config options that simply depend on which
compiler we have, and we have no sane way to show them at
configuration time.

Though Andrew's tree I got yet another ugly hack
(CONFIG_CC_STACKPROTECTOR_AUTO) that handles just _one_ special case
by turning it into a special magic Kconfig entry in the main Makefile.
See commit 44c6dc940b19 ("Makefile: introduce
CONFIG_CC_STACKPROTECTOR_AUTO"). I wasn't sure if I really wanted it,
and honestly, I'm still thinking of just reverting it, because it's
_so_ ugly and _so_ wrong.

What we need is an extension to the Kconfig language itself so that we can do

   config CC_HAS_RETPOLINE
        cc_option "-mindirect-branch=thunk -mindirect-branch-table"

or something. And then we can make sane _conditional_ dependencies at
Kconfig time, and our makefiles would be much cleaner too when you
could just do

     cflags-$(USE_RETPOLINE) += -mfunction-return=thunk -mindirect-branch-table

because the validity of the C compiler flag has been tested when configuring.

And then we could add that warning at configure time (or just disable
the option there thanks to "depends on CC_HAS_xyz" logic).

All our compiler option handling right now is just nasty nasty nasty crud.

Adding more people in the hopes that somebody gets motivated.. I've
talked about this before, so far we haven't made any progress.

              Linus

Re: [PATCH 6/6] s390: introduce execute-trampolines for branches

[Masahiro Yamada]

2018-02-08 2:55 GMT+09:00 Linus Torvalds <torvalds@linux-foundation.org>:
> On Wed, Feb 7, 2018 at 4:17 AM, Martin Schwidefsky
> <schwidefsky@de.ibm.com> wrote:
>>> That isn't, though. Linus asked us to drop the $(warning) part.
>>>
>>> ... and then spent a week building with a non-retpoline compiler and
>>> not noticing, so he might have changed his mind ;)
>>
>> I found the warning to have some value, it helps for the case where my
>> fingers are faster than my brain and I type "make" instead of "smake"
>> which uses the alternative compiler with the required support.
>>
>> @Linus: do you want a warning or prefer not to have one ?
>
> Honestly, I think I'd be much happier with the warning as part of the
> "make config" phase.
>
> What really annoyed me was that it showed up at every build.
>
> What I would really want - and this is entirely unrelated to this
> particular case - is to have those damn compiler option tests as part
> of the config phase in general. We now have about a million of these
> crazy things, where we have config options that simply depend on which
> compiler we have, and we have no sane way to show them at
> configuration time.
>
> Though Andrew's tree I got yet another ugly hack
> (CONFIG_CC_STACKPROTECTOR_AUTO) that handles just _one_ special case
> by turning it into a special magic Kconfig entry in the main Makefile.
> See commit 44c6dc940b19 ("Makefile: introduce
> CONFIG_CC_STACKPROTECTOR_AUTO"). I wasn't sure if I really wanted it,
> and honestly, I'm still thinking of just reverting it, because it's
> _so_ ugly and _so_ wrong.
>
> What we need is an extension to the Kconfig language itself so that we can do
>
>    config CC_HAS_RETPOLINE
>         cc_option "-mindirect-branch=thunk -mindirect-branch-table"
>
> or something. And then we can make sane _conditional_ dependencies at
> Kconfig time, and our makefiles would be much cleaner too when you
> could just do
>
>      cflags-$(USE_RETPOLINE) += -mfunction-return=thunk -mindirect-branch-table
>
> because the validity of the C compiler flag has been tested when configuring.
>
> And then we could add that warning at configure time (or just disable
> the option there thanks to "depends on CC_HAS_xyz" logic).
>
> All our compiler option handling right now is just nasty nasty nasty crud.
>
> Adding more people in the hopes that somebody gets motivated.. I've
> talked about this before, so far we haven't made any progress.


Sorry for slow progress.

I agreed this before, and still motivated.
(because I also motivated to remove kbuild cache.
This turned out not so clever as I first thought)

I was trying to do this, but in this development cycle
I spent most of my time to flush out lots of piled up Kconfig patches.
Sorry.

Unless somebody is working, I will.



-- 
Best Regards
Masahiro Yamada

Re: [PATCH 6/6] s390: introduce execute-trampolines for branches

[Linus Torvalds]

On Wed, Feb 7, 2018 at 3:44 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
>
> I agreed this before, and still motivated.
> (because I also motivated to remove kbuild cache.

I actually wish I still had my old "run shell script" thing. I had
some very preliminary patches that actually worked for simple things,
and you could do something like

      config SOME_NAME
            bool
            option shell "true"

and it would act basically like "option env", except it didn't do
"getenv()", it did "system()" and checked the return value. So the
above would make SOME_NAME have the value 'y', because when you
executed "true" it was successful. I have this dim memory of allowing
it to set strings too (filling in the default value with the stdout
output from the shell execution).

So I had some experimental patch like that, and it kind of worked, but
I never finished it.

But the reason I never completed it was that for the compiler option
case, it really wanted more than a shell command, it needed to get the
whole $(CC) etc from the make environment.

I don't remember the exact syntax I used, but I think it was based on
that "option env" syntax, just replacing "env" with "shell". But
searching my mail archives I can't find anything, so I may never have
sent anything out. And so the patch is long gone.

Maybe I'll get frustrated enough and try to recreate it. I don't think
the patch was that big (but as mentioned, it really wasn't in a form
where it was _useful_ yet).

             Linus

Re: [PATCH 6/6] s390: introduce execute-trampolines for branches

[Kees Cook]

On Thu, Feb 8, 2018 at 10:44 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2018-02-08 2:55 GMT+09:00 Linus Torvalds <torvalds@linux-foundation.org>:
>> What I would really want - and this is entirely unrelated to this
>> particular case - is to have those damn compiler option tests as part
>> of the config phase in general. We now have about a million of these
>> crazy things, where we have config options that simply depend on which
>> compiler we have, and we have no sane way to show them at
>> configuration time.
>>
>> Though Andrew's tree I got yet another ugly hack
>> (CONFIG_CC_STACKPROTECTOR_AUTO) that handles just _one_ special case
>> by turning it into a special magic Kconfig entry in the main Makefile.
>> See commit 44c6dc940b19 ("Makefile: introduce
>> CONFIG_CC_STACKPROTECTOR_AUTO"). I wasn't sure if I really wanted it,
>> and honestly, I'm still thinking of just reverting it, because it's
>> _so_ ugly and _so_ wrong.
>>
>> What we need is an extension to the Kconfig language itself so that we can do
>>
>>    config CC_HAS_RETPOLINE
>>         cc_option "-mindirect-branch=thunk -mindirect-branch-table"
>>
>> or something. And then we can make sane _conditional_ dependencies at
>> Kconfig time, and our makefiles would be much cleaner too when you
>> could just do
>>
>>      cflags-$(USE_RETPOLINE) += -mfunction-return=thunk -mindirect-branch-table
>>
>> because the validity of the C compiler flag has been tested when configuring.
>>
>> And then we could add that warning at configure time (or just disable
>> the option there thanks to "depends on CC_HAS_xyz" logic).
>>
>> All our compiler option handling right now is just nasty nasty nasty crud.
>>
>> Adding more people in the hopes that somebody gets motivated.. I've
>> talked about this before, so far we haven't made any progress.
>
>
> Sorry for slow progress.
>
> I agreed this before, and still motivated.
> (because I also motivated to remove kbuild cache.
> This turned out not so clever as I first thought)
>
> I was trying to do this, but in this development cycle
> I spent most of my time to flush out lots of piled up Kconfig patches.
> Sorry.
>
> Unless somebody is working, I will.

FWIW, I did try to do this before I went with the STACKPROTECTOR_AUTO
solution, and it defeated me at every turn. Between the circular
dependency of the Makefile setting up KBUILD flags and Kconfig wanting
to know about the compiler, I got stuck. And it also seemed like the
cache was a problem too, as I couldn't find a way to re-evaluate the
script-controlled CONFIG items once it got cached.

And ultimately, a lot of the operational logic ended up sticking
around in the Makefile anyway (to provide fallback decisions, warns,
and errors). To correctly deal with the complex corner-cases for
stack-protector, I end up with three pieces:

the user config:

- do you want auto, weak, strong, or off?

the compiler tests:

- which of weak, strong, or off are supported?
- does the flag _actually produce working code_ for this
architecture/compiler version/etc?

and the build behavior:

- for auto, choose best available flag and warn if none
- for user-selected weak or strong, stop if unavailable/non-functional
- for off (or auto-none) use the "off" flag, if it is supported (to
force-disable it on ssp-by-default distro compilers)

And all of this needs to bypass the Kconfig cache, since if it gets
cached, it won't get re-evaluated when a selection is changed. If we
had a working "option shell $(CC) ..." then I could do the "compiler
tests" part in Kconfig, and I could probably do the bulk of the "build
behavior" logic in Kconfig too, though any intermediate configs would
need to avoid getting cached too.

The ssp handling has always been extremely complex due to all its gory
details, but I've tried _really_ hard to improve it, keep it
documented, and in one place (e.g. the compiler tests used to be
stuffed in per-arch Makefile).

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH 6/6] s390: introduce execute-trampolines for branches

[Masahiro Yamada]

2018-02-08 9:57 GMT+09:00 Kees Cook <keescook@chromium.org>:
> On Thu, Feb 8, 2018 at 10:44 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> 2018-02-08 2:55 GMT+09:00 Linus Torvalds <torvalds@linux-foundation.org>:
>>> What I would really want - and this is entirely unrelated to this
>>> particular case - is to have those damn compiler option tests as part
>>> of the config phase in general. We now have about a million of these
>>> crazy things, where we have config options that simply depend on which
>>> compiler we have, and we have no sane way to show them at
>>> configuration time.
>>>
>>> Though Andrew's tree I got yet another ugly hack
>>> (CONFIG_CC_STACKPROTECTOR_AUTO) that handles just _one_ special case
>>> by turning it into a special magic Kconfig entry in the main Makefile.
>>> See commit 44c6dc940b19 ("Makefile: introduce
>>> CONFIG_CC_STACKPROTECTOR_AUTO"). I wasn't sure if I really wanted it,
>>> and honestly, I'm still thinking of just reverting it, because it's
>>> _so_ ugly and _so_ wrong.
>>>
>>> What we need is an extension to the Kconfig language itself so that we can do
>>>
>>>    config CC_HAS_RETPOLINE
>>>         cc_option "-mindirect-branch=thunk -mindirect-branch-table"
>>>
>>> or something. And then we can make sane _conditional_ dependencies at
>>> Kconfig time, and our makefiles would be much cleaner too when you
>>> could just do
>>>
>>>      cflags-$(USE_RETPOLINE) += -mfunction-return=thunk -mindirect-branch-table
>>>
>>> because the validity of the C compiler flag has been tested when configuring.
>>>
>>> And then we could add that warning at configure time (or just disable
>>> the option there thanks to "depends on CC_HAS_xyz" logic).
>>>
>>> All our compiler option handling right now is just nasty nasty nasty crud.
>>>
>>> Adding more people in the hopes that somebody gets motivated.. I've
>>> talked about this before, so far we haven't made any progress.
>>
>>
>> Sorry for slow progress.
>>
>> I agreed this before, and still motivated.
>> (because I also motivated to remove kbuild cache.
>> This turned out not so clever as I first thought)
>>
>> I was trying to do this, but in this development cycle
>> I spent most of my time to flush out lots of piled up Kconfig patches.
>> Sorry.
>>
>> Unless somebody is working, I will.

I sent the first draft.

For people interested, I pushed the series in the following too.

git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git
kconfig-opt-shell-rfc



-- 
Best Regards
Masahiro Yamada