[PATCH v5sub2 0/8] arm64: implement virtual KASLR

[PATCH v5sub2 0/8] arm64: implement virtual KASLR

[Ard Biesheuvel]

At the request of Catalin, this series has been split off from my series
'arm64: implement support for KASLR v4' [1]. This sub-series deals with
creating a relocatable binary, and randomizing the virtual placement of
the kernel itself, the modules and the linear region based on entropy
supplied by the bootloader in the /chosen/kaslr-seed DT property.

Changes since v4:
- add randomization of the linear region, i.e., if the linear region is
  substantially larger than the space spanned by RAM, the mapping of RAM
  is moved to a random offset inside the linear region.
- dropped the CRC check in kaslr_early_init(), since the only code that may
  modify .data through __fixmap_remap_fdt() is instrumentation that should
  deal with that correctly (For instance, the branch profiling records taken
  branches in structs allocated in .data, and these counts will not be reset
  to 0 between the first and the second call to __mmap_switched())
- add Mark's ack to patch #4

Patch #1 adds support for resolving cross-module branches via veneers if the
module is located too far from the target. This should not be necessary in the
common case even with KASRL enabled, but since the space around the kernel image
will no longer be strictly reserved for modules, there is a slight chance that
the region may be exhausted, in which case modules will be moved further away
from the kernel (and each other)

Patches #2 and #3 deal with the fact that relocations of 64-bit values are
deferred to runtime when building a PIE executable, and some values are required
before the relocation routine executes (either in the code, or as an externally
visible constant in the Image header)

Patch #4 rehuffles asm/elf.h so we can use its preprocessor constants in head.S

Patch #5 and #6 implement building vmlinux as a PIE (Position Independent
Executable) binary, which is emitted with relocation information that is
processed by the program itself in the early boot code.

Patch #7 implements KASLR, i.e., CONFIG_RANDOMIZE_BASE for the core kernel and
for the module region.

Patch #8 implements KASLR for the linear region.

[1] http://thread.gmane.org/gmane.linux.kernel/2135931

Ard Biesheuvel (8):
  arm64: add support for module PLTs
  arm64: avoid R_AARCH64_ABS64 relocations for Image header fields
  arm64: avoid dynamic relocations in early boot code
  arm64: make asm/elf.h available to asm files
  scripts/sortextable: add support for ET_DYN binaries
  arm64: add support for building vmlinux as a relocatable PIE binary
  arm64: add support for kernel ASLR
  arm64: kaslr: randomize the linear region

 arch/arm64/Kconfig                 |  34 ++++
 arch/arm64/Makefile                |  10 +-
 arch/arm64/include/asm/assembler.h |  11 ++
 arch/arm64/include/asm/elf.h       |  24 ++-
 arch/arm64/include/asm/memory.h    |   5 +-
 arch/arm64/include/asm/module.h    |  11 ++
 arch/arm64/kernel/Makefile         |   2 +
 arch/arm64/kernel/efi-entry.S      |   2 +-
 arch/arm64/kernel/head.S           | 128 +++++++++++--
 arch/arm64/kernel/image.h          |  32 ++--
 arch/arm64/kernel/kaslr.c          | 143 ++++++++++++++
 arch/arm64/kernel/module-plts.c    | 201 ++++++++++++++++++++
 arch/arm64/kernel/module.c         |  20 +-
 arch/arm64/kernel/module.lds       |   3 +
 arch/arm64/kernel/setup.c          |  29 +++
 arch/arm64/kernel/vmlinux.lds.S    |  16 ++
 arch/arm64/mm/init.c               |  22 ++-
 arch/arm64/mm/mmu.c                |  33 +++-
 scripts/sortextable.c              |   8 +-
 19 files changed, 672 insertions(+), 62 deletions(-)
 create mode 100644 arch/arm64/kernel/kaslr.c
 create mode 100644 arch/arm64/kernel/module-plts.c
 create mode 100644 arch/arm64/kernel/module.lds

-- 
2.5.0

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Ard Biesheuvel]

This adds support for emitting PLTs at module load time for relative
branches that are out of range. This is a prerequisite for KASLR, which
may place the kernel and the modules anywhere in the vmalloc area,
making it more likely that branch target offsets exceed the maximum
range of +/- 128 MB.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---

In this version, I removed the distinction between relocations against
.init executable sections and ordinary executable sections. The reason
is that it is hardly worth the trouble, given that .init.text usually
does not contain that many far branches, and this version now only
reserves PLT entry space for jump and call relocations against undefined
symbols (since symbols defined in the same module can be assumed to be
within +/- 128 MB)

For example, the mac80211.ko module (which is fairly sizable at ~400 KB)
built with -mcmodel=large gives the following relocation counts:

                    relocs    branches   unique     !local
  .text              3925       3347       518        219
  .init.text           11          8         7          1
  .exit.text            4          4         4          1
  .text.unlikely       81         67        36         17

('unique' means branches to unique type/symbol/addend combos, of which
!local is the subset referring to undefined symbols)

IOW, we are only emitting a single PLT entry for the .init sections, and
we are better off just adding it to the core PLT section instead.
---
 arch/arm64/Kconfig              |   9 +
 arch/arm64/Makefile             |   6 +-
 arch/arm64/include/asm/module.h |  11 ++
 arch/arm64/kernel/Makefile      |   1 +
 arch/arm64/kernel/module-plts.c | 201 ++++++++++++++++++++
 arch/arm64/kernel/module.c      |  12 ++
 arch/arm64/kernel/module.lds    |   3 +
 7 files changed, 242 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index cd767fa3037a..141f65ab0ed5 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -394,6 +394,7 @@ config ARM64_ERRATUM_843419
 	bool "Cortex-A53: 843419: A load or store might access an incorrect address"
 	depends on MODULES
 	default y
+	select ARM64_MODULE_CMODEL_LARGE
 	help
 	  This option builds kernel modules using the large memory model in
 	  order to avoid the use of the ADRP instruction, which can cause
@@ -753,6 +754,14 @@ config ARM64_LSE_ATOMICS
 
 endmenu
 
+config ARM64_MODULE_CMODEL_LARGE
+	bool
+
+config ARM64_MODULE_PLTS
+	bool
+	select ARM64_MODULE_CMODEL_LARGE
+	select HAVE_MOD_ARCH_SPECIFIC
+
 endmenu
 
 menu "Boot options"
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index 307237cfe728..a6bba9623836 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -43,10 +43,14 @@ endif
 
 CHECKFLAGS	+= -D__aarch64__
 
-ifeq ($(CONFIG_ARM64_ERRATUM_843419), y)
+ifeq ($(CONFIG_ARM64_MODULE_CMODEL_LARGE), y)
 KBUILD_CFLAGS_MODULE	+= -mcmodel=large
 endif
 
+ifeq ($(CONFIG_ARM64_MODULE_PLTS),y)
+KBUILD_LDFLAGS_MODULE	+= -T $(srctree)/arch/arm64/kernel/module.lds
+endif
+
 # Default value
 head-y		:= arch/arm64/kernel/head.o
 
diff --git a/arch/arm64/include/asm/module.h b/arch/arm64/include/asm/module.h
index e80e232b730e..8652fb613304 100644
--- a/arch/arm64/include/asm/module.h
+++ b/arch/arm64/include/asm/module.h
@@ -20,4 +20,15 @@
 
 #define MODULE_ARCH_VERMAGIC	"aarch64"
 
+#ifdef CONFIG_ARM64_MODULE_PLTS
+struct mod_arch_specific {
+	struct elf64_shdr	*plt;
+	int			plt_num_entries;
+	int			plt_max_entries;
+};
+#endif
+
+u64 module_emit_plt_entry(struct module *mod, const Elf64_Rela *rela,
+			  Elf64_Sym *sym);
+
 #endif /* __ASM_MODULE_H */
diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
index 83cd7e68e83b..e2f0a755beaa 100644
--- a/arch/arm64/kernel/Makefile
+++ b/arch/arm64/kernel/Makefile
@@ -30,6 +30,7 @@ arm64-obj-$(CONFIG_COMPAT)		+= sys32.o kuser32.o signal32.o 	\
 					   ../../arm/kernel/opcodes.o
 arm64-obj-$(CONFIG_FUNCTION_TRACER)	+= ftrace.o entry-ftrace.o
 arm64-obj-$(CONFIG_MODULES)		+= arm64ksyms.o module.o
+arm64-obj-$(CONFIG_ARM64_MODULE_PLTS)	+= module-plts.o
 arm64-obj-$(CONFIG_PERF_EVENTS)		+= perf_regs.o perf_callchain.o
 arm64-obj-$(CONFIG_HW_PERF_EVENTS)	+= perf_event.o
 arm64-obj-$(CONFIG_HAVE_HW_BREAKPOINT)	+= hw_breakpoint.o
diff --git a/arch/arm64/kernel/module-plts.c b/arch/arm64/kernel/module-plts.c
new file mode 100644
index 000000000000..1ce90d8450ae
--- /dev/null
+++ b/arch/arm64/kernel/module-plts.c
@@ -0,0 +1,201 @@
+/*
+ * Copyright (C) 2014-2016 Linaro Ltd. <ard.biesheuvel@linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/elf.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/sort.h>
+
+struct plt_entry {
+	/*
+	 * A program that conforms to the AArch64 Procedure Call Standard
+	 * (AAPCS64) must assume that a veneer that alters IP0 (x16) and/or
+	 * IP1 (x17) may be inserted at any branch instruction that is
+	 * exposed to a relocation that supports long branches. Since that
+	 * is exactly what we are dealing with here, we are free to use x16
+	 * as a scratch register in the PLT veneers.
+	 */
+	__le32	mov0;	/* movn	x16, #0x....			*/
+	__le32	mov1;	/* movk	x16, #0x...., lsl #16		*/
+	__le32	mov2;	/* movk	x16, #0x...., lsl #32		*/
+	__le32	br;	/* br	x16				*/
+};
+
+u64 module_emit_plt_entry(struct module *mod, const Elf64_Rela *rela,
+			  Elf64_Sym *sym)
+{
+	struct plt_entry *plt = (struct plt_entry *)mod->arch.plt->sh_addr;
+	int i = mod->arch.plt_num_entries;
+	u64 val = sym->st_value + rela->r_addend;
+
+	/*
+	 * We only emit PLT entries against undefined (SHN_UNDEF) symbols,
+	 * which are listed in the ELF symtab section, but without a type
+	 * or a size.
+	 * So, similar to how the module loader uses the Elf64_Sym::st_value
+	 * field to store the resolved addresses of undefined symbols, let's
+	 * borrow the Elf64_Sym::st_size field (whose value is never used by
+	 * the module loader, even for symbols that are defined) to record
+	 * the address of a symbol's associated PLT entry as we emit it for a
+	 * zero addend relocation (which is the only kind we have to deal with
+	 * in practice). This allows us to find duplicates without having to
+	 * go through the table every time.
+	 */
+	if (rela->r_addend == 0 && sym->st_size != 0) {
+		BUG_ON(sym->st_size < (u64)plt || sym->st_size >= (u64)&plt[i]);
+		return sym->st_size;
+	}
+
+	mod->arch.plt_num_entries++;
+	BUG_ON(mod->arch.plt_num_entries > mod->arch.plt_max_entries);
+
+	/*
+	 * MOVK/MOVN/MOVZ opcode:
+	 * +--------+------------+--------+-----------+-------------+---------+
+	 * | sf[31] | opc[30:29] | 100101 | hw[22:21] | imm16[20:5] | Rd[4:0] |
+	 * +--------+------------+--------+-----------+-------------+---------+
+	 *
+	 * Rd     := 0x10 (x16)
+	 * hw     := 0b00 (no shift), 0b01 (lsl #16), 0b10 (lsl #32)
+	 * opc    := 0b11 (MOVK), 0b00 (MOVN), 0b10 (MOVZ)
+	 * sf     := 1 (64-bit variant)
+	 */
+	plt[i] = (struct plt_entry){
+		cpu_to_le32(0x92800010 | (((~val      ) & 0xffff)) << 5),
+		cpu_to_le32(0xf2a00010 | ((( val >> 16) & 0xffff)) << 5),
+		cpu_to_le32(0xf2c00010 | ((( val >> 32) & 0xffff)) << 5),
+		cpu_to_le32(0xd61f0200)
+	};
+
+	if (rela->r_addend == 0)
+		sym->st_size = (u64)&plt[i];
+
+	return (u64)&plt[i];
+}
+
+#define cmp_3way(a,b)	((a) < (b) ? -1 : (a) > (b))
+
+static int cmp_rela(const void *a, const void *b)
+{
+	const Elf64_Rela *x = a, *y = b;
+	int i;
+
+	/* sort by type, symbol index and addend */
+	i = cmp_3way(ELF64_R_TYPE(x->r_info), ELF64_R_TYPE(y->r_info));
+	if (i == 0)
+		i = cmp_3way(ELF64_R_SYM(x->r_info), ELF64_R_SYM(y->r_info));
+	if (i == 0)
+		i = cmp_3way(x->r_addend, y->r_addend);
+	return i;
+}
+
+static bool duplicate_rel(const Elf64_Rela *rela, int num)
+{
+	/*
+	 * Entries are sorted by type, symbol index and addend. That means
+	 * that, if a duplicate entry exists, it must be in the preceding
+	 * slot.
+	 */
+	return num > 0 && cmp_rela(rela + num, rela + num - 1) == 0;
+}
+
+static unsigned int count_plts(Elf64_Sym *syms, Elf64_Rela *rela, int num)
+{
+	unsigned int ret = 0;
+	Elf64_Sym *s;
+	int i;
+
+	for (i = 0; i < num; i++) {
+		switch (ELF64_R_TYPE(rela[i].r_info)) {
+		case R_AARCH64_JUMP26:
+		case R_AARCH64_CALL26:
+			/*
+			 * We only have to consider branch targets that resolve
+			 * to undefined symbols. This is not simply a heuristic,
+			 * it is a fundamental limitation, since the PLT itself
+			 * is part of the module, and needs to be within 128 MB
+			 * as well, so modules can never grow beyond that limit.
+			 */
+			s = syms + ELF64_R_SYM(rela[i].r_info);
+			if (s->st_shndx != SHN_UNDEF)
+				break;
+
+			/*
+			 * Jump relocations with non-zero addends against
+			 * undefined symbols are supported by the ELF spec, but
+			 * do not occur in practice (e.g., 'jump n bytes past
+			 * the entry point of undefined function symbol f').
+			 * So we need to support them, but there is no need to
+			 * take them into consideration when trying to optimize
+			 * this code. So let's only check for duplicates when
+			 * the addend is zero: this allows us to record the PLT
+			 * entry address in the symbol table itself, rather than
+			 * having to search the list for duplicates each time we
+			 * emit one.
+			 */
+			if (rela[i].r_addend != 0 || !duplicate_rel(rela, i))
+				ret++;
+			break;
+		}
+	}
+	return ret;
+}
+
+int module_frob_arch_sections(Elf_Ehdr *ehdr, Elf_Shdr *sechdrs,
+			      char *secstrings, struct module *mod)
+{
+	unsigned long plt_max_entries = 0;
+	Elf64_Sym *syms = NULL;
+	int i;
+
+	/*
+	 * Find the empty .plt section so we can expand it to store the PLT
+	 * entries. Record the symtab address as well.
+	 */
+	for (i = 0; i < ehdr->e_shnum; i++) {
+		if (strcmp(".plt", secstrings + sechdrs[i].sh_name) == 0)
+			mod->arch.plt = sechdrs + i;
+		else if (sechdrs[i].sh_type == SHT_SYMTAB)
+			syms = (Elf64_Sym *)sechdrs[i].sh_addr;
+	}
+
+	if (!mod->arch.plt) {
+		pr_err("%s: module PLT section missing\n", mod->name);
+		return -ENOEXEC;
+	}
+	if (!syms) {
+		pr_err("%s: module symtab section missing\n", mod->name);
+		return -ENOEXEC;
+	}
+
+	for (i = 0; i < ehdr->e_shnum; i++) {
+		Elf64_Rela *rels = (void *)ehdr + sechdrs[i].sh_offset;
+		int numrels = sechdrs[i].sh_size / sizeof(Elf64_Rela);
+		Elf64_Shdr *dstsec = sechdrs + sechdrs[i].sh_info;
+
+		if (sechdrs[i].sh_type != SHT_RELA)
+			continue;
+
+		/* ignore relocations that operate on non-exec sections */
+		if (!(dstsec->sh_flags & SHF_EXECINSTR))
+			continue;
+
+		/* sort by type, symbol index and addend */
+		sort(rels, numrels, sizeof(Elf64_Rela), cmp_rela, NULL);
+
+		plt_max_entries += count_plts(syms, rels, numrels);
+	}
+
+	mod->arch.plt->sh_type = SHT_NOBITS;
+	mod->arch.plt->sh_flags = SHF_EXECINSTR | SHF_ALLOC;
+	mod->arch.plt->sh_addralign = L1_CACHE_BYTES;
+	mod->arch.plt->sh_size = plt_max_entries * sizeof(struct plt_entry);
+	mod->arch.plt_num_entries = 0;
+	mod->arch.plt_max_entries = plt_max_entries;
+	return 0;
+}
diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c
index 93e970231ca9..84113d3e1df1 100644
--- a/arch/arm64/kernel/module.c
+++ b/arch/arm64/kernel/module.c
@@ -38,6 +38,11 @@ void *module_alloc(unsigned long size)
 				GFP_KERNEL, PAGE_KERNEL_EXEC, 0,
 				NUMA_NO_NODE, __builtin_return_address(0));
 
+	if (!p && IS_ENABLED(CONFIG_ARM64_MODULE_PLTS))
+		p = __vmalloc_node_range(size, MODULE_ALIGN, VMALLOC_START,
+				VMALLOC_END, GFP_KERNEL, PAGE_KERNEL_EXEC, 0,
+				NUMA_NO_NODE, __builtin_return_address(0));
+
 	if (p && (kasan_module_alloc(p, size) < 0)) {
 		vfree(p);
 		return NULL;
@@ -361,6 +366,13 @@ int apply_relocate_add(Elf64_Shdr *sechdrs,
 		case R_AARCH64_CALL26:
 			ovf = reloc_insn_imm(RELOC_OP_PREL, loc, val, 2, 26,
 					     AARCH64_INSN_IMM_26);
+
+			if (IS_ENABLED(CONFIG_ARM64_MODULE_PLTS) &&
+			    ovf == -ERANGE) {
+				val = module_emit_plt_entry(me, &rel[i], sym);
+				ovf = reloc_insn_imm(RELOC_OP_PREL, loc, val, 2,
+						     26, AARCH64_INSN_IMM_26);
+			}
 			break;
 
 		default:
diff --git a/arch/arm64/kernel/module.lds b/arch/arm64/kernel/module.lds
new file mode 100644
index 000000000000..8949f6c6f729
--- /dev/null
+++ b/arch/arm64/kernel/module.lds
@@ -0,0 +1,3 @@
+SECTIONS {
+	.plt (NOLOAD) : { BYTE(0) }
+}
-- 
2.5.0

[PATCH v5sub2 2/8] arm64: avoid R_AARCH64_ABS64 relocations for Image header fields

[Ard Biesheuvel]

Unfortunately, the current way of using the linker to emit build time
constants into the Image header will no longer work once we switch to
the use of PIE executables. The reason is that such constants are emitted
into the binary using R_AARCH64_ABS64 relocations, which are resolved at
runtime, not at build time, and the places targeted by those relocations
will contain zeroes before that.

So refactor the endian swapping linker script constant generation code so
that it emits the upper and lower 32-bit words separately.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 arch/arm64/include/asm/assembler.h | 11 +++++++
 arch/arm64/kernel/head.S           |  6 ++--
 arch/arm64/kernel/image.h          | 32 ++++++++++++--------
 3 files changed, 33 insertions(+), 16 deletions(-)

diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index d8bfcc1ce923..70f7b9e04598 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -222,4 +222,15 @@ lr	.req	x30		// link register
 	.size	__pi_##x, . - x;	\
 	ENDPROC(x)
 
+	/*
+	 * Emit a 64-bit absolute little endian symbol reference in a way that
+	 * ensures that it will be resolved at build time, even when building a
+	 * PIE binary. This requires cooperation from the linker script, which
+	 * must emit the lo32/hi32 halves individually.
+	 */
+	.macro	le64sym, sym
+	.long	\sym\()_lo32
+	.long	\sym\()_hi32
+	.endm
+
 #endif	/* __ASM_ASSEMBLER_H */
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 05b98289093e..f076debf392d 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -83,9 +83,9 @@ efi_head:
 	b	stext				// branch to kernel start, magic
 	.long	0				// reserved
 #endif
-	.quad	_kernel_offset_le		// Image load offset from start of RAM, little-endian
-	.quad	_kernel_size_le			// Effective size of kernel image, little-endian
-	.quad	_kernel_flags_le		// Informative flags, little-endian
+	le64sym	_kernel_offset_le		// Image load offset from start of RAM, little-endian
+	le64sym	_kernel_size_le			// Effective size of kernel image, little-endian
+	le64sym	_kernel_flags_le		// Informative flags, little-endian
 	.quad	0				// reserved
 	.quad	0				// reserved
 	.quad	0				// reserved
diff --git a/arch/arm64/kernel/image.h b/arch/arm64/kernel/image.h
index c9c62cab25a4..db1bf57948f1 100644
--- a/arch/arm64/kernel/image.h
+++ b/arch/arm64/kernel/image.h
@@ -26,21 +26,27 @@
  * There aren't any ELF relocations we can use to endian-swap values known only
  * at link time (e.g. the subtraction of two symbol addresses), so we must get
  * the linker to endian-swap certain values before emitting them.
+ *
+ * Note that, in order for this to work when building the ELF64 PIE executable
+ * (for KASLR), these values should not be referenced via R_AARCH64_ABS64
+ * relocations, since these are fixed up at runtime rather than at build time
+ * when PIE is in effect. So we need to split them up in 32-bit high and low
+ * words.
  */
 #ifdef CONFIG_CPU_BIG_ENDIAN
-#define DATA_LE64(data)					\
-	((((data) & 0x00000000000000ff) << 56) |	\
-	 (((data) & 0x000000000000ff00) << 40) |	\
-	 (((data) & 0x0000000000ff0000) << 24) |	\
-	 (((data) & 0x00000000ff000000) << 8)  |	\
-	 (((data) & 0x000000ff00000000) >> 8)  |	\
-	 (((data) & 0x0000ff0000000000) >> 24) |	\
-	 (((data) & 0x00ff000000000000) >> 40) |	\
-	 (((data) & 0xff00000000000000) >> 56))
+#define DATA_LE32(data)				\
+	((((data) & 0x000000ff) << 24) |	\
+	 (((data) & 0x0000ff00) << 8)  |	\
+	 (((data) & 0x00ff0000) >> 8)  |	\
+	 (((data) & 0xff000000) >> 24))
 #else
-#define DATA_LE64(data) ((data) & 0xffffffffffffffff)
+#define DATA_LE32(data) ((data) & 0xffffffff)
 #endif
 
+#define DEFINE_IMAGE_LE64(sym, data)				\
+	sym##_lo32 = DATA_LE32((data) & 0xffffffff);		\
+	sym##_hi32 = DATA_LE32((data) >> 32)
+
 #ifdef CONFIG_CPU_BIG_ENDIAN
 #define __HEAD_FLAG_BE		1
 #else
@@ -61,9 +67,9 @@
  * endian swapped in head.S, all are done here for consistency.
  */
 #define HEAD_SYMBOLS						\
-	_kernel_size_le		= DATA_LE64(_end - _text);	\
-	_kernel_offset_le	= DATA_LE64(TEXT_OFFSET);	\
-	_kernel_flags_le	= DATA_LE64(__HEAD_FLAGS);
+	DEFINE_IMAGE_LE64(_kernel_size_le, _end - _text);	\
+	DEFINE_IMAGE_LE64(_kernel_offset_le, TEXT_OFFSET);	\
+	DEFINE_IMAGE_LE64(_kernel_flags_le, __HEAD_FLAGS);
 
 #ifdef CONFIG_EFI
 
-- 
2.5.0

[PATCH v5sub2 3/8] arm64: avoid dynamic relocations in early boot code

[Ard Biesheuvel]

Before implementing KASLR for arm64 by building a self-relocating PIE
executable, we have to ensure that values we use before the relocation
routine is executed are not subject to dynamic relocation themselves.
This applies not only to virtual addresses, but also to values that are
supplied by the linker at build time and relocated using R_AARCH64_ABS64
relocations.

So instead, use assemble time constants, or force the use of static
relocations by folding the constants into the instructions.

Reviewed-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 arch/arm64/kernel/efi-entry.S |  2 +-
 arch/arm64/kernel/head.S      | 39 +++++++++++++-------
 2 files changed, 27 insertions(+), 14 deletions(-)

diff --git a/arch/arm64/kernel/efi-entry.S b/arch/arm64/kernel/efi-entry.S
index a773db92908b..f82036e02485 100644
--- a/arch/arm64/kernel/efi-entry.S
+++ b/arch/arm64/kernel/efi-entry.S
@@ -61,7 +61,7 @@ ENTRY(entry)
 	 */
 	mov	x20, x0		// DTB address
 	ldr	x0, [sp, #16]	// relocated _text address
-	ldr	x21, =stext_offset
+	movz	x21, #:abs_g0:stext_offset
 	add	x21, x0, x21
 
 	/*
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index f076debf392d..4cad8f9f2268 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -67,12 +67,11 @@
  * in the entry routines.
  */
 	__HEAD
-
+_head:
 	/*
 	 * DO NOT MODIFY. Image header expected by Linux boot-loaders.
 	 */
 #ifdef CONFIG_EFI
-efi_head:
 	/*
 	 * This add instruction has no meaningful effect except that
 	 * its opcode forms the magic "MZ" signature required by UEFI.
@@ -94,14 +93,14 @@ efi_head:
 	.byte	0x4d
 	.byte	0x64
 #ifdef CONFIG_EFI
-	.long	pe_header - efi_head		// Offset to the PE header.
+	.long	pe_header - _head		// Offset to the PE header.
 #else
 	.word	0				// reserved
 #endif
 
 #ifdef CONFIG_EFI
 	.globl	__efistub_stext_offset
-	.set	__efistub_stext_offset, stext - efi_head
+	.set	__efistub_stext_offset, stext - _head
 	.align 3
 pe_header:
 	.ascii	"PE"
@@ -124,7 +123,7 @@ optional_header:
 	.long	_end - stext			// SizeOfCode
 	.long	0				// SizeOfInitializedData
 	.long	0				// SizeOfUninitializedData
-	.long	__efistub_entry - efi_head	// AddressOfEntryPoint
+	.long	__efistub_entry - _head		// AddressOfEntryPoint
 	.long	__efistub_stext_offset		// BaseOfCode
 
 extra_header_fields:
@@ -139,7 +138,7 @@ extra_header_fields:
 	.short	0				// MinorSubsystemVersion
 	.long	0				// Win32VersionValue
 
-	.long	_end - efi_head			// SizeOfImage
+	.long	_end - _head			// SizeOfImage
 
 	// Everything before the kernel image is considered part of the header
 	.long	__efistub_stext_offset		// SizeOfHeaders
@@ -219,11 +218,13 @@ ENTRY(stext)
 	 * On return, the CPU will be ready for the MMU to be turned on and
 	 * the TCR will have been set.
 	 */
-	ldr	x27, =__mmap_switched		// address to jump to after
+	ldr	x27, 0f				// address to jump to after
 						// MMU has been enabled
 	adr_l	lr, __enable_mmu		// return (PIC) address
 	b	__cpu_setup			// initialise processor
 ENDPROC(stext)
+	.align	3
+0:	.quad	__mmap_switched - (_head - TEXT_OFFSET) + KIMAGE_VADDR
 
 /*
  * Preserve the arguments passed by the bootloader in x0 .. x3
@@ -391,7 +392,8 @@ __create_page_tables:
 	mov	x0, x26				// swapper_pg_dir
 	ldr	x5, =KIMAGE_VADDR
 	create_pgd_entry x0, x5, x3, x6
-	ldr	x6, =KERNEL_END			// __va(KERNEL_END)
+	ldr	w6, kernel_img_size
+	add	x6, x6, x5
 	mov	x3, x24				// phys offset
 	create_block_map x0, x7, x3, x5, x6
 
@@ -408,6 +410,9 @@ __create_page_tables:
 	mov	lr, x27
 	ret
 ENDPROC(__create_page_tables)
+
+kernel_img_size:
+	.long	_end - (_head - TEXT_OFFSET)
 	.ltorg
 
 /*
@@ -415,6 +420,10 @@ ENDPROC(__create_page_tables)
  */
 	.set	initial_sp, init_thread_union + THREAD_START_SP
 __mmap_switched:
+	adr_l	x8, vectors			// load VBAR_EL1 with virtual
+	msr	vbar_el1, x8			// vector table address
+	isb
+
 	// Clear BSS
 	adr_l	x0, __bss_start
 	mov	x1, xzr
@@ -610,13 +619,19 @@ ENTRY(secondary_startup)
 	adrp	x26, swapper_pg_dir
 	bl	__cpu_setup			// initialise processor
 
-	ldr	x21, =secondary_data
-	ldr	x27, =__secondary_switched	// address to jump to after enabling the MMU
+	ldr	x8, =KIMAGE_VADDR
+	ldr	w9, 0f
+	sub	x27, x8, w9, sxtw		// address to jump to after enabling the MMU
 	b	__enable_mmu
 ENDPROC(secondary_startup)
+0:	.long	(_text - TEXT_OFFSET) - __secondary_switched
 
 ENTRY(__secondary_switched)
-	ldr	x0, [x21]			// get secondary_data.stack
+	adr_l	x5, vectors
+	msr	vbar_el1, x5
+	isb
+
+	ldr_l	x0, secondary_data		// get secondary_data.stack
 	mov	sp, x0
 	and	x0, x0, #~(THREAD_SIZE - 1)
 	msr	sp_el0, x0			// save thread_info
@@ -641,8 +656,6 @@ __enable_mmu:
 	ubfx	x2, x1, #ID_AA64MMFR0_TGRAN_SHIFT, 4
 	cmp	x2, #ID_AA64MMFR0_TGRAN_SUPPORTED
 	b.ne	__no_granule_support
-	ldr	x5, =vectors
-	msr	vbar_el1, x5
 	msr	ttbr0_el1, x25			// load TTBR0
 	msr	ttbr1_el1, x26			// load TTBR1
 	isb
-- 
2.5.0

[PATCH v5sub2 4/8] arm64: make asm/elf.h available to asm files

[Ard Biesheuvel]

This reshuffles some code in asm/elf.h and puts a #ifndef __ASSEMBLY__
around its C definitions so that the CPP defines can be used in asm
source files as well.

Acked-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 arch/arm64/include/asm/elf.h | 22 ++++++++++++--------
 1 file changed, 13 insertions(+), 9 deletions(-)

diff --git a/arch/arm64/include/asm/elf.h b/arch/arm64/include/asm/elf.h
index faad6df49e5b..435f55952e1f 100644
--- a/arch/arm64/include/asm/elf.h
+++ b/arch/arm64/include/asm/elf.h
@@ -24,15 +24,6 @@
 #include <asm/ptrace.h>
 #include <asm/user.h>
 
-typedef unsigned long elf_greg_t;
-
-#define ELF_NGREG (sizeof(struct user_pt_regs) / sizeof(elf_greg_t))
-#define ELF_CORE_COPY_REGS(dest, regs)	\
-	*(struct user_pt_regs *)&(dest) = (regs)->user_regs;
-
-typedef elf_greg_t elf_gregset_t[ELF_NGREG];
-typedef struct user_fpsimd_state elf_fpregset_t;
-
 /*
  * AArch64 static relocation types.
  */
@@ -127,6 +118,17 @@ typedef struct user_fpsimd_state elf_fpregset_t;
  */
 #define ELF_ET_DYN_BASE	(2 * TASK_SIZE_64 / 3)
 
+#ifndef __ASSEMBLY__
+
+typedef unsigned long elf_greg_t;
+
+#define ELF_NGREG (sizeof(struct user_pt_regs) / sizeof(elf_greg_t))
+#define ELF_CORE_COPY_REGS(dest, regs)	\
+	*(struct user_pt_regs *)&(dest) = (regs)->user_regs;
+
+typedef elf_greg_t elf_gregset_t[ELF_NGREG];
+typedef struct user_fpsimd_state elf_fpregset_t;
+
 /*
  * When the program starts, a1 contains a pointer to a function to be
  * registered with atexit, as per the SVR4 ABI.  A value of 0 means we have no
@@ -186,4 +188,6 @@ extern int aarch32_setup_vectors_page(struct linux_binprm *bprm,
 
 #endif /* CONFIG_COMPAT */
 
+#endif /* !__ASSEMBLY__ */
+
 #endif
-- 
2.5.0

[PATCH v5sub2 5/8] scripts/sortextable: add support for ET_DYN binaries

[Ard Biesheuvel]

Add support to scripts/sortextable for handling relocatable (PIE)
executables, whose ELF type is ET_DYN, not ET_EXEC. Other than adding
support for the new type, no changes are needed.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 scripts/sortextable.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/scripts/sortextable.c b/scripts/sortextable.c
index af247c70fb66..19d83647846c 100644
--- a/scripts/sortextable.c
+++ b/scripts/sortextable.c
@@ -266,9 +266,9 @@ do_file(char const *const fname)
 		break;
 	}  /* end switch */
 	if (memcmp(ELFMAG, ehdr->e_ident, SELFMAG) != 0
-	||  r2(&ehdr->e_type) != ET_EXEC
+	||  (r2(&ehdr->e_type) != ET_EXEC && r2(&ehdr->e_type) != ET_DYN)
 	||  ehdr->e_ident[EI_VERSION] != EV_CURRENT) {
-		fprintf(stderr, "unrecognized ET_EXEC file %s\n", fname);
+		fprintf(stderr, "unrecognized ET_EXEC/ET_DYN file %s\n", fname);
 		fail_file();
 	}
 
@@ -304,7 +304,7 @@ do_file(char const *const fname)
 		if (r2(&ehdr->e_ehsize) != sizeof(Elf32_Ehdr)
 		||  r2(&ehdr->e_shentsize) != sizeof(Elf32_Shdr)) {
 			fprintf(stderr,
-				"unrecognized ET_EXEC file: %s\n", fname);
+				"unrecognized ET_EXEC/ET_DYN file: %s\n", fname);
 			fail_file();
 		}
 		do32(ehdr, fname, custom_sort);
@@ -314,7 +314,7 @@ do_file(char const *const fname)
 		if (r2(&ghdr->e_ehsize) != sizeof(Elf64_Ehdr)
 		||  r2(&ghdr->e_shentsize) != sizeof(Elf64_Shdr)) {
 			fprintf(stderr,
-				"unrecognized ET_EXEC file: %s\n", fname);
+				"unrecognized ET_EXEC/ET_DYN file: %s\n", fname);
 			fail_file();
 		}
 		do64(ghdr, fname, custom_sort);
-- 
2.5.0

[PATCH v5sub2 6/8] arm64: add support for building vmlinux as a relocatable PIE binary

[Ard Biesheuvel]

This implements CONFIG_RELOCATABLE, which links the final vmlinux
image with a dynamic relocation section, allowing the early boot code
to perform a relocation to a different virtual address at runtime.

This is a prerequisite for KASLR (CONFIG_RANDOMIZE_BASE).

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 arch/arm64/Kconfig              | 11 +++++++
 arch/arm64/Makefile             |  4 +++
 arch/arm64/include/asm/elf.h    |  2 ++
 arch/arm64/kernel/head.S        | 32 ++++++++++++++++++++
 arch/arm64/kernel/vmlinux.lds.S | 16 ++++++++++
 5 files changed, 65 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 141f65ab0ed5..5e155cc7b49f 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -762,6 +762,17 @@ config ARM64_MODULE_PLTS
 	select ARM64_MODULE_CMODEL_LARGE
 	select HAVE_MOD_ARCH_SPECIFIC
 
+config RELOCATABLE
+	bool
+	help
+	  This builds the kernel as a Position Independent Executable (PIE),
+	  which retains all relocation metadata required to relocate the
+	  kernel binary at runtime to a different virtual address than the
+	  address it was linked at.
+	  Since AArch64 uses the RELA relocation format, this requires a
+	  relocation pass at runtime even if the kernel is loaded at the
+	  same address it was linked at.
+
 endmenu
 
 menu "Boot options"
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index a6bba9623836..6b5508a17495 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -15,6 +15,10 @@ CPPFLAGS_vmlinux.lds = -DTEXT_OFFSET=$(TEXT_OFFSET)
 OBJCOPYFLAGS	:=-O binary -R .note -R .note.gnu.build-id -R .comment -S
 GZFLAGS		:=-9
 
+ifneq ($(CONFIG_RELOCATABLE),)
+LDFLAGS_vmlinux		+= -pie
+endif
+
 KBUILD_DEFCONFIG := defconfig
 
 # Check for binutils support for specific extensions
diff --git a/arch/arm64/include/asm/elf.h b/arch/arm64/include/asm/elf.h
index 435f55952e1f..24ed037f09fd 100644
--- a/arch/arm64/include/asm/elf.h
+++ b/arch/arm64/include/asm/elf.h
@@ -77,6 +77,8 @@
 #define R_AARCH64_MOVW_PREL_G2_NC	292
 #define R_AARCH64_MOVW_PREL_G3		293
 
+#define R_AARCH64_RELATIVE		1027
+
 /*
  * These are used to set parameters in the core dumps.
  */
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 4cad8f9f2268..4e69412a7323 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -29,6 +29,7 @@
 #include <asm/asm-offsets.h>
 #include <asm/cache.h>
 #include <asm/cputype.h>
+#include <asm/elf.h>
 #include <asm/kernel-pgtable.h>
 #include <asm/memory.h>
 #include <asm/pgtable-hwdef.h>
@@ -432,6 +433,37 @@ __mmap_switched:
 	bl	__pi_memset
 	dsb	ishst				// Make zero page visible to PTW
 
+#ifdef CONFIG_RELOCATABLE
+
+	/*
+	 * Iterate over each entry in the relocation table, and apply the
+	 * relocations in place.
+	 */
+	adr_l	x8, __dynsym_start		// start of symbol table
+	adr_l	x9, __reloc_start		// start of reloc table
+	adr_l	x10, __reloc_end		// end of reloc table
+
+0:	cmp	x9, x10
+	b.hs	2f
+	ldp	x11, x12, [x9], #24
+	ldr	x13, [x9, #-8]
+	cmp	w12, #R_AARCH64_RELATIVE
+	b.ne	1f
+	str	x13, [x11]
+	b	0b
+
+1:	cmp	w12, #R_AARCH64_ABS64
+	b.ne	0b
+	add	x12, x12, x12, lsl #1		// symtab offset: 24x top word
+	add	x12, x8, x12, lsr #(32 - 3)	// ... shifted into bottom word
+	ldr	x15, [x12, #8]			// Elf64_Sym::st_value
+	add	x15, x13, x15
+	str	x15, [x11]
+	b	0b
+
+2:
+#endif
+
 	adr_l	sp, initial_sp, x4
 	mov	x4, sp
 	and	x4, x4, #~(THREAD_SIZE - 1)
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index 282e3e64a17e..e3f6cd740ea3 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -87,6 +87,7 @@ SECTIONS
 		EXIT_CALL
 		*(.discard)
 		*(.discard.*)
+		*(.interp .dynamic)
 	}
 
 	. = KIMAGE_VADDR + TEXT_OFFSET;
@@ -149,6 +150,21 @@ SECTIONS
 	.altinstr_replacement : {
 		*(.altinstr_replacement)
 	}
+	.rela : ALIGN(8) {
+		__reloc_start = .;
+		*(.rela .rela*)
+		__reloc_end = .;
+	}
+	.dynsym : ALIGN(8) {
+		__dynsym_start = .;
+		*(.dynsym)
+	}
+	.dynstr : {
+		*(.dynstr)
+	}
+	.hash : {
+		*(.hash)
+	}
 
 	. = ALIGN(PAGE_SIZE);
 	__init_end = .;
-- 
2.5.0

[PATCH v5sub2 7/8] arm64: add support for kernel ASLR

[Ard Biesheuvel]

This adds support for KASLR is implemented, based on entropy provided by
the bootloader in the /chosen/kaslr-seed DT property. Depending on the size
of the address space (VA_BITS) and the page size, the entropy in the
virtual displacement is up to 13 bits (16k/2 levels) and up to 25 bits (all
4 levels), with the sidenote that displacements that result in the kernel
image straddling a 1GB/32MB/512MB alignment boundary (for 4KB/16KB/64KB
granule kernels, respectively) are not allowed, and will be rounded up to
an acceptable value.

The module region is randomized by choosing a page aligned 128 MB region
inside the interval [_etext - 128 MB, _stext + 128 MB). This gives between
10 and 14 bits of entropy (depending on page size), independently of the
kernel randomization, but still guarantees that modules are within the
range of relative branch and jump instructions (with the caveat that, since
the module region is shared with other uses of the vmalloc area, modules
may need to be loaded further away if the module region is exhausted)

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 arch/arm64/Kconfig              |  14 ++
 arch/arm64/include/asm/memory.h |   5 +-
 arch/arm64/kernel/Makefile      |   1 +
 arch/arm64/kernel/head.S        |  59 +++++++--
 arch/arm64/kernel/kaslr.c       | 139 ++++++++++++++++++++
 arch/arm64/kernel/module.c      |   8 +-
 arch/arm64/kernel/setup.c       |  29 ++++
 arch/arm64/mm/mmu.c             |  33 +++--
 8 files changed, 268 insertions(+), 20 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 5e155cc7b49f..e3049d5c1246 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -773,6 +773,20 @@ config RELOCATABLE
 	  relocation pass at runtime even if the kernel is loaded at the
 	  same address it was linked at.
 
+config RANDOMIZE_BASE
+	bool "Randomize the address of the kernel image"
+	select ARM64_MODULE_PLTS
+	select RELOCATABLE
+	help
+	  Randomizes the virtual address at which the kernel image is
+	  loaded, as a security feature that deters exploit attempts
+	  relying on knowledge of the location of kernel internals.
+
+	  It is the bootloader's job to provide entropy, by passing a
+	  random u64 value in /chosen/kaslr-seed at kernel entry.
+
+	  If unsure, say N.
+
 endmenu
 
 menu "Boot options"
diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h
index 61005e7dd6cb..083361531a61 100644
--- a/arch/arm64/include/asm/memory.h
+++ b/arch/arm64/include/asm/memory.h
@@ -52,7 +52,7 @@
 #define KIMAGE_VADDR		(MODULES_END)
 #define MODULES_END		(MODULES_VADDR + MODULES_VSIZE)
 #define MODULES_VADDR		(VA_START + KASAN_SHADOW_SIZE)
-#define MODULES_VSIZE		(SZ_64M)
+#define MODULES_VSIZE		(SZ_128M)
 #define PCI_IO_END		(PAGE_OFFSET - SZ_2M)
 #define PCI_IO_START		(PCI_IO_END - PCI_IO_SIZE)
 #define FIXADDR_TOP		(PCI_IO_START - SZ_2M)
@@ -127,6 +127,9 @@ extern phys_addr_t		memstart_addr;
 /* PHYS_OFFSET - the physical address of the start of memory. */
 #define PHYS_OFFSET		({ memstart_addr; })
 
+/* the virtual base of the kernel image (minus TEXT_OFFSET) */
+extern u64			kimage_vaddr;
+
 /* the offset between the kernel virtual and physical mappings */
 extern u64			kimage_voffset;
 
diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
index e2f0a755beaa..c9aaecddb941 100644
--- a/arch/arm64/kernel/Makefile
+++ b/arch/arm64/kernel/Makefile
@@ -43,6 +43,7 @@ arm64-obj-$(CONFIG_PCI)			+= pci.o
 arm64-obj-$(CONFIG_ARMV8_DEPRECATED)	+= armv8_deprecated.o
 arm64-obj-$(CONFIG_ACPI)		+= acpi.o
 arm64-obj-$(CONFIG_PARAVIRT)		+= paravirt.o
+arm64-obj-$(CONFIG_RANDOMIZE_BASE)	+= kaslr.o
 
 obj-y					+= $(arm64-obj-y) vdso/
 obj-m					+= $(arm64-obj-m)
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 4e69412a7323..319f896c6e74 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -210,6 +210,7 @@ section_table:
 ENTRY(stext)
 	bl	preserve_boot_args
 	bl	el2_setup			// Drop to EL1, w20=cpu_boot_mode
+	mov	x23, xzr			// KASLR offset, defaults to 0
 	adrp	x24, __PHYS_OFFSET
 	bl	set_cpu_boot_mode_flag
 	bl	__create_page_tables		// x25=TTBR0, x26=TTBR1
@@ -313,7 +314,7 @@ ENDPROC(preserve_boot_args)
 __create_page_tables:
 	adrp	x25, idmap_pg_dir
 	adrp	x26, swapper_pg_dir
-	mov	x27, lr
+	mov	x28, lr
 
 	/*
 	 * Invalidate the idmap and swapper page tables to avoid potential
@@ -392,6 +393,7 @@ __create_page_tables:
 	 */
 	mov	x0, x26				// swapper_pg_dir
 	ldr	x5, =KIMAGE_VADDR
+	add	x5, x5, x23			// add KASLR displacement
 	create_pgd_entry x0, x5, x3, x6
 	ldr	w6, kernel_img_size
 	add	x6, x6, x5
@@ -408,8 +410,7 @@ __create_page_tables:
 	dmb	sy
 	bl	__inval_cache_range
 
-	mov	lr, x27
-	ret
+	ret	x28
 ENDPROC(__create_page_tables)
 
 kernel_img_size:
@@ -421,6 +422,7 @@ kernel_img_size:
  */
 	.set	initial_sp, init_thread_union + THREAD_START_SP
 __mmap_switched:
+	mov	x28, lr				// preserve LR
 	adr_l	x8, vectors			// load VBAR_EL1 with virtual
 	msr	vbar_el1, x8			// vector table address
 	isb
@@ -449,19 +451,26 @@ __mmap_switched:
 	ldr	x13, [x9, #-8]
 	cmp	w12, #R_AARCH64_RELATIVE
 	b.ne	1f
-	str	x13, [x11]
+	add	x13, x13, x23			// relocate
+	str	x13, [x11, x23]
 	b	0b
 
 1:	cmp	w12, #R_AARCH64_ABS64
 	b.ne	0b
 	add	x12, x12, x12, lsl #1		// symtab offset: 24x top word
 	add	x12, x8, x12, lsr #(32 - 3)	// ... shifted into bottom word
+	ldrsh	w14, [x12, #6]			// Elf64_Sym::st_shndx
 	ldr	x15, [x12, #8]			// Elf64_Sym::st_value
+	cmp	w14, #-0xf			// SHN_ABS (0xfff1) ?
+	add	x14, x15, x23			// relocate
+	csel	x15, x14, x15, ne
 	add	x15, x13, x15
-	str	x15, [x11]
+	str	x15, [x11, x23]
 	b	0b
 
-2:
+2:	adr_l	x8, kimage_vaddr		// make relocated kimage_vaddr
+	dc	cvac, x8			// value visible to secondaries
+	dsb	sy				// with MMU off
 #endif
 
 	adr_l	sp, initial_sp, x4
@@ -470,7 +479,7 @@ __mmap_switched:
 	msr	sp_el0, x4			// Save thread_info
 	str_l	x21, __fdt_pointer, x5		// Save FDT pointer
 
-	ldr	x4, =KIMAGE_VADDR		// Save the offset between
+	ldr_l	x4, kimage_vaddr		// Save the offset between
 	sub	x4, x4, x24			// the kernel virtual and
 	str_l	x4, kimage_voffset, x5		// physical mappings
 
@@ -478,6 +487,16 @@ __mmap_switched:
 #ifdef CONFIG_KASAN
 	bl	kasan_early_init
 #endif
+#ifdef CONFIG_RANDOMIZE_BASE
+	cbnz	x23, 0f				// already running randomized?
+	mov	x0, x21				// pass FDT address in x0
+	bl	kaslr_early_init		// parse FDT for KASLR options
+	cbz	x0, 0f				// KASLR disabled? just proceed
+	mov	x23, x0				// record KASLR offset
+	ret	x28				// we must enable KASLR, return
+						// to __enable_mmu()
+0:
+#endif
 	b	start_kernel
 ENDPROC(__mmap_switched)
 
@@ -486,6 +505,10 @@ ENDPROC(__mmap_switched)
  * hotplug and needs to have the same protections as the text region
  */
 	.section ".text","ax"
+
+ENTRY(kimage_vaddr)
+	.quad		_text - TEXT_OFFSET
+
 /*
  * If we're fortunate enough to boot at EL2, ensure that the world is
  * sane before dropping to EL1.
@@ -651,7 +674,7 @@ ENTRY(secondary_startup)
 	adrp	x26, swapper_pg_dir
 	bl	__cpu_setup			// initialise processor
 
-	ldr	x8, =KIMAGE_VADDR
+	ldr	x8, kimage_vaddr
 	ldr	w9, 0f
 	sub	x27, x8, w9, sxtw		// address to jump to after enabling the MMU
 	b	__enable_mmu
@@ -684,6 +707,7 @@ ENDPROC(__secondary_switched)
  */
 	.section	".idmap.text", "ax"
 __enable_mmu:
+	mrs	x18, sctlr_el1			// preserve old SCTLR_EL1 value
 	mrs	x1, ID_AA64MMFR0_EL1
 	ubfx	x2, x1, #ID_AA64MMFR0_TGRAN_SHIFT, 4
 	cmp	x2, #ID_AA64MMFR0_TGRAN_SUPPORTED
@@ -701,6 +725,25 @@ __enable_mmu:
 	ic	iallu
 	dsb	nsh
 	isb
+#ifdef CONFIG_RANDOMIZE_BASE
+	mov	x19, x0				// preserve new SCTLR_EL1 value
+	blr	x27
+
+	/*
+	 * If we return here, we have a KASLR displacement in x23 which we need
+	 * to take into account by discarding the current kernel mapping and
+	 * creating a new one.
+	 */
+	msr	sctlr_el1, x18			// disable the MMU
+	isb
+	bl	__create_page_tables		// recreate kernel mapping
+
+	msr	sctlr_el1, x19			// re-enable the MMU
+	isb
+	ic	ialluis				// flush instructions fetched
+	isb					// via old mapping
+	add	x27, x27, x23			// relocated __mmap_switched
+#endif
 	br	x27
 ENDPROC(__enable_mmu)
 
diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c
new file mode 100644
index 000000000000..b0bf628ba51f
--- /dev/null
+++ b/arch/arm64/kernel/kaslr.c
@@ -0,0 +1,139 @@
+/*
+ * Copyright (C) 2016 Linaro Ltd <ard.biesheuvel@linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/crc32.h>
+#include <linux/init.h>
+#include <linux/libfdt.h>
+#include <linux/mm_types.h>
+#include <linux/sched.h>
+#include <linux/types.h>
+
+#include <asm/fixmap.h>
+#include <asm/kernel-pgtable.h>
+#include <asm/memory.h>
+#include <asm/mmu.h>
+#include <asm/pgtable.h>
+#include <asm/sections.h>
+
+u32 __read_mostly module_load_offset;
+
+static __init u64 get_kaslr_seed(void *fdt)
+{
+	int node, len;
+	u64 *prop;
+	u64 ret;
+
+	node = fdt_path_offset(fdt, "/chosen");
+	if (node < 0)
+		return 0;
+
+	prop = fdt_getprop_w(fdt, node, "kaslr-seed", &len);
+	if (!prop || len != sizeof(u64))
+		return 0;
+
+	ret = fdt64_to_cpu(*prop);
+	*prop = 0;
+	return ret;
+}
+
+static __init const u8 *get_cmdline(void *fdt)
+{
+	static __initconst const u8 default_cmdline[] = CONFIG_CMDLINE;
+
+	if (!IS_ENABLED(CONFIG_CMDLINE_FORCE)) {
+		int node;
+		const u8 *prop;
+
+		node = fdt_path_offset(fdt, "/chosen");
+		if (node < 0)
+			goto out;
+
+		prop = fdt_getprop(fdt, node, "bootargs", NULL);
+		if (!prop)
+			goto out;
+		return prop;
+	}
+out:
+	return default_cmdline;
+}
+
+extern void *__init __fixmap_remap_fdt(phys_addr_t dt_phys, int *size,
+				       pgprot_t prot);
+
+/*
+ * This routine will be executed with the kernel mapped at its default virtual
+ * address, and if it returns successfully, the kernel will be remapped, and
+ * start_kernel() will be executed from a randomized virtual offset. The
+ * relocation will result in all absolute references (e.g., static variables
+ * containing function pointers) to be reinitialized, and zero-initialized
+ * .bss variables will be reset to 0.
+ */
+u64 __init kaslr_early_init(u64 dt_phys)
+{
+	void *fdt;
+	u64 seed, offset, mask, module_range;
+	const u8 *cmdline, *str;
+	int size;
+
+	/*
+	 * Try to map the FDT early. If this fails, we simply bail,
+	 * and proceed with KASLR disabled. We will make another
+	 * attempt@mapping the FDT in setup_machine()
+	 */
+	early_fixmap_init();
+	fdt = __fixmap_remap_fdt(dt_phys, &size, PAGE_KERNEL);
+	if (!fdt)
+		return 0;
+
+	/*
+	 * Retrieve (and wipe) the seed from the FDT
+	 */
+	seed = get_kaslr_seed(fdt);
+	if (!seed)
+		return 0;
+
+	/*
+	 * Check if 'nokaslr' appears on the command line, and
+	 * return 0 if that is the case.
+	 */
+	cmdline = get_cmdline(fdt);
+	str = strstr(cmdline, "nokaslr");
+	if (str == cmdline || (str > cmdline && *(str - 1) == ' '))
+		return 0;
+
+	/*
+	 * OK, so we are proceeding with KASLR enabled. Calculate a suitable
+	 * kernel image offset from the seed. Let's place the kernel in the
+	 * lower half of the VMALLOC area (VA_BITS - 2).
+	 * Even if we could randomize at page granularity for 16k and 64k pages,
+	 * let's always round to 2 MB so we don't interfere with the ability to
+	 * map using contiguous PTEs
+	 */
+	mask = ((1UL << (VA_BITS - 2)) - 1) & ~(SZ_2M - 1);
+	offset = seed & mask;
+
+	/*
+	 * The kernel Image should not extend across a 1GB/32MB/512MB alignment
+	 * boundary (for 4KB/16KB/64KB granule kernels, respectively). If this
+	 * happens, increase the KASLR offset by the size of the kernel image.
+	 */
+	if ((((u64)_text + offset) >> SWAPPER_TABLE_SHIFT) !=
+	    (((u64)_end + offset) >> SWAPPER_TABLE_SHIFT))
+		offset = (offset + (u64)(_end - _text)) & mask;
+
+	/*
+	 * Randomize the module region, by setting module_load_offset to
+	 * a PAGE_SIZE multiple in the interval [0, module_range). This
+	 * ensures that the resulting region still covers [_stext, _etext],
+	 * and that all relative branches can be resolved without veneers.
+	 */
+	module_range = MODULES_VSIZE - (u64)(_etext - _stext);
+	module_load_offset = ((module_range * (u16)seed) >> 16) & PAGE_MASK;
+
+	return offset;
+}
diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c
index 84113d3e1df1..54702d456680 100644
--- a/arch/arm64/kernel/module.c
+++ b/arch/arm64/kernel/module.c
@@ -33,8 +33,14 @@
 void *module_alloc(unsigned long size)
 {
 	void *p;
+	u64 base = (u64)_etext - MODULES_VSIZE;
 
-	p = __vmalloc_node_range(size, MODULE_ALIGN, MODULES_VADDR, MODULES_END,
+	if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) {
+		extern u32 module_load_offset;
+		base += module_load_offset;
+	}
+
+	p = __vmalloc_node_range(size, MODULE_ALIGN, base, base + MODULES_VSIZE,
 				GFP_KERNEL, PAGE_KERNEL_EXEC, 0,
 				NUMA_NO_NODE, __builtin_return_address(0));
 
diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index cfed56f0ad26..42371f69def3 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -388,3 +388,32 @@ static int __init topology_init(void)
 	return 0;
 }
 subsys_initcall(topology_init);
+
+/*
+ * Dump out kernel offset information on panic.
+ */
+static int dump_kernel_offset(struct notifier_block *self, unsigned long v,
+			      void *p)
+{
+	u64 const kaslr_offset = kimage_vaddr - KIMAGE_VADDR;
+
+	if (IS_ENABLED(CONFIG_RANDOMIZE_BASE) && kaslr_offset > 0) {
+		pr_emerg("Kernel Offset: 0x%llx from 0x%lx\n",
+			 kaslr_offset, KIMAGE_VADDR);
+	} else {
+		pr_emerg("Kernel Offset: disabled\n");
+	}
+	return 0;
+}
+
+static struct notifier_block kernel_offset_notifier = {
+	.notifier_call = dump_kernel_offset
+};
+
+static int __init register_kernel_offset_dumper(void)
+{
+	atomic_notifier_chain_register(&panic_notifier_list,
+				       &kernel_offset_notifier);
+	return 0;
+}
+__initcall(register_kernel_offset_dumper);
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 8dda38378959..5d7e0b801ab7 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -636,7 +636,8 @@ void __init early_fixmap_init(void)
 	unsigned long addr = FIXADDR_START;
 
 	pgd = pgd_offset_k(addr);
-	if (CONFIG_PGTABLE_LEVELS > 3 && !pgd_none(*pgd)) {
+	if (CONFIG_PGTABLE_LEVELS > 3 &&
+	    !(pgd_none(*pgd) || pgd_page_paddr(*pgd) == __pa(bm_pud))) {
 		/*
 		 * We only end up here if the kernel mapping and the fixmap
 		 * share the top level pgd entry, which should only happen on
@@ -693,11 +694,10 @@ void __set_fixmap(enum fixed_addresses idx,
 	}
 }
 
-void *__init fixmap_remap_fdt(phys_addr_t dt_phys)
+void *__init __fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot)
 {
 	const u64 dt_virt_base = __fix_to_virt(FIX_FDT);
-	pgprot_t prot = PAGE_KERNEL_RO;
-	int size, offset;
+	int offset;
 	void *dt_virt;
 
 	/*
@@ -736,16 +736,29 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys)
 	if (fdt_check_header(dt_virt) != 0)
 		return NULL;
 
-	size = fdt_totalsize(dt_virt);
-	if (size > MAX_FDT_SIZE)
+	*size = fdt_totalsize(dt_virt);
+	if (*size > MAX_FDT_SIZE)
 		return NULL;
 
-	if (offset + size > SWAPPER_BLOCK_SIZE)
-		create_mapping(round_down(dt_phys, SWAPPER_BLOCK_SIZE), dt_virt_base,
-			       round_up(offset + size, SWAPPER_BLOCK_SIZE), prot);
+	if (offset + *size > SWAPPER_BLOCK_SIZE)
+		create_mapping(round_down(dt_phys, SWAPPER_BLOCK_SIZE),
+			       dt_virt_base,
+			       round_up(offset + *size, SWAPPER_BLOCK_SIZE),
+			       prot);
 
-	memblock_reserve(dt_phys, size);
+	return dt_virt;
+}
 
+void *__init fixmap_remap_fdt(phys_addr_t dt_phys)
+{
+	void *dt_virt;
+	int size;
+
+	dt_virt = __fixmap_remap_fdt(dt_phys, &size, PAGE_KERNEL_RO);
+	if (!dt_virt)
+		return NULL;
+
+	memblock_reserve(dt_phys, size);
 	return dt_virt;
 }
 
-- 
2.5.0

[PATCH v5sub2 8/8] arm64: kaslr: randomize the linear region

[Ard Biesheuvel]

When KASLR is enabled (CONFIG_RANDOMIZE_BASE=y), and entropy has been
provided by the bootloader, randomize the placement of RAM inside the
linear region if sufficient space is available. For instance, on a 4KB
granule/3 levels kernel, the linear region is 256 GB in size, and we can
choose any 1 GB aligned offset that is far enough from the top of the
address space to fit the distance between the start of the lowest memblock
and the top of the highest memblock.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 arch/arm64/kernel/kaslr.c |  4 ++++
 arch/arm64/mm/init.c      | 22 ++++++++++++++++++--
 2 files changed, 24 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c
index b0bf628ba51f..c96f63670537 100644
--- a/arch/arm64/kernel/kaslr.c
+++ b/arch/arm64/kernel/kaslr.c
@@ -21,6 +21,7 @@
 #include <asm/sections.h>
 
 u32 __read_mostly module_load_offset;
+u16 __initdata memstart_offset_seed;
 
 static __init u64 get_kaslr_seed(void *fdt)
 {
@@ -135,5 +136,8 @@ u64 __init kaslr_early_init(u64 dt_phys)
 	module_range = MODULES_VSIZE - (u64)(_etext - _stext);
 	module_load_offset = ((module_range * (u16)seed) >> 16) & PAGE_MASK;
 
+	/* use the top 16 bits to randomize the linear region */
+	memstart_offset_seed = seed >> 48;
+
 	return offset;
 }
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index e8e853a1024c..01fdb3d04bba 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -215,6 +215,23 @@ void __init arm64_memblock_init(void)
 	if (memblock_end_of_DRAM() > linear_region_size)
 		memblock_remove(0, memblock_end_of_DRAM() - linear_region_size);
 
+	if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) {
+		extern u16 memstart_offset_seed;
+		u64 range = linear_region_size -
+			    (memblock_end_of_DRAM() - memblock_start_of_DRAM());
+
+		/*
+		 * If the size of the linear region exceeds, by a sufficient
+		 * margin, the size of the region that the available physical
+		 * memory spans, randomize the linear region as well.
+		 */
+		if (memstart_offset_seed > 0 && range >= ARM64_MEMSTART_ALIGN) {
+			range = range / ARM64_MEMSTART_ALIGN + 1;
+			memstart_addr -= ARM64_MEMSTART_ALIGN *
+					 ((range * memstart_offset_seed) >> 16);
+		}
+	}
+
 	if (memory_limit != (phys_addr_t)ULLONG_MAX) {
 		u64 kbase = round_down(__pa(_text), MIN_KIMG_ALIGN);
 		u64 kend = PAGE_ALIGN(__pa(_end));
@@ -399,12 +416,13 @@ void __init mem_init(void)
 #ifdef CONFIG_SPARSEMEM_VMEMMAP
 		  MLG((unsigned long)vmemmap,
 		      (unsigned long)vmemmap + VMEMMAP_SIZE),
-		  MLM((unsigned long)virt_to_page(PAGE_OFFSET),
+		  MLM((unsigned long)phys_to_page(memblock_start_of_DRAM()),
 		      (unsigned long)virt_to_page(high_memory)),
 #endif
 		  MLK(FIXADDR_START, FIXADDR_TOP),
 		  MLM(PCI_IO_START, PCI_IO_END),
-		  MLM(PAGE_OFFSET, (unsigned long)high_memory));
+		  MLM(__phys_to_virt(memblock_start_of_DRAM()),
+		      (unsigned long)high_memory));
 
 #undef MLK
 #undef MLM
-- 
2.5.0

[PATCH v5sub2 0/8] arm64: implement virtual KASLR

[Ard Biesheuvel]

On 1 February 2016 at 14:09, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> At the request of Catalin, this series has been split off from my series
> 'arm64: implement support for KASLR v4' [1]. This sub-series deals with
> creating a relocatable binary, and randomizing the virtual placement of
> the kernel itself, the modules and the linear region based on entropy
> supplied by the bootloader in the /chosen/kaslr-seed DT property.
>
> Changes since v4:
> - add randomization of the linear region, i.e., if the linear region is
>   substantially larger than the space spanned by RAM, the mapping of RAM
>   is moved to a random offset inside the linear region.
> - dropped the CRC check in kaslr_early_init(), since the only code that may
>   modify .data through __fixmap_remap_fdt() is instrumentation that should
>   deal with that correctly (For instance, the branch profiling records taken
>   branches in structs allocated in .data, and these counts will not be reset
>   to 0 between the first and the second call to __mmap_switched())
> - add Mark's ack to patch #4
>

NOTE: I have omitted the relative kallsyms and relative extable
patches, since they already queued in akpm's tree. However, while the
kallsyms patch is merely an optimization, the extable patches are in
fact required for correct operation, since the build time sorting does
not work on absolute extables (this is due to the fact that the
resolution of the relocations is deferred until runtime, and so the
extable addresses are all zero at sorting time)

http://ozlabs.org/~akpm/mmotm/broken-out/extable-add-support-for-relative-extables-to-search-and-sort-routines.patch
http://ozlabs.org/~akpm/mmotm/broken-out/arm64-switch-to-relative-exception-tables.patch



> Patch #1 adds support for resolving cross-module branches via veneers if the
> module is located too far from the target. This should not be necessary in the
> common case even with KASRL enabled, but since the space around the kernel image
> will no longer be strictly reserved for modules, there is a slight chance that
> the region may be exhausted, in which case modules will be moved further away
> from the kernel (and each other)
>
> Patches #2 and #3 deal with the fact that relocations of 64-bit values are
> deferred to runtime when building a PIE executable, and some values are required
> before the relocation routine executes (either in the code, or as an externally
> visible constant in the Image header)
>
> Patch #4 rehuffles asm/elf.h so we can use its preprocessor constants in head.S
>
> Patch #5 and #6 implement building vmlinux as a PIE (Position Independent
> Executable) binary, which is emitted with relocation information that is
> processed by the program itself in the early boot code.
>
> Patch #7 implements KASLR, i.e., CONFIG_RANDOMIZE_BASE for the core kernel and
> for the module region.
>
> Patch #8 implements KASLR for the linear region.
>
> [1] http://thread.gmane.org/gmane.linux.kernel/2135931
>
> Ard Biesheuvel (8):
>   arm64: add support for module PLTs
>   arm64: avoid R_AARCH64_ABS64 relocations for Image header fields
>   arm64: avoid dynamic relocations in early boot code
>   arm64: make asm/elf.h available to asm files
>   scripts/sortextable: add support for ET_DYN binaries
>   arm64: add support for building vmlinux as a relocatable PIE binary
>   arm64: add support for kernel ASLR
>   arm64: kaslr: randomize the linear region
>
>  arch/arm64/Kconfig                 |  34 ++++
>  arch/arm64/Makefile                |  10 +-
>  arch/arm64/include/asm/assembler.h |  11 ++
>  arch/arm64/include/asm/elf.h       |  24 ++-
>  arch/arm64/include/asm/memory.h    |   5 +-
>  arch/arm64/include/asm/module.h    |  11 ++
>  arch/arm64/kernel/Makefile         |   2 +
>  arch/arm64/kernel/efi-entry.S      |   2 +-
>  arch/arm64/kernel/head.S           | 128 +++++++++++--
>  arch/arm64/kernel/image.h          |  32 ++--
>  arch/arm64/kernel/kaslr.c          | 143 ++++++++++++++
>  arch/arm64/kernel/module-plts.c    | 201 ++++++++++++++++++++
>  arch/arm64/kernel/module.c         |  20 +-
>  arch/arm64/kernel/module.lds       |   3 +
>  arch/arm64/kernel/setup.c          |  29 +++
>  arch/arm64/kernel/vmlinux.lds.S    |  16 ++
>  arch/arm64/mm/init.c               |  22 ++-
>  arch/arm64/mm/mmu.c                |  33 +++-
>  scripts/sortextable.c              |   8 +-
>  19 files changed, 672 insertions(+), 62 deletions(-)
>  create mode 100644 arch/arm64/kernel/kaslr.c
>  create mode 100644 arch/arm64/kernel/module-plts.c
>  create mode 100644 arch/arm64/kernel/module.lds
>
> --
> 2.5.0
>

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Catalin Marinas]

On Mon, Feb 01, 2016 at 02:09:31PM +0100, Ard Biesheuvel wrote:
> This adds support for emitting PLTs at module load time for relative
> branches that are out of range. This is a prerequisite for KASLR, which
> may place the kernel and the modules anywhere in the vmalloc area,
> making it more likely that branch target offsets exceed the maximum
> range of +/- 128 MB.

Any downside to trying to keep the kernel+modules coupled together so
that we avoid the PLT?

-- 
Catalin

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Ard Biesheuvel]

On 4 February 2016 at 16:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Mon, Feb 01, 2016 at 02:09:31PM +0100, Ard Biesheuvel wrote:
>> This adds support for emitting PLTs at module load time for relative
>> branches that are out of range. This is a prerequisite for KASLR, which
>> may place the kernel and the modules anywhere in the vmalloc area,
>> making it more likely that branch target offsets exceed the maximum
>> range of +/- 128 MB.
>
> Any downside to trying to keep the kernel+modules coupled together so
> that we avoid the PLT?
>

First of all, note that it is unlikely that the PLTs are ever required
in practice, considering that either
a) the kernel is loaded at the default location right at the base of
the vmalloc range, and in this case, the module space is reserved for
modules only, or
b) the kernel is loaded at some random offset in the 240+ GB vmalloc
space, and it is unlikely that all VMA space around the kernel happens
to be given out to non-randomized vmalloc/ioremap allocations

So while this patch was a requirement at first (since in the first
version of the series, the modules always remained below the vmalloc
area so they would always be out of range when KASLR was enabled), in
the current series, it is simply a last resort (although still
required) to make sure that modules can be loaded far away from the
kernel in the unlikely event that all VMA space in close proximity is
given to another user.

Reserving a dedicated module region around the kernel while it is
loaded at a random offset is not straight forward, since it affects
all other users of the vmalloc area. So generic vmalloc/vmap/ioremap
code would need to be updated to disregard the area around the kernel,
since simply reserving the VMA area for the modules would make it
inaccessible to module_alloc() as well (unless we reserve the whole
region and manage the module allocations separately)

So the bottom line is that it may be possible, but it is unlikely to
be worth the effort imo

-- 
Ard.

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Catalin Marinas]

On Thu, Feb 04, 2016 at 04:31:59PM +0100, Ard Biesheuvel wrote:
> On 4 February 2016 at 16:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Mon, Feb 01, 2016 at 02:09:31PM +0100, Ard Biesheuvel wrote:
> >> This adds support for emitting PLTs at module load time for relative
> >> branches that are out of range. This is a prerequisite for KASLR, which
> >> may place the kernel and the modules anywhere in the vmalloc area,
> >> making it more likely that branch target offsets exceed the maximum
> >> range of +/- 128 MB.
> >
> > Any downside to trying to keep the kernel+modules coupled together so
> > that we avoid the PLT?
> 
> First of all, note that it is unlikely that the PLTs are ever required
> in practice, considering that either
> a) the kernel is loaded at the default location right at the base of
> the vmalloc range, and in this case, the module space is reserved for
> modules only, or
> b) the kernel is loaded at some random offset in the 240+ GB vmalloc
> space, and it is unlikely that all VMA space around the kernel happens
> to be given out to non-randomized vmalloc/ioremap allocations

My worry is that we merge some code that's rarely tested.

> So the bottom line is that it may be possible, but it is unlikely to
> be worth the effort imo

The alternative of having our own allocator for modules isn't that
appealing either, so let's stick with the PLTs.

-- 
Catalin

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Ard Biesheuvel]

On 5 February 2016 at 16:42, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Thu, Feb 04, 2016 at 04:31:59PM +0100, Ard Biesheuvel wrote:
>> On 4 February 2016 at 16:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> > On Mon, Feb 01, 2016 at 02:09:31PM +0100, Ard Biesheuvel wrote:
>> >> This adds support for emitting PLTs at module load time for relative
>> >> branches that are out of range. This is a prerequisite for KASLR, which
>> >> may place the kernel and the modules anywhere in the vmalloc area,
>> >> making it more likely that branch target offsets exceed the maximum
>> >> range of +/- 128 MB.
>> >
>> > Any downside to trying to keep the kernel+modules coupled together so
>> > that we avoid the PLT?
>>
>> First of all, note that it is unlikely that the PLTs are ever required
>> in practice, considering that either
>> a) the kernel is loaded at the default location right at the base of
>> the vmalloc range, and in this case, the module space is reserved for
>> modules only, or
>> b) the kernel is loaded at some random offset in the 240+ GB vmalloc
>> space, and it is unlikely that all VMA space around the kernel happens
>> to be given out to non-randomized vmalloc/ioremap allocations
>
> My worry is that we merge some code that's rarely tested.
>

I understand. But unfortunately, having corner cases that are unlikely
but not impossible comes with the territory of randomization.

Alternatively, we could take the performance hit if KASLR is in effect
and allocate each module completely randomly as well. This way, the
code is always exercised (for now), and we can always backpedal later
if the performance is measurably worse.

>> So the bottom line is that it may be possible, but it is unlikely to
>> be worth the effort imo
>
> The alternative of having our own allocator for modules isn't that
> appealing either, so let's stick with the PLTs.
>
> --
> Catalin

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Catalin Marinas]

On Fri, Feb 05, 2016 at 04:53:10PM +0100, Ard Biesheuvel wrote:
> On 5 February 2016 at 16:42, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Thu, Feb 04, 2016 at 04:31:59PM +0100, Ard Biesheuvel wrote:
> >> On 4 February 2016 at 16:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
> >> > On Mon, Feb 01, 2016 at 02:09:31PM +0100, Ard Biesheuvel wrote:
> >> >> This adds support for emitting PLTs at module load time for relative
> >> >> branches that are out of range. This is a prerequisite for KASLR, which
> >> >> may place the kernel and the modules anywhere in the vmalloc area,
> >> >> making it more likely that branch target offsets exceed the maximum
> >> >> range of +/- 128 MB.
> >> >
> >> > Any downside to trying to keep the kernel+modules coupled together so
> >> > that we avoid the PLT?
> >>
> >> First of all, note that it is unlikely that the PLTs are ever required
> >> in practice, considering that either
> >> a) the kernel is loaded at the default location right at the base of
> >> the vmalloc range, and in this case, the module space is reserved for
> >> modules only, or
> >> b) the kernel is loaded at some random offset in the 240+ GB vmalloc
> >> space, and it is unlikely that all VMA space around the kernel happens
> >> to be given out to non-randomized vmalloc/ioremap allocations
> >
> > My worry is that we merge some code that's rarely tested.
> 
> I understand. But unfortunately, having corner cases that are unlikely
> but not impossible comes with the territory of randomization.
> 
> Alternatively, we could take the performance hit if KASLR is in effect
> and allocate each module completely randomly as well. This way, the
> code is always exercised (for now), and we can always backpedal later
> if the performance is measurably worse.

I'm fine with this. You can post it as a separate patch that we can
easily revert/modify later (like turning it into a config option).

-- 
Catalin

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Ard Biesheuvel]

On 5 February 2016 at 17:00, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Fri, Feb 05, 2016 at 04:53:10PM +0100, Ard Biesheuvel wrote:
>> On 5 February 2016 at 16:42, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> > On Thu, Feb 04, 2016 at 04:31:59PM +0100, Ard Biesheuvel wrote:
>> >> On 4 February 2016 at 16:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> >> > On Mon, Feb 01, 2016 at 02:09:31PM +0100, Ard Biesheuvel wrote:
>> >> >> This adds support for emitting PLTs at module load time for relative
>> >> >> branches that are out of range. This is a prerequisite for KASLR, which
>> >> >> may place the kernel and the modules anywhere in the vmalloc area,
>> >> >> making it more likely that branch target offsets exceed the maximum
>> >> >> range of +/- 128 MB.
>> >> >
>> >> > Any downside to trying to keep the kernel+modules coupled together so
>> >> > that we avoid the PLT?
>> >>
>> >> First of all, note that it is unlikely that the PLTs are ever required
>> >> in practice, considering that either
>> >> a) the kernel is loaded at the default location right at the base of
>> >> the vmalloc range, and in this case, the module space is reserved for
>> >> modules only, or
>> >> b) the kernel is loaded at some random offset in the 240+ GB vmalloc
>> >> space, and it is unlikely that all VMA space around the kernel happens
>> >> to be given out to non-randomized vmalloc/ioremap allocations
>> >
>> > My worry is that we merge some code that's rarely tested.
>>
>> I understand. But unfortunately, having corner cases that are unlikely
>> but not impossible comes with the territory of randomization.
>>
>> Alternatively, we could take the performance hit if KASLR is in effect
>> and allocate each module completely randomly as well. This way, the
>> code is always exercised (for now), and we can always backpedal later
>> if the performance is measurably worse.
>
> I'm fine with this. You can post it as a separate patch that we can
> easily revert/modify later (like turning it into a config option).
>

OK, I will hack something up

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Catalin Marinas]

On Fri, Feb 05, 2016 at 05:20:14PM +0100, Ard Biesheuvel wrote:
> On 5 February 2016 at 17:00, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Fri, Feb 05, 2016 at 04:53:10PM +0100, Ard Biesheuvel wrote:
> >> On 5 February 2016 at 16:42, Catalin Marinas <catalin.marinas@arm.com> wrote:
> >> > On Thu, Feb 04, 2016 at 04:31:59PM +0100, Ard Biesheuvel wrote:
> >> >> On 4 February 2016 at 16:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
> >> >> > On Mon, Feb 01, 2016 at 02:09:31PM +0100, Ard Biesheuvel wrote:
> >> >> >> This adds support for emitting PLTs at module load time for relative
> >> >> >> branches that are out of range. This is a prerequisite for KASLR, which
> >> >> >> may place the kernel and the modules anywhere in the vmalloc area,
> >> >> >> making it more likely that branch target offsets exceed the maximum
> >> >> >> range of +/- 128 MB.
> >> >> >
> >> >> > Any downside to trying to keep the kernel+modules coupled together so
> >> >> > that we avoid the PLT?
> >> >>
> >> >> First of all, note that it is unlikely that the PLTs are ever required
> >> >> in practice, considering that either
> >> >> a) the kernel is loaded at the default location right at the base of
> >> >> the vmalloc range, and in this case, the module space is reserved for
> >> >> modules only, or
> >> >> b) the kernel is loaded at some random offset in the 240+ GB vmalloc
> >> >> space, and it is unlikely that all VMA space around the kernel happens
> >> >> to be given out to non-randomized vmalloc/ioremap allocations
> >> >
> >> > My worry is that we merge some code that's rarely tested.
> >>
> >> I understand. But unfortunately, having corner cases that are unlikely
> >> but not impossible comes with the territory of randomization.
> >>
> >> Alternatively, we could take the performance hit if KASLR is in effect
> >> and allocate each module completely randomly as well. This way, the
> >> code is always exercised (for now), and we can always backpedal later
> >> if the performance is measurably worse.
> >
> > I'm fine with this. You can post it as a separate patch that we can
> > easily revert/modify later (like turning it into a config option).
> 
> OK, I will hack something up

If it's simpler, you can just add a config option but defaulting to the
full vmalloc space for modules.

-- 
Catalin

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Ard Biesheuvel]

On 5 February 2016 at 17:46, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Fri, Feb 05, 2016 at 05:20:14PM +0100, Ard Biesheuvel wrote:
>> On 5 February 2016 at 17:00, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> > On Fri, Feb 05, 2016 at 04:53:10PM +0100, Ard Biesheuvel wrote:
>> >> On 5 February 2016 at 16:42, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> >> > On Thu, Feb 04, 2016 at 04:31:59PM +0100, Ard Biesheuvel wrote:
>> >> >> On 4 February 2016 at 16:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> >> >> > On Mon, Feb 01, 2016 at 02:09:31PM +0100, Ard Biesheuvel wrote:
>> >> >> >> This adds support for emitting PLTs at module load time for relative
>> >> >> >> branches that are out of range. This is a prerequisite for KASLR, which
>> >> >> >> may place the kernel and the modules anywhere in the vmalloc area,
>> >> >> >> making it more likely that branch target offsets exceed the maximum
>> >> >> >> range of +/- 128 MB.
>> >> >> >
>> >> >> > Any downside to trying to keep the kernel+modules coupled together so
>> >> >> > that we avoid the PLT?
>> >> >>
>> >> >> First of all, note that it is unlikely that the PLTs are ever required
>> >> >> in practice, considering that either
>> >> >> a) the kernel is loaded at the default location right at the base of
>> >> >> the vmalloc range, and in this case, the module space is reserved for
>> >> >> modules only, or
>> >> >> b) the kernel is loaded at some random offset in the 240+ GB vmalloc
>> >> >> space, and it is unlikely that all VMA space around the kernel happens
>> >> >> to be given out to non-randomized vmalloc/ioremap allocations
>> >> >
>> >> > My worry is that we merge some code that's rarely tested.
>> >>
>> >> I understand. But unfortunately, having corner cases that are unlikely
>> >> but not impossible comes with the territory of randomization.
>> >>
>> >> Alternatively, we could take the performance hit if KASLR is in effect
>> >> and allocate each module completely randomly as well. This way, the
>> >> code is always exercised (for now), and we can always backpedal later
>> >> if the performance is measurably worse.
>> >
>> > I'm fine with this. You can post it as a separate patch that we can
>> > easily revert/modify later (like turning it into a config option).
>>
>> OK, I will hack something up
>
> If it's simpler, you can just add a config option but defaulting to the
> full vmalloc space for modules.
>

What would be the simplest is to randomize the 128 MB module region as
a whole, and either put it close to the kernel (which is what I am
doing now), or put it at a random offset inside the vmalloc space, in
which case all branches will be resolved via PLTs. My suggestion to
randomize each module_alloc() call separately is actually not that
straight-forward.

So what I propose now is to keep a single module_load_offset that
randomizes the base of the region, and add a default-n config option
that shrinks the interval it is chosen from so that PLTs are usually
not needed.

-- 
Ard.

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Catalin Marinas]

On Fri, Feb 05, 2016 at 05:54:46PM +0100, Ard Biesheuvel wrote:
> On 5 February 2016 at 17:46, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Fri, Feb 05, 2016 at 05:20:14PM +0100, Ard Biesheuvel wrote:
> >> On 5 February 2016 at 17:00, Catalin Marinas <catalin.marinas@arm.com> wrote:
> >> > On Fri, Feb 05, 2016 at 04:53:10PM +0100, Ard Biesheuvel wrote:
> >> >> On 5 February 2016 at 16:42, Catalin Marinas <catalin.marinas@arm.com> wrote:
> >> >> > On Thu, Feb 04, 2016 at 04:31:59PM +0100, Ard Biesheuvel wrote:
> >> >> >> On 4 February 2016 at 16:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
> >> >> >> > On Mon, Feb 01, 2016 at 02:09:31PM +0100, Ard Biesheuvel wrote:
> >> >> >> >> This adds support for emitting PLTs at module load time for relative
> >> >> >> >> branches that are out of range. This is a prerequisite for KASLR, which
> >> >> >> >> may place the kernel and the modules anywhere in the vmalloc area,
> >> >> >> >> making it more likely that branch target offsets exceed the maximum
> >> >> >> >> range of +/- 128 MB.
> >> >> >> >
> >> >> >> > Any downside to trying to keep the kernel+modules coupled together so
> >> >> >> > that we avoid the PLT?
> >> >> >>
> >> >> >> First of all, note that it is unlikely that the PLTs are ever required
> >> >> >> in practice, considering that either
> >> >> >> a) the kernel is loaded at the default location right at the base of
> >> >> >> the vmalloc range, and in this case, the module space is reserved for
> >> >> >> modules only, or
> >> >> >> b) the kernel is loaded at some random offset in the 240+ GB vmalloc
> >> >> >> space, and it is unlikely that all VMA space around the kernel happens
> >> >> >> to be given out to non-randomized vmalloc/ioremap allocations
> >> >> >
> >> >> > My worry is that we merge some code that's rarely tested.
> >> >>
> >> >> I understand. But unfortunately, having corner cases that are unlikely
> >> >> but not impossible comes with the territory of randomization.
> >> >>
> >> >> Alternatively, we could take the performance hit if KASLR is in effect
> >> >> and allocate each module completely randomly as well. This way, the
> >> >> code is always exercised (for now), and we can always backpedal later
> >> >> if the performance is measurably worse.
> >> >
> >> > I'm fine with this. You can post it as a separate patch that we can
> >> > easily revert/modify later (like turning it into a config option).
> >>
> >> OK, I will hack something up
> >
> > If it's simpler, you can just add a config option but defaulting to the
> > full vmalloc space for modules.
> 
> What would be the simplest is to randomize the 128 MB module region as
> a whole, and either put it close to the kernel (which is what I am
> doing now), or put it at a random offset inside the vmalloc space, in
> which case all branches will be resolved via PLTs. My suggestion to
> randomize each module_alloc() call separately is actually not that
> straight-forward.
> 
> So what I propose now is to keep a single module_load_offset that
> randomizes the base of the region, and add a default-n config option
> that shrinks the interval it is chosen from so that PLTs are usually
> not needed.

Sounds fine.

-- 
Catalin

[PATCH v5sub2 0/8] arm64: implement virtual KASLR

[Catalin Marinas]

On Mon, Feb 01, 2016 at 02:35:03PM +0100, Ard Biesheuvel wrote:
> On 1 February 2016 at 14:09, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> > At the request of Catalin, this series has been split off from my series
> > 'arm64: implement support for KASLR v4' [1]. This sub-series deals with
> > creating a relocatable binary, and randomizing the virtual placement of
> > the kernel itself, the modules and the linear region based on entropy
> > supplied by the bootloader in the /chosen/kaslr-seed DT property.
> >
> > Changes since v4:
> > - add randomization of the linear region, i.e., if the linear region is
> >   substantially larger than the space spanned by RAM, the mapping of RAM
> >   is moved to a random offset inside the linear region.
> > - dropped the CRC check in kaslr_early_init(), since the only code that may
> >   modify .data through __fixmap_remap_fdt() is instrumentation that should
> >   deal with that correctly (For instance, the branch profiling records taken
> >   branches in structs allocated in .data, and these counts will not be reset
> >   to 0 between the first and the second call to __mmap_switched())
> > - add Mark's ack to patch #4
> >
> 
> NOTE: I have omitted the relative kallsyms and relative extable
> patches, since they already queued in akpm's tree. However, while the
> kallsyms patch is merely an optimization, the extable patches are in
> fact required for correct operation, since the build time sorting does
> not work on absolute extables (this is due to the fact that the
> resolution of the relocations is deferred until runtime, and so the
> extable addresses are all zero at sorting time)
> 
> http://ozlabs.org/~akpm/mmotm/broken-out/extable-add-support-for-relative-extables-to-search-and-sort-routines.patch
> http://ozlabs.org/~akpm/mmotm/broken-out/arm64-switch-to-relative-exception-tables.patch

I'm still trying to get my head around how we merge those. Since I
assume akpm will push them during the merging window, part of your code
cannot be tested before.

Can we deselect CONFIG_BUILDTIME_EXTABLE_SORT temporarily while we use
absolute extable addresses? We patch them at boot-time and sort them
at run-time. Once the above patches go in, we can revert to build-time
extable sorting.

-- 
Catalin

[PATCH v5sub2 0/8] arm64: implement virtual KASLR

[Ard Biesheuvel]

On 5 February 2016 at 18:32, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Mon, Feb 01, 2016 at 02:35:03PM +0100, Ard Biesheuvel wrote:
>> On 1 February 2016 at 14:09, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
>> > At the request of Catalin, this series has been split off from my series
>> > 'arm64: implement support for KASLR v4' [1]. This sub-series deals with
>> > creating a relocatable binary, and randomizing the virtual placement of
>> > the kernel itself, the modules and the linear region based on entropy
>> > supplied by the bootloader in the /chosen/kaslr-seed DT property.
>> >
>> > Changes since v4:
>> > - add randomization of the linear region, i.e., if the linear region is
>> >   substantially larger than the space spanned by RAM, the mapping of RAM
>> >   is moved to a random offset inside the linear region.
>> > - dropped the CRC check in kaslr_early_init(), since the only code that may
>> >   modify .data through __fixmap_remap_fdt() is instrumentation that should
>> >   deal with that correctly (For instance, the branch profiling records taken
>> >   branches in structs allocated in .data, and these counts will not be reset
>> >   to 0 between the first and the second call to __mmap_switched())
>> > - add Mark's ack to patch #4
>> >
>>
>> NOTE: I have omitted the relative kallsyms and relative extable
>> patches, since they already queued in akpm's tree. However, while the
>> kallsyms patch is merely an optimization, the extable patches are in
>> fact required for correct operation, since the build time sorting does
>> not work on absolute extables (this is due to the fact that the
>> resolution of the relocations is deferred until runtime, and so the
>> extable addresses are all zero at sorting time)
>>
>> http://ozlabs.org/~akpm/mmotm/broken-out/extable-add-support-for-relative-extables-to-search-and-sort-routines.patch
>> http://ozlabs.org/~akpm/mmotm/broken-out/arm64-switch-to-relative-exception-tables.patch
>
> I'm still trying to get my head around how we merge those. Since I
> assume akpm will push them during the merging window, part of your code
> cannot be tested before.
>

Actually, my original idea was for akpm to take them as a late merge
after rebasing to -rc1, since they touch a variety of architectures,
but I am not sure if that came across.

You could always take the series through your tree instead, I guess?

> Can we deselect CONFIG_BUILDTIME_EXTABLE_SORT temporarily while we use
> absolute extable addresses? We patch them at boot-time and sort them
> at run-time. Once the above patches go in, we can revert to build-time
> extable sorting.
>

I suppose that would be possible, but I simply haven't tried. I
noticed that ppc does not use build time extable sorting, that is why
they don't need to scripts/extable patch while they do use PIE
binaries.

[PATCH v5sub2 0/8] arm64: implement virtual KASLR

[Catalin Marinas]

On Fri, Feb 05, 2016 at 06:38:33PM +0100, Ard Biesheuvel wrote:
> On 5 February 2016 at 18:32, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Mon, Feb 01, 2016 at 02:35:03PM +0100, Ard Biesheuvel wrote:
> >> On 1 February 2016 at 14:09, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> >> > At the request of Catalin, this series has been split off from my series
> >> > 'arm64: implement support for KASLR v4' [1]. This sub-series deals with
> >> > creating a relocatable binary, and randomizing the virtual placement of
> >> > the kernel itself, the modules and the linear region based on entropy
> >> > supplied by the bootloader in the /chosen/kaslr-seed DT property.
> >> >
> >> > Changes since v4:
> >> > - add randomization of the linear region, i.e., if the linear region is
> >> >   substantially larger than the space spanned by RAM, the mapping of RAM
> >> >   is moved to a random offset inside the linear region.
> >> > - dropped the CRC check in kaslr_early_init(), since the only code that may
> >> >   modify .data through __fixmap_remap_fdt() is instrumentation that should
> >> >   deal with that correctly (For instance, the branch profiling records taken
> >> >   branches in structs allocated in .data, and these counts will not be reset
> >> >   to 0 between the first and the second call to __mmap_switched())
> >> > - add Mark's ack to patch #4
> >> >
> >>
> >> NOTE: I have omitted the relative kallsyms and relative extable
> >> patches, since they already queued in akpm's tree. However, while the
> >> kallsyms patch is merely an optimization, the extable patches are in
> >> fact required for correct operation, since the build time sorting does
> >> not work on absolute extables (this is due to the fact that the
> >> resolution of the relocations is deferred until runtime, and so the
> >> extable addresses are all zero at sorting time)
> >>
> >> http://ozlabs.org/~akpm/mmotm/broken-out/extable-add-support-for-relative-extables-to-search-and-sort-routines.patch
> >> http://ozlabs.org/~akpm/mmotm/broken-out/arm64-switch-to-relative-exception-tables.patch
> >
> > I'm still trying to get my head around how we merge those. Since I
> > assume akpm will push them during the merging window, part of your code
> > cannot be tested before.
> 
> Actually, my original idea was for akpm to take them as a late merge
> after rebasing to -rc1, since they touch a variety of architectures,
> but I am not sure if that came across.
> 
> You could always take the series through your tree instead, I guess?

They all have acks from maintainers, so that's an option.

> > Can we deselect CONFIG_BUILDTIME_EXTABLE_SORT temporarily while we use
> > absolute extable addresses? We patch them at boot-time and sort them
> > at run-time. Once the above patches go in, we can revert to build-time
> > extable sorting.
> 
> I suppose that would be possible, but I simply haven't tried. I
> noticed that ppc does not use build time extable sorting, that is why
> they don't need to scripts/extable patch while they do use PIE
> binaries.

I did a quick check with:

--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -22,7 +22,7 @@ config ARM64
 	select ARM_GIC_V3
 	select ARM_GIC_V3_ITS if PCI_MSI
 	select ARM_PSCI_FW
-	select BUILDTIME_EXTABLE_SORT
+	select BUILDTIME_EXTABLE_SORT if !RELOCATABLE
 	select CLONE_BACKWARDS
 	select COMMON_CLK
 	select CPU_PM if (SUSPEND || CPU_IDLE)

and it seems that extable entries end up in the relocation symbols but I
haven't fully checked the patching/sorting yet (it will probably be on
Monday).

-- 
Catalin

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Kees Cook]

On Fri, Feb 5, 2016 at 8:54 AM, Ard Biesheuvel
<ard.biesheuvel@linaro.org> wrote:
> On 5 February 2016 at 17:46, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> On Fri, Feb 05, 2016 at 05:20:14PM +0100, Ard Biesheuvel wrote:
>>> On 5 February 2016 at 17:00, Catalin Marinas <catalin.marinas@arm.com> wrote:
>>> > On Fri, Feb 05, 2016 at 04:53:10PM +0100, Ard Biesheuvel wrote:
>>> >> On 5 February 2016 at 16:42, Catalin Marinas <catalin.marinas@arm.com> wrote:
>>> >> > On Thu, Feb 04, 2016 at 04:31:59PM +0100, Ard Biesheuvel wrote:
>>> >> >> On 4 February 2016 at 16:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
>>> >> >> > On Mon, Feb 01, 2016 at 02:09:31PM +0100, Ard Biesheuvel wrote:
>>> >> >> >> This adds support for emitting PLTs at module load time for relative
>>> >> >> >> branches that are out of range. This is a prerequisite for KASLR, which
>>> >> >> >> may place the kernel and the modules anywhere in the vmalloc area,
>>> >> >> >> making it more likely that branch target offsets exceed the maximum
>>> >> >> >> range of +/- 128 MB.
>>> >> >> >
>>> >> >> > Any downside to trying to keep the kernel+modules coupled together so
>>> >> >> > that we avoid the PLT?
>>> >> >>
>>> >> >> First of all, note that it is unlikely that the PLTs are ever required
>>> >> >> in practice, considering that either
>>> >> >> a) the kernel is loaded at the default location right at the base of
>>> >> >> the vmalloc range, and in this case, the module space is reserved for
>>> >> >> modules only, or
>>> >> >> b) the kernel is loaded at some random offset in the 240+ GB vmalloc
>>> >> >> space, and it is unlikely that all VMA space around the kernel happens
>>> >> >> to be given out to non-randomized vmalloc/ioremap allocations
>>> >> >
>>> >> > My worry is that we merge some code that's rarely tested.
>>> >>
>>> >> I understand. But unfortunately, having corner cases that are unlikely
>>> >> but not impossible comes with the territory of randomization.
>>> >>
>>> >> Alternatively, we could take the performance hit if KASLR is in effect
>>> >> and allocate each module completely randomly as well. This way, the
>>> >> code is always exercised (for now), and we can always backpedal later
>>> >> if the performance is measurably worse.
>>> >
>>> > I'm fine with this. You can post it as a separate patch that we can
>>> > easily revert/modify later (like turning it into a config option).
>>>
>>> OK, I will hack something up
>>
>> If it's simpler, you can just add a config option but defaulting to the
>> full vmalloc space for modules.
>>
>
> What would be the simplest is to randomize the 128 MB module region as
> a whole, and either put it close to the kernel (which is what I am
> doing now), or put it at a random offset inside the vmalloc space, in
> which case all branches will be resolved via PLTs. My suggestion to
> randomize each module_alloc() call separately is actually not that
> straight-forward.

FWIW, it also turns out that this ends up not being very random in a
limited address range. It ends up turning into a greedy-allocation
hole-filling routine, and for a given system ends up being pretty
deterministic.

-Kees

-- 
Kees Cook
Chrome OS & Brillo Security

[PATCH v5sub2 0/8] arm64: implement virtual KASLR

[Kees Cook]

On Fri, Feb 5, 2016 at 9:38 AM, Ard Biesheuvel
<ard.biesheuvel@linaro.org> wrote:
> On 5 February 2016 at 18:32, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> I'm still trying to get my head around how we merge those. Since I
>> assume akpm will push them during the merging window, part of your code
>> cannot be tested before.
>
> Actually, my original idea was for akpm to take them as a late merge
> after rebasing to -rc1, since they touch a variety of architectures,
> but I am not sure if that came across.
>
> You could always take the series through your tree instead, I guess?

Traditionally akpm will de-duplicate patches he's carrying that appear
in another tree. I think it should be okay to carry them in both
places. (Though I'm CCing akpm just to see if I'm talking crazy.)

-Kees

-- 
Kees Cook
Chrome OS & Brillo Security

[PATCH] arm64: allow the module region to be randomized independently

[Ard Biesheuvel]

This patch applies directly onto '[PATCH v5sub2 7/8] arm64: add support for
kernel ASLR' so it can be folded in or applied separately, as desired.

As pointed out by Kees, fully randomizing each module allocation does not add
anything in terms of security, and only hurts performance even more, since it
would cause all inter-module branches to be resolved via veneers as well.

----------8<--------------
This adds the option to randomize the module region independently from the
core kernel, and enables it by default. This makes it less likely that the
location of core kernel data structures can be determined by an adversary,
but causes all function calls from modules into the core kernel to be
resolved via entries in the module PLTs.

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 arch/arm64/Kconfig              | 15 ++++++++
 arch/arm64/include/asm/module.h |  6 ++++
 arch/arm64/kernel/kaslr.c       | 36 +++++++++++++++-----
 arch/arm64/kernel/module.c      |  9 ++---
 4 files changed, 50 insertions(+), 16 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index e3049d5c1246..666aacc4c763 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -787,6 +787,21 @@ config RANDOMIZE_BASE
 
 	  If unsure, say N.
 
+config RANDOMIZE_MODULE_REGION_FULL
+	bool "Randomize the module region independently from the core kernel"
+	depends on RANDOMIZE_BASE
+	default y
+	help
+	  Randomizes the location of the module region without considering the
+	  location of the core kernel. This way, it is impossible for modules
+	  to leak information about the location of core kernel data structures
+	  but it does imply that function calls between modules and the core
+	  kernel will need to be resolved via veneers in the module PLT.
+
+	  When this option is not set, the module region will be randomized over
+	  a limited range that contains the [_stext, _etext] interval of the
+	  core kernel, so branch relocations are always in range.
+
 endmenu
 
 menu "Boot options"
diff --git a/arch/arm64/include/asm/module.h b/arch/arm64/include/asm/module.h
index 8652fb613304..e12af6754634 100644
--- a/arch/arm64/include/asm/module.h
+++ b/arch/arm64/include/asm/module.h
@@ -31,4 +31,10 @@ struct mod_arch_specific {
 u64 module_emit_plt_entry(struct module *mod, const Elf64_Rela *rela,
 			  Elf64_Sym *sym);
 
+#ifdef CONFIG_RANDOMIZE_BASE
+extern u64 module_alloc_base;
+#else
+#define module_alloc_base	((u64)_etext - MODULES_VSIZE)
+#endif
+
 #endif /* __ASM_MODULE_H */
diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c
index b0bf628ba51f..7a40aa4ba93d 100644
--- a/arch/arm64/kernel/kaslr.c
+++ b/arch/arm64/kernel/kaslr.c
@@ -20,7 +20,7 @@
 #include <asm/pgtable.h>
 #include <asm/sections.h>
 
-u32 __read_mostly module_load_offset;
+u64 __read_mostly module_alloc_base;
 
 static __init u64 get_kaslr_seed(void *fdt)
 {
@@ -126,14 +126,32 @@ u64 __init kaslr_early_init(u64 dt_phys)
 	    (((u64)_end + offset) >> SWAPPER_TABLE_SHIFT))
 		offset = (offset + (u64)(_end - _text)) & mask;
 
-	/*
-	 * Randomize the module region, by setting module_load_offset to
-	 * a PAGE_SIZE multiple in the interval [0, module_range). This
-	 * ensures that the resulting region still covers [_stext, _etext],
-	 * and that all relative branches can be resolved without veneers.
-	 */
-	module_range = MODULES_VSIZE - (u64)(_etext - _stext);
-	module_load_offset = ((module_range * (u16)seed) >> 16) & PAGE_MASK;
+	if (IS_ENABLED(CONFIG_RANDOMIZE_MODULE_REGION_FULL)) {
+		/*
+		 * Randomize the module region independently from the core
+		 * kernel. This prevents modules from leaking any information
+		 * about the address of the kernel itself, but results in
+		 * branches between modules and the core kernel that are
+		 * resolved via PLTs. (Branches between modules will be
+		 * resolved normally.)
+		 */
+		module_range = VMALLOC_END - VMALLOC_START - MODULES_VSIZE;
+		module_alloc_base = VMALLOC_START;
+	} else {
+		/*
+		 * Randomize the module region by setting module_alloc_base to
+		 * a PAGE_SIZE multiple in the range [_etext - MODULES_VSIZE,
+		 * _stext) . This guarantees that the resulting region still
+		 * covers [_stext, _etext], and that all relative branches can
+		 * be resolved without veneers.
+		 */
+		module_range = MODULES_VSIZE - (u64)(_etext - _stext);
+		module_alloc_base = (u64)_etext + offset - MODULES_VSIZE;
+	}
+
+	/* use the lower 21 bits to randomize the base of the module region */
+	module_alloc_base += (module_range * (seed & ((1 << 21) - 1))) >> 21;
+	module_alloc_base &= PAGE_MASK;
 
 	return offset;
 }
diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c
index 54702d456680..dfa1ffaa9844 100644
--- a/arch/arm64/kernel/module.c
+++ b/arch/arm64/kernel/module.c
@@ -33,14 +33,9 @@
 void *module_alloc(unsigned long size)
 {
 	void *p;
-	u64 base = (u64)_etext - MODULES_VSIZE;
 
-	if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) {
-		extern u32 module_load_offset;
-		base += module_load_offset;
-	}
-
-	p = __vmalloc_node_range(size, MODULE_ALIGN, base, base + MODULES_VSIZE,
+	p = __vmalloc_node_range(size, MODULE_ALIGN, module_alloc_base,
+				module_alloc_base + MODULES_VSIZE,
 				GFP_KERNEL, PAGE_KERNEL_EXEC, 0,
 				NUMA_NO_NODE, __builtin_return_address(0));
 
-- 
2.5.0

[PATCH v5sub2 0/8] arm64: implement virtual KASLR

[Catalin Marinas]

On Fri, Feb 05, 2016 at 12:42:30PM -0800, Kees Cook wrote:
> On Fri, Feb 5, 2016 at 9:38 AM, Ard Biesheuvel
> <ard.biesheuvel@linaro.org> wrote:
> > On 5 February 2016 at 18:32, Catalin Marinas <catalin.marinas@arm.com> wrote:
> >> I'm still trying to get my head around how we merge those. Since I
> >> assume akpm will push them during the merging window, part of your code
> >> cannot be tested before.
> >
> > Actually, my original idea was for akpm to take them as a late merge
> > after rebasing to -rc1, since they touch a variety of architectures,
> > but I am not sure if that came across.
> >
> > You could always take the series through your tree instead, I guess?
> 
> Traditionally akpm will de-duplicate patches he's carrying that appear
> in another tree. I think it should be okay to carry them in both
> places. (Though I'm CCing akpm just to see if I'm talking crazy.)

For now, I'll merge this series in the arm64 tree and push it to next:

http://lkml.kernel.org/r/1452007180-27411-1-git-send-email-ard.biesheuvel at linaro.org

If there are any objections, I can drop the patches and do the
BUILDTIME_EXTABLE_SORT disabling trick until they end up in mainline.

-- 
Catalin

[PATCH v5sub2 0/8] arm64: implement virtual KASLR

[Ard Biesheuvel]

On 8 February 2016 at 13:14, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Fri, Feb 05, 2016 at 12:42:30PM -0800, Kees Cook wrote:
>> On Fri, Feb 5, 2016 at 9:38 AM, Ard Biesheuvel
>> <ard.biesheuvel@linaro.org> wrote:
>> > On 5 February 2016 at 18:32, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> >> I'm still trying to get my head around how we merge those. Since I
>> >> assume akpm will push them during the merging window, part of your code
>> >> cannot be tested before.
>> >
>> > Actually, my original idea was for akpm to take them as a late merge
>> > after rebasing to -rc1, since they touch a variety of architectures,
>> > but I am not sure if that came across.
>> >
>> > You could always take the series through your tree instead, I guess?
>>
>> Traditionally akpm will de-duplicate patches he's carrying that appear
>> in another tree. I think it should be okay to carry them in both
>> places. (Though I'm CCing akpm just to see if I'm talking crazy.)
>
> For now, I'll merge this series in the arm64 tree and push it to next:
>
> http://lkml.kernel.org/r/1452007180-27411-1-git-send-email-ard.biesheuvel at linaro.org
>
> If there are any objections, I can drop the patches and do the
> BUILDTIME_EXTABLE_SORT disabling trick until they end up in mainline.
>

Latest version is here:
http://lkml.kernel.org/r/1453892123-17973-1-git-send-email-ard.biesheuvel at linaro.org
(only difference is an ack from that Alpha maintainter/supporter to
patches #1 and #2)

However, the arm64 patch (#6) now conflicts with futex.h in -rc3 after
the PAN fix, not sure how to best address that ...

-- 
Ard.

[PATCH v5sub2 0/8] arm64: implement virtual KASLR

[Catalin Marinas]

On Mon, Feb 08, 2016 at 03:30:47PM +0100, Ard Biesheuvel wrote:
> On 8 February 2016 at 13:14, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Fri, Feb 05, 2016 at 12:42:30PM -0800, Kees Cook wrote:
> >> On Fri, Feb 5, 2016 at 9:38 AM, Ard Biesheuvel
> >> <ard.biesheuvel@linaro.org> wrote:
> >> > On 5 February 2016 at 18:32, Catalin Marinas <catalin.marinas@arm.com> wrote:
> >> >> I'm still trying to get my head around how we merge those. Since I
> >> >> assume akpm will push them during the merging window, part of your code
> >> >> cannot be tested before.
> >> >
> >> > Actually, my original idea was for akpm to take them as a late merge
> >> > after rebasing to -rc1, since they touch a variety of architectures,
> >> > but I am not sure if that came across.
> >> >
> >> > You could always take the series through your tree instead, I guess?
> >>
> >> Traditionally akpm will de-duplicate patches he's carrying that appear
> >> in another tree. I think it should be okay to carry them in both
> >> places. (Though I'm CCing akpm just to see if I'm talking crazy.)
> >
> > For now, I'll merge this series in the arm64 tree and push it to next:
> >
> > http://lkml.kernel.org/r/1452007180-27411-1-git-send-email-ard.biesheuvel at linaro.org
> >
> > If there are any objections, I can drop the patches and do the
> > BUILDTIME_EXTABLE_SORT disabling trick until they end up in mainline.
> 
> Latest version is here:
> http://lkml.kernel.org/r/1453892123-17973-1-git-send-email-ard.biesheuvel at linaro.org
> (only difference is an ack from that Alpha maintainter/supporter to
> patches #1 and #2)

I applied the acks manually but I'll double-check to make sure I haven't
missed anything.

> However, the arm64 patch (#6) now conflicts with futex.h in -rc3 after
> the PAN fix, not sure how to best address that ...

I'll have a look.

-- 
Catalin

[PATCH v5sub2 0/8] arm64: implement virtual KASLR

[Ard Biesheuvel]

On 8 February 2016 at 17:19, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Mon, Feb 08, 2016 at 03:30:47PM +0100, Ard Biesheuvel wrote:
>> On 8 February 2016 at 13:14, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> > On Fri, Feb 05, 2016 at 12:42:30PM -0800, Kees Cook wrote:
>> >> On Fri, Feb 5, 2016 at 9:38 AM, Ard Biesheuvel
>> >> <ard.biesheuvel@linaro.org> wrote:
>> >> > On 5 February 2016 at 18:32, Catalin Marinas <catalin.marinas@arm.com> wrote:
>> >> >> I'm still trying to get my head around how we merge those. Since I
>> >> >> assume akpm will push them during the merging window, part of your code
>> >> >> cannot be tested before.
>> >> >
>> >> > Actually, my original idea was for akpm to take them as a late merge
>> >> > after rebasing to -rc1, since they touch a variety of architectures,
>> >> > but I am not sure if that came across.
>> >> >
>> >> > You could always take the series through your tree instead, I guess?
>> >>
>> >> Traditionally akpm will de-duplicate patches he's carrying that appear
>> >> in another tree. I think it should be okay to carry them in both
>> >> places. (Though I'm CCing akpm just to see if I'm talking crazy.)
>> >
>> > For now, I'll merge this series in the arm64 tree and push it to next:
>> >
>> > http://lkml.kernel.org/r/1452007180-27411-1-git-send-email-ard.biesheuvel at linaro.org
>> >
>> > If there are any objections, I can drop the patches and do the
>> > BUILDTIME_EXTABLE_SORT disabling trick until they end up in mainline.
>>
>> Latest version is here:
>> http://lkml.kernel.org/r/1453892123-17973-1-git-send-email-ard.biesheuvel at linaro.org
>> (only difference is an ack from that Alpha maintainter/supporter to
>> patches #1 and #2)
>
> I applied the acks manually but I'll double-check to make sure I haven't
> missed anything.
>
>> However, the arm64 patch (#6) now conflicts with futex.h in -rc3 after
>> the PAN fix, not sure how to best address that ...
>
> I'll have a look.
>

Note that the fix is trivial, but I don't know who should be carrying
the fix is all

[PATCH v5sub2 0/8] arm64: implement virtual KASLR

[Catalin Marinas]

On Mon, Feb 08, 2016 at 05:20:41PM +0100, Ard Biesheuvel wrote:
> On 8 February 2016 at 17:19, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Mon, Feb 08, 2016 at 03:30:47PM +0100, Ard Biesheuvel wrote:
> >> However, the arm64 patch (#6) now conflicts with futex.h in -rc3 after
> >> the PAN fix, not sure how to best address that ...
> >
> > I'll have a look.
> 
> Note that the fix is trivial, but I don't know who should be carrying
> the fix is all

I'll push the arm64 for-next/core branch (including your series) out
against 4.5-rc3, so there won't be any conflict.

-- 
Catalin

[PATCH] arm64: allow the module region to be randomized independently

[Catalin Marinas]

On Mon, Feb 08, 2016 at 11:12:12AM +0100, Ard Biesheuvel wrote:
> This adds the option to randomize the module region independently from the
> core kernel, and enables it by default. This makes it less likely that the
> location of core kernel data structures can be determined by an adversary,
> but causes all function calls from modules into the core kernel to be
> resolved via entries in the module PLTs.
> 
> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---
>  arch/arm64/Kconfig              | 15 ++++++++
>  arch/arm64/include/asm/module.h |  6 ++++
>  arch/arm64/kernel/kaslr.c       | 36 +++++++++++++++-----
>  arch/arm64/kernel/module.c      |  9 ++---
>  4 files changed, 50 insertions(+), 16 deletions(-)

With this patch I get an unhandled paging request, coming from
kernel/module.c:2982 (the memset). The PC is wrongly attributed but it's
in arch/arm64/lib/memset.S:

[    7.140606] Unable to handle kernel paging request at virtual address 00004000
[    7.147794] pgd = ffffffc060171000
[    7.151190] [00004000] *pgd=0000000000000000, *pud=0000000000000000
[    7.157447] Internal error: Oops: 96000045 [#1] PREEMPT SMP
[    7.162962] Modules linked in:
[    7.165995] CPU: 1 PID: 875 Comm: systemd-modules Not tainted 4.5.0-rc1+ #95
[    7.172976] Hardware name: Juno (DT)
[    7.176520] task: ffffffc9760bb000 ti: ffffffc079538000 task.ti: ffffffc079538000
[    7.183939] PC is at __efistub_memset+0x1ac/0x200
[    7.188601] LR is at load_module+0xfc8/0x1df8
[    7.192912] pc : [<ffffff8008336fac>] lr : [<ffffff8008120d88>] pstate: 40000145
[    7.200233] sp : ffffffc07953bd40
[    7.203514] x29: ffffffc07953bd40 x28: 0000000000002361
[    7.208791] x27: ffffff80086bb000 x26: ffffff8008f84aa0
[    7.214054] x25: 0000000000000111 x24: 000000000000006e
[    7.219317] x23: 0000007f7bc01918 x22: ffffff8008f0e100
[    7.224580] x21: ffffff8008f4d2c0 x20: 0000000000004000
[    7.229855] x19: ffffffc07953be70 x18: 0000000000000000
[    7.235127] x17: 0000000000000000 x16: 0000000000000002
[    7.240398] x15: ffffffffffffffff x14: ffffff0000000000
[    7.245667] x13: ffffffbdc3e55340 x12: 0000000000006fff
[    7.250934] x11: ffffffc97fed46a8 x10: 0000000000000010
[    7.256198] x9 : 0000000000000000 x8 : 0000000000004000
[    7.261462] x7 : 0000000000000000 x6 : 000000000000003f
[    7.266823] x5 : 0000000000000040 x4 : 0000000000000000 [    7.271219] systemd-journald[864]: Received request to flush runtime journal from PID 1

[    7.279835]
[    7.281487] x3 : 0000000000000004 x2 : 000000000000229e
[    7.286758] x1 : 0000000000000000 x0 : 0000000000004000
[    7.292019]
[    7.293495] Process systemd-modules (pid: 875, stack limit = 0xffffffc079538020)
[    7.300822] Stack: (0xffffffc07953bd40 to 0xffffffc07953c000)
[    7.306522] bd40: ffffffc07953be40 ffffff8008121de0 0000000000000000 0000000000000005
[    7.314276] bd60: 0000007f7bc01918 0000007f7bb24ad4 0000000080000000 0000000000000015
[    7.322029] bd80: 000000000000011e 0000000000000111 ffffff80086b0000 ffffffc079538000
[    7.329781] bda0: 0000000000000000 0000000000000005 0000007f7bc01918 0000007f7bb24ad4
[    7.337536] bdc0: ffffff8008f0e288 ffffff8008f84ae0 ffffff8008f0e2d8 ffffff8008f0d000
[    7.345288] bde0: ffff81a40000000f 0000000000000001 0000000000000000 0000000000077b20
[    7.353041] be00: 0000000056b8d7f8 00000000134c2b98 0000000056b8d7f8 000000001163e398
[    7.360793] be20: 0000000056b8d7f8 000000001163e398 0000000000001000 00000000000003c0
[    7.368545] be40: 0000000000000000 ffffff8008085d30 0000000000000000 0000000000000000
[    7.376298] be60: ffffffffffffffff 0000005571c2aa60 ffffff8008f0d000 0000000000077b20
[    7.384051] be80: ffffff8008f84120 ffffff8008f4b7af ffffff8008f4d2c0 0000000000001388
[    7.391803] bea0: 0000000000001dd8 0000000000000000 0000000000000000 0000002700000026
[    7.399555] bec0: 0000000000000011 000000000000000b 0000000000000005 0000007f7bc01918
[    7.407307] bee0: 0000000000000000 0000000000000005 0000000000000000 60ceffffffffffff
[    7.415060] bf00: ffffffffffffffff ffffffffffffffff 0000000000000111 0000000000000038
[    7.422812] bf20: 0101010101010101 0000000000000001 0000000000000000 ffffffffffff0000
[    7.430565] bf40: 0000007f7bc43000 0000007f7ba626b8 0000007f7bb24ab0 0000007f7bc132d8
[    7.438317] bf60: 0000005565850710 0000005571c2a8a0 0000000000000000 0000007f7bc01918
[    7.446069] bf80: 0000005571c2a920 0000000000020000 0000000000000000 0000000000000000
[    7.453821] bfa0: 0000005571c29330 0000000000000000 0000000000000000 0000007ff3bc1e80
[    7.461575] bfc0: 0000007f7bbfa1ac 0000007ff3bc1e80 0000007f7bb24ad4 0000000080000000
[    7.469327] bfe0: 0000000000000005 0000000000000111 f712e45f3fdb5baf 5d70fcf3d73b5fa3
[    7.477075] Call trace:
[    7.479494] Exception stack(0xffffffc07953bb80 to 0xffffffc07953bca0)
[    7.485871] bb80: ffffffc07953be70 0000000000004000 ffffffc07953bd40 ffffff8008336fac
[    7.493624] bba0: 0000000000400000 00000000024000c0 ffffffc975853300 00c8000000000713
[    7.501376] bbc0: ffffff80086bb000 0000000000002361 0000000000004000 0000000000000000
[    7.509128] bbe0: ffffffc07953bc60 ffffff80081885d8 ffffffc07953bca0 ffffff8008187fb8
[    7.516880] bc00: 0000000000000003 ffffffc975853480 00000000ffffffff 00000000024002c0
[    7.524631] bc20: 0000000000004000 0000000000000000 000000000000229e 0000000000000004
[    7.532383] bc40: 0000000000000000 0000000000000040 000000000000003f 0000000000000000
[    7.540135] bc60: 0000000000004000 0000000000000000 0000000000000010 ffffffc97fed46a8
[    7.547888] bc80: 0000000000006fff ffffffbdc3e55340 ffffff0000000000 ffffffffffffffff
[    7.555646] [<ffffff8008336fac>] __efistub_memset+0x1ac/0x200
[    7.561334] [<ffffff8008121de0>] SyS_finit_module+0xb0/0xc0
[    7.566852] [<ffffff8008085d30>] el0_svc_naked+0x24/0x28
[    7.572112] Code: 91010108 54ffff4a 8b040108 cb050042 (d50b7428)
[    7.578196] ---[ end trace 13bd770b734da68a ]---

-- 
Catalin

[PATCH] arm64: allow the module region to be randomized independently

[Ard Biesheuvel]

On 8 February 2016 at 19:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Mon, Feb 08, 2016 at 11:12:12AM +0100, Ard Biesheuvel wrote:
>> This adds the option to randomize the module region independently from the
>> core kernel, and enables it by default. This makes it less likely that the
>> location of core kernel data structures can be determined by an adversary,
>> but causes all function calls from modules into the core kernel to be
>> resolved via entries in the module PLTs.
>>
>> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
>> ---
>>  arch/arm64/Kconfig              | 15 ++++++++
>>  arch/arm64/include/asm/module.h |  6 ++++
>>  arch/arm64/kernel/kaslr.c       | 36 +++++++++++++++-----
>>  arch/arm64/kernel/module.c      |  9 ++---
>>  4 files changed, 50 insertions(+), 16 deletions(-)
>
> With this patch I get an unhandled paging request, coming from
> kernel/module.c:2982 (the memset). The PC is wrongly attributed but it's
> in arch/arm64/lib/memset.S:
>

That is quite surprising tbh. The only thing that is supposed to
happen here is that the low and high limits of the
__vmalloc_node_range() call in module_alloc() are modified. I don't
see how a successful __vmalloc_node_range() call should end up here.

I will try to reproduce tomorrow.

-- 
Ard.


> [    7.140606] Unable to handle kernel paging request at virtual address 00004000
> [    7.147794] pgd = ffffffc060171000
> [    7.151190] [00004000] *pgd=0000000000000000, *pud=0000000000000000
> [    7.157447] Internal error: Oops: 96000045 [#1] PREEMPT SMP
> [    7.162962] Modules linked in:
> [    7.165995] CPU: 1 PID: 875 Comm: systemd-modules Not tainted 4.5.0-rc1+ #95
> [    7.172976] Hardware name: Juno (DT)
> [    7.176520] task: ffffffc9760bb000 ti: ffffffc079538000 task.ti: ffffffc079538000
> [    7.183939] PC is at __efistub_memset+0x1ac/0x200
> [    7.188601] LR is at load_module+0xfc8/0x1df8
> [    7.192912] pc : [<ffffff8008336fac>] lr : [<ffffff8008120d88>] pstate: 40000145
> [    7.200233] sp : ffffffc07953bd40
> [    7.203514] x29: ffffffc07953bd40 x28: 0000000000002361
> [    7.208791] x27: ffffff80086bb000 x26: ffffff8008f84aa0
> [    7.214054] x25: 0000000000000111 x24: 000000000000006e
> [    7.219317] x23: 0000007f7bc01918 x22: ffffff8008f0e100
> [    7.224580] x21: ffffff8008f4d2c0 x20: 0000000000004000
> [    7.229855] x19: ffffffc07953be70 x18: 0000000000000000
> [    7.235127] x17: 0000000000000000 x16: 0000000000000002
> [    7.240398] x15: ffffffffffffffff x14: ffffff0000000000
> [    7.245667] x13: ffffffbdc3e55340 x12: 0000000000006fff
> [    7.250934] x11: ffffffc97fed46a8 x10: 0000000000000010
> [    7.256198] x9 : 0000000000000000 x8 : 0000000000004000
> [    7.261462] x7 : 0000000000000000 x6 : 000000000000003f
> [    7.266823] x5 : 0000000000000040 x4 : 0000000000000000 [    7.271219] systemd-journald[864]: Received request to flush runtime journal from PID 1
>
> [    7.279835]
> [    7.281487] x3 : 0000000000000004 x2 : 000000000000229e
> [    7.286758] x1 : 0000000000000000 x0 : 0000000000004000
> [    7.292019]
> [    7.293495] Process systemd-modules (pid: 875, stack limit = 0xffffffc079538020)
> [    7.300822] Stack: (0xffffffc07953bd40 to 0xffffffc07953c000)
> [    7.306522] bd40: ffffffc07953be40 ffffff8008121de0 0000000000000000 0000000000000005
> [    7.314276] bd60: 0000007f7bc01918 0000007f7bb24ad4 0000000080000000 0000000000000015
> [    7.322029] bd80: 000000000000011e 0000000000000111 ffffff80086b0000 ffffffc079538000
> [    7.329781] bda0: 0000000000000000 0000000000000005 0000007f7bc01918 0000007f7bb24ad4
> [    7.337536] bdc0: ffffff8008f0e288 ffffff8008f84ae0 ffffff8008f0e2d8 ffffff8008f0d000
> [    7.345288] bde0: ffff81a40000000f 0000000000000001 0000000000000000 0000000000077b20
> [    7.353041] be00: 0000000056b8d7f8 00000000134c2b98 0000000056b8d7f8 000000001163e398
> [    7.360793] be20: 0000000056b8d7f8 000000001163e398 0000000000001000 00000000000003c0
> [    7.368545] be40: 0000000000000000 ffffff8008085d30 0000000000000000 0000000000000000
> [    7.376298] be60: ffffffffffffffff 0000005571c2aa60 ffffff8008f0d000 0000000000077b20
> [    7.384051] be80: ffffff8008f84120 ffffff8008f4b7af ffffff8008f4d2c0 0000000000001388
> [    7.391803] bea0: 0000000000001dd8 0000000000000000 0000000000000000 0000002700000026
> [    7.399555] bec0: 0000000000000011 000000000000000b 0000000000000005 0000007f7bc01918
> [    7.407307] bee0: 0000000000000000 0000000000000005 0000000000000000 60ceffffffffffff
> [    7.415060] bf00: ffffffffffffffff ffffffffffffffff 0000000000000111 0000000000000038
> [    7.422812] bf20: 0101010101010101 0000000000000001 0000000000000000 ffffffffffff0000
> [    7.430565] bf40: 0000007f7bc43000 0000007f7ba626b8 0000007f7bb24ab0 0000007f7bc132d8
> [    7.438317] bf60: 0000005565850710 0000005571c2a8a0 0000000000000000 0000007f7bc01918
> [    7.446069] bf80: 0000005571c2a920 0000000000020000 0000000000000000 0000000000000000
> [    7.453821] bfa0: 0000005571c29330 0000000000000000 0000000000000000 0000007ff3bc1e80
> [    7.461575] bfc0: 0000007f7bbfa1ac 0000007ff3bc1e80 0000007f7bb24ad4 0000000080000000
> [    7.469327] bfe0: 0000000000000005 0000000000000111 f712e45f3fdb5baf 5d70fcf3d73b5fa3
> [    7.477075] Call trace:
> [    7.479494] Exception stack(0xffffffc07953bb80 to 0xffffffc07953bca0)
> [    7.485871] bb80: ffffffc07953be70 0000000000004000 ffffffc07953bd40 ffffff8008336fac
> [    7.493624] bba0: 0000000000400000 00000000024000c0 ffffffc975853300 00c8000000000713
> [    7.501376] bbc0: ffffff80086bb000 0000000000002361 0000000000004000 0000000000000000
> [    7.509128] bbe0: ffffffc07953bc60 ffffff80081885d8 ffffffc07953bca0 ffffff8008187fb8
> [    7.516880] bc00: 0000000000000003 ffffffc975853480 00000000ffffffff 00000000024002c0
> [    7.524631] bc20: 0000000000004000 0000000000000000 000000000000229e 0000000000000004
> [    7.532383] bc40: 0000000000000000 0000000000000040 000000000000003f 0000000000000000
> [    7.540135] bc60: 0000000000004000 0000000000000000 0000000000000010 ffffffc97fed46a8
> [    7.547888] bc80: 0000000000006fff ffffffbdc3e55340 ffffff0000000000 ffffffffffffffff
> [    7.555646] [<ffffff8008336fac>] __efistub_memset+0x1ac/0x200
> [    7.561334] [<ffffff8008121de0>] SyS_finit_module+0xb0/0xc0
> [    7.566852] [<ffffff8008085d30>] el0_svc_naked+0x24/0x28
> [    7.572112] Code: 91010108 54ffff4a 8b040108 cb050042 (d50b7428)
> [    7.578196] ---[ end trace 13bd770b734da68a ]---
>
> --
> Catalin

[PATCH] arm64: allow the module region to be randomized independently

[Ard Biesheuvel]

On 8 February 2016 at 19:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Mon, Feb 08, 2016 at 11:12:12AM +0100, Ard Biesheuvel wrote:
>> This adds the option to randomize the module region independently from the
>> core kernel, and enables it by default. This makes it less likely that the
>> location of core kernel data structures can be determined by an adversary,
>> but causes all function calls from modules into the core kernel to be
>> resolved via entries in the module PLTs.
>>
>> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
>> ---
>>  arch/arm64/Kconfig              | 15 ++++++++
>>  arch/arm64/include/asm/module.h |  6 ++++
>>  arch/arm64/kernel/kaslr.c       | 36 +++++++++++++++-----
>>  arch/arm64/kernel/module.c      |  9 ++---
>>  4 files changed, 50 insertions(+), 16 deletions(-)
>
> With this patch I get an unhandled paging request, coming from
> kernel/module.c:2982 (the memset). The PC is wrongly attributed but it's
> in arch/arm64/lib/memset.S:
>

This was due to a thinko in kaslr_early_init(). Folding the following
hunk will fix it (or I can resend the patch if you prefer)

-------8<----------
diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c
index f8ad30160026..d7eba99ab140 100644
--- a/arch/arm64/kernel/kaslr.c
+++ b/arch/arm64/kernel/kaslr.c
@@ -82,6 +82,12 @@ u64 __init kaslr_early_init(u64 dt_phys)
        int size;

        /*
+        * Set a reasonable default for module_alloc_base in case
+        * we end up running with KASLR disabled.
+        */
+       module_alloc_base = (u64)_etext - MODULES_VSIZE;
+
+       /*
         * Try to map the FDT early. If this fails, we simply bail,
         * and proceed with KASLR disabled. We will make another
         * attempt@mapping the FDT in setup_machine()
-------8<----------

[PATCH] arm64: allow the module region to be randomized independently

[Catalin Marinas]

On Tue, Feb 09, 2016 at 11:03:01AM +0100, Ard Biesheuvel wrote:
> On 8 February 2016 at 19:13, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > On Mon, Feb 08, 2016 at 11:12:12AM +0100, Ard Biesheuvel wrote:
> >> This adds the option to randomize the module region independently from the
> >> core kernel, and enables it by default. This makes it less likely that the
> >> location of core kernel data structures can be determined by an adversary,
> >> but causes all function calls from modules into the core kernel to be
> >> resolved via entries in the module PLTs.
> >>
> >> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> >> ---
> >>  arch/arm64/Kconfig              | 15 ++++++++
> >>  arch/arm64/include/asm/module.h |  6 ++++
> >>  arch/arm64/kernel/kaslr.c       | 36 +++++++++++++++-----
> >>  arch/arm64/kernel/module.c      |  9 ++---
> >>  4 files changed, 50 insertions(+), 16 deletions(-)
> >
> > With this patch I get an unhandled paging request, coming from
> > kernel/module.c:2982 (the memset). The PC is wrongly attributed but it's
> > in arch/arm64/lib/memset.S:
> >
> 
> This was due to a thinko in kaslr_early_init(). Folding the following
> hunk will fix it (or I can resend the patch if you prefer)
> 
> -------8<----------
> diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c
> index f8ad30160026..d7eba99ab140 100644
> --- a/arch/arm64/kernel/kaslr.c
> +++ b/arch/arm64/kernel/kaslr.c
> @@ -82,6 +82,12 @@ u64 __init kaslr_early_init(u64 dt_phys)
>         int size;
> 
>         /*
> +        * Set a reasonable default for module_alloc_base in case
> +        * we end up running with KASLR disabled.
> +        */
> +       module_alloc_base = (u64)_etext - MODULES_VSIZE;
> +
> +       /*
>          * Try to map the FDT early. If this fails, we simply bail,
>          * and proceed with KASLR disabled. We will make another
>          * attempt at mapping the FDT in setup_machine()
> -------8<----------

I'll fold the hunk in and give it a try. Thanks.

-- 
Catalin

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Will Deacon]

Hi Ard,

On Mon, Feb 01, 2016 at 02:09:31PM +0100, Ard Biesheuvel wrote:
> This adds support for emitting PLTs at module load time for relative
> branches that are out of range. This is a prerequisite for KASLR, which
> may place the kernel and the modules anywhere in the vmalloc area,
> making it more likely that branch target offsets exceed the maximum
> range of +/- 128 MB.
> 
> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---
> 
> In this version, I removed the distinction between relocations against
> .init executable sections and ordinary executable sections. The reason
> is that it is hardly worth the trouble, given that .init.text usually
> does not contain that many far branches, and this version now only
> reserves PLT entry space for jump and call relocations against undefined
> symbols (since symbols defined in the same module can be assumed to be
> within +/- 128 MB)
> 
> For example, the mac80211.ko module (which is fairly sizable at ~400 KB)
> built with -mcmodel=large gives the following relocation counts:
> 
>                     relocs    branches   unique     !local
>   .text              3925       3347       518        219
>   .init.text           11          8         7          1
>   .exit.text            4          4         4          1
>   .text.unlikely       81         67        36         17
> 
> ('unique' means branches to unique type/symbol/addend combos, of which
> !local is the subset referring to undefined symbols)
> 
> IOW, we are only emitting a single PLT entry for the .init sections, and
> we are better off just adding it to the core PLT section instead.
> ---
>  arch/arm64/Kconfig              |   9 +
>  arch/arm64/Makefile             |   6 +-
>  arch/arm64/include/asm/module.h |  11 ++
>  arch/arm64/kernel/Makefile      |   1 +
>  arch/arm64/kernel/module-plts.c | 201 ++++++++++++++++++++
>  arch/arm64/kernel/module.c      |  12 ++
>  arch/arm64/kernel/module.lds    |   3 +
>  7 files changed, 242 insertions(+), 1 deletion(-)

[...]

> +struct plt_entry {
> +	/*
> +	 * A program that conforms to the AArch64 Procedure Call Standard
> +	 * (AAPCS64) must assume that a veneer that alters IP0 (x16) and/or
> +	 * IP1 (x17) may be inserted at any branch instruction that is
> +	 * exposed to a relocation that supports long branches. Since that
> +	 * is exactly what we are dealing with here, we are free to use x16
> +	 * as a scratch register in the PLT veneers.
> +	 */
> +	__le32	mov0;	/* movn	x16, #0x....			*/
> +	__le32	mov1;	/* movk	x16, #0x...., lsl #16		*/
> +	__le32	mov2;	/* movk	x16, #0x...., lsl #32		*/
> +	__le32	br;	/* br	x16				*/
> +};

I'm worried about this code when CONFIG_ARM64_LSE_ATOMICS=y, but we don't
detect them on the CPU at runtime. In this case, all atomic operations
are moved out-of-line and called using a bl instruction from inline asm.

The out-of-line code is compiled with magic GCC options to force the
explicit save/restore of all used registers (see arch/arm64/lib/Makefile),
otherwise we'd have to clutter the inline asm with constraints that
wouldn't be needed had we managed to patch the bl with an LSE atomic
instruction.

If you're emitting a PLT, couldn't we end up with silent corruption of
x16 for modules using out-of-line atomics like this?

Will

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Ard Biesheuvel]

On 25 February 2016 at 17:07, Will Deacon <will.deacon@arm.com> wrote:
> Hi Ard,
>
> On Mon, Feb 01, 2016 at 02:09:31PM +0100, Ard Biesheuvel wrote:
>> This adds support for emitting PLTs at module load time for relative
>> branches that are out of range. This is a prerequisite for KASLR, which
>> may place the kernel and the modules anywhere in the vmalloc area,
>> making it more likely that branch target offsets exceed the maximum
>> range of +/- 128 MB.
>>
>> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
>> ---
>>
>> In this version, I removed the distinction between relocations against
>> .init executable sections and ordinary executable sections. The reason
>> is that it is hardly worth the trouble, given that .init.text usually
>> does not contain that many far branches, and this version now only
>> reserves PLT entry space for jump and call relocations against undefined
>> symbols (since symbols defined in the same module can be assumed to be
>> within +/- 128 MB)
>>
>> For example, the mac80211.ko module (which is fairly sizable at ~400 KB)
>> built with -mcmodel=large gives the following relocation counts:
>>
>>                     relocs    branches   unique     !local
>>   .text              3925       3347       518        219
>>   .init.text           11          8         7          1
>>   .exit.text            4          4         4          1
>>   .text.unlikely       81         67        36         17
>>
>> ('unique' means branches to unique type/symbol/addend combos, of which
>> !local is the subset referring to undefined symbols)
>>
>> IOW, we are only emitting a single PLT entry for the .init sections, and
>> we are better off just adding it to the core PLT section instead.
>> ---
>>  arch/arm64/Kconfig              |   9 +
>>  arch/arm64/Makefile             |   6 +-
>>  arch/arm64/include/asm/module.h |  11 ++
>>  arch/arm64/kernel/Makefile      |   1 +
>>  arch/arm64/kernel/module-plts.c | 201 ++++++++++++++++++++
>>  arch/arm64/kernel/module.c      |  12 ++
>>  arch/arm64/kernel/module.lds    |   3 +
>>  7 files changed, 242 insertions(+), 1 deletion(-)
>
> [...]
>
>> +struct plt_entry {
>> +     /*
>> +      * A program that conforms to the AArch64 Procedure Call Standard
>> +      * (AAPCS64) must assume that a veneer that alters IP0 (x16) and/or
>> +      * IP1 (x17) may be inserted at any branch instruction that is
>> +      * exposed to a relocation that supports long branches. Since that
>> +      * is exactly what we are dealing with here, we are free to use x16
>> +      * as a scratch register in the PLT veneers.
>> +      */
>> +     __le32  mov0;   /* movn x16, #0x....                    */
>> +     __le32  mov1;   /* movk x16, #0x...., lsl #16           */
>> +     __le32  mov2;   /* movk x16, #0x...., lsl #32           */
>> +     __le32  br;     /* br   x16                             */
>> +};
>
> I'm worried about this code when CONFIG_ARM64_LSE_ATOMICS=y, but we don't
> detect them on the CPU at runtime. In this case, all atomic operations
> are moved out-of-line and called using a bl instruction from inline asm.
>
> The out-of-line code is compiled with magic GCC options

Which options are those exactly?

> to force the
> explicit save/restore of all used registers (see arch/arm64/lib/Makefile),
> otherwise we'd have to clutter the inline asm with constraints that
> wouldn't be needed had we managed to patch the bl with an LSE atomic
> instruction.
>
> If you're emitting a PLT, couldn't we end up with silent corruption of
> x16 for modules using out-of-line atomics like this?
>

If you violate the AAPCS64 ABI, then obviously the claim above does not hold.

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Ard Biesheuvel]

On 25 February 2016 at 17:12, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> On 25 February 2016 at 17:07, Will Deacon <will.deacon@arm.com> wrote:
>> Hi Ard,
>>
>> On Mon, Feb 01, 2016 at 02:09:31PM +0100, Ard Biesheuvel wrote:
>>> This adds support for emitting PLTs at module load time for relative
>>> branches that are out of range. This is a prerequisite for KASLR, which
>>> may place the kernel and the modules anywhere in the vmalloc area,
>>> making it more likely that branch target offsets exceed the maximum
>>> range of +/- 128 MB.
>>>
>>> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
>>> ---
>>>
>>> In this version, I removed the distinction between relocations against
>>> .init executable sections and ordinary executable sections. The reason
>>> is that it is hardly worth the trouble, given that .init.text usually
>>> does not contain that many far branches, and this version now only
>>> reserves PLT entry space for jump and call relocations against undefined
>>> symbols (since symbols defined in the same module can be assumed to be
>>> within +/- 128 MB)
>>>
>>> For example, the mac80211.ko module (which is fairly sizable at ~400 KB)
>>> built with -mcmodel=large gives the following relocation counts:
>>>
>>>                     relocs    branches   unique     !local
>>>   .text              3925       3347       518        219
>>>   .init.text           11          8         7          1
>>>   .exit.text            4          4         4          1
>>>   .text.unlikely       81         67        36         17
>>>
>>> ('unique' means branches to unique type/symbol/addend combos, of which
>>> !local is the subset referring to undefined symbols)
>>>
>>> IOW, we are only emitting a single PLT entry for the .init sections, and
>>> we are better off just adding it to the core PLT section instead.
>>> ---
>>>  arch/arm64/Kconfig              |   9 +
>>>  arch/arm64/Makefile             |   6 +-
>>>  arch/arm64/include/asm/module.h |  11 ++
>>>  arch/arm64/kernel/Makefile      |   1 +
>>>  arch/arm64/kernel/module-plts.c | 201 ++++++++++++++++++++
>>>  arch/arm64/kernel/module.c      |  12 ++
>>>  arch/arm64/kernel/module.lds    |   3 +
>>>  7 files changed, 242 insertions(+), 1 deletion(-)
>>
>> [...]
>>
>>> +struct plt_entry {
>>> +     /*
>>> +      * A program that conforms to the AArch64 Procedure Call Standard
>>> +      * (AAPCS64) must assume that a veneer that alters IP0 (x16) and/or
>>> +      * IP1 (x17) may be inserted at any branch instruction that is
>>> +      * exposed to a relocation that supports long branches. Since that
>>> +      * is exactly what we are dealing with here, we are free to use x16
>>> +      * as a scratch register in the PLT veneers.
>>> +      */
>>> +     __le32  mov0;   /* movn x16, #0x....                    */
>>> +     __le32  mov1;   /* movk x16, #0x...., lsl #16           */
>>> +     __le32  mov2;   /* movk x16, #0x...., lsl #32           */
>>> +     __le32  br;     /* br   x16                             */
>>> +};
>>
>> I'm worried about this code when CONFIG_ARM64_LSE_ATOMICS=y, but we don't
>> detect them on the CPU at runtime. In this case, all atomic operations
>> are moved out-of-line and called using a bl instruction from inline asm.
>>
>> The out-of-line code is compiled with magic GCC options
>
> Which options are those exactly?
>
>> to force the
>> explicit save/restore of all used registers (see arch/arm64/lib/Makefile),
>> otherwise we'd have to clutter the inline asm with constraints that
>> wouldn't be needed had we managed to patch the bl with an LSE atomic
>> instruction.
>>
>> If you're emitting a PLT, couldn't we end up with silent corruption of
>> x16 for modules using out-of-line atomics like this?
>>
>
> If you violate the AAPCS64 ABI, then obviously the claim above does not hold.

IOW, yes, x16 will be corrupted unexpectedly if it you are expecting
it to be preserved across a bl or b instruction that is exposed to a
jump/call relocation

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Will Deacon]

On Thu, Feb 25, 2016 at 05:12:01PM +0100, Ard Biesheuvel wrote:
> On 25 February 2016 at 17:07, Will Deacon <will.deacon@arm.com> wrote:
> > On Mon, Feb 01, 2016 at 02:09:31PM +0100, Ard Biesheuvel wrote:
> >> +struct plt_entry {
> >> +     /*
> >> +      * A program that conforms to the AArch64 Procedure Call Standard
> >> +      * (AAPCS64) must assume that a veneer that alters IP0 (x16) and/or
> >> +      * IP1 (x17) may be inserted at any branch instruction that is
> >> +      * exposed to a relocation that supports long branches. Since that
> >> +      * is exactly what we are dealing with here, we are free to use x16
> >> +      * as a scratch register in the PLT veneers.
> >> +      */
> >> +     __le32  mov0;   /* movn x16, #0x....                    */
> >> +     __le32  mov1;   /* movk x16, #0x...., lsl #16           */
> >> +     __le32  mov2;   /* movk x16, #0x...., lsl #32           */
> >> +     __le32  br;     /* br   x16                             */
> >> +};
> >
> > I'm worried about this code when CONFIG_ARM64_LSE_ATOMICS=y, but we don't
> > detect them on the CPU at runtime. In this case, all atomic operations
> > are moved out-of-line and called using a bl instruction from inline asm.
> >
> > The out-of-line code is compiled with magic GCC options
> 
> Which options are those exactly?

# Tell the compiler to treat all general purpose registers as
# callee-saved, which allows for efficient runtime patching of the bl
# instruction in the caller with an atomic instruction when supported by
# the CPU. Result and argument registers are handled correctly, based on
# the function prototype.
lib-$(CONFIG_ARM64_LSE_ATOMICS) += atomic_ll_sc.o
CFLAGS_atomic_ll_sc.o	:= -fcall-used-x0 -ffixed-x1 -ffixed-x2		\
		   -ffixed-x3 -ffixed-x4 -ffixed-x5 -ffixed-x6		\
		   -ffixed-x7 -fcall-saved-x8 -fcall-saved-x9		\
		   -fcall-saved-x10 -fcall-saved-x11 -fcall-saved-x12	\
		   -fcall-saved-x13 -fcall-saved-x14 -fcall-saved-x15	\
		   -fcall-saved-x16 -fcall-saved-x17 -fcall-saved-x18

> > to force the
> > explicit save/restore of all used registers (see arch/arm64/lib/Makefile),
> > otherwise we'd have to clutter the inline asm with constraints that
> > wouldn't be needed had we managed to patch the bl with an LSE atomic
> > instruction.
> >
> > If you're emitting a PLT, couldn't we end up with silent corruption of
> > x16 for modules using out-of-line atomics like this?
> >
> 
> If you violate the AAPCS64 ABI, then obviously the claim above does not hold.

Indeed, but this is what mainline is doing today and I'm not keen on
breaking it. One way to fix it would be to generate a different type of
plt for branches to the atomic functions that would use the stack
instead of x16.

Will

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Ard Biesheuvel]

On 25 February 2016 at 17:26, Will Deacon <will.deacon@arm.com> wrote:
> On Thu, Feb 25, 2016 at 05:12:01PM +0100, Ard Biesheuvel wrote:
>> On 25 February 2016 at 17:07, Will Deacon <will.deacon@arm.com> wrote:
>> > On Mon, Feb 01, 2016 at 02:09:31PM +0100, Ard Biesheuvel wrote:
>> >> +struct plt_entry {
>> >> +     /*
>> >> +      * A program that conforms to the AArch64 Procedure Call Standard
>> >> +      * (AAPCS64) must assume that a veneer that alters IP0 (x16) and/or
>> >> +      * IP1 (x17) may be inserted at any branch instruction that is
>> >> +      * exposed to a relocation that supports long branches. Since that
>> >> +      * is exactly what we are dealing with here, we are free to use x16
>> >> +      * as a scratch register in the PLT veneers.
>> >> +      */
>> >> +     __le32  mov0;   /* movn x16, #0x....                    */
>> >> +     __le32  mov1;   /* movk x16, #0x...., lsl #16           */
>> >> +     __le32  mov2;   /* movk x16, #0x...., lsl #32           */
>> >> +     __le32  br;     /* br   x16                             */
>> >> +};
>> >
>> > I'm worried about this code when CONFIG_ARM64_LSE_ATOMICS=y, but we don't
>> > detect them on the CPU at runtime. In this case, all atomic operations
>> > are moved out-of-line and called using a bl instruction from inline asm.
>> >
>> > The out-of-line code is compiled with magic GCC options
>>
>> Which options are those exactly?
>
> # Tell the compiler to treat all general purpose registers as
> # callee-saved, which allows for efficient runtime patching of the bl
> # instruction in the caller with an atomic instruction when supported by
> # the CPU. Result and argument registers are handled correctly, based on
> # the function prototype.
> lib-$(CONFIG_ARM64_LSE_ATOMICS) += atomic_ll_sc.o
> CFLAGS_atomic_ll_sc.o   := -fcall-used-x0 -ffixed-x1 -ffixed-x2         \
>                    -ffixed-x3 -ffixed-x4 -ffixed-x5 -ffixed-x6          \
>                    -ffixed-x7 -fcall-saved-x8 -fcall-saved-x9           \
>                    -fcall-saved-x10 -fcall-saved-x11 -fcall-saved-x12   \
>                    -fcall-saved-x13 -fcall-saved-x14 -fcall-saved-x15   \
>                    -fcall-saved-x16 -fcall-saved-x17 -fcall-saved-x18
>

Yikes. Is that safe?

>> > to force the
>> > explicit save/restore of all used registers (see arch/arm64/lib/Makefile),
>> > otherwise we'd have to clutter the inline asm with constraints that
>> > wouldn't be needed had we managed to patch the bl with an LSE atomic
>> > instruction.
>> >
>> > If you're emitting a PLT, couldn't we end up with silent corruption of
>> > x16 for modules using out-of-line atomics like this?
>> >
>>
>> If you violate the AAPCS64 ABI, then obviously the claim above does not hold.
>
> Indeed, but this is what mainline is doing today and I'm not keen on
> breaking it. One way to fix it would be to generate a different type of
> plt for branches to the atomic functions that would use the stack
> instead of x16.
>

AFAIK, gcc never uses x18 (the platform register) so we may be able to
use that instead. We'd need confirmation from the toolchain guys,
though ...

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Will Deacon]

On Thu, Feb 25, 2016 at 05:33:25PM +0100, Ard Biesheuvel wrote:
> On 25 February 2016 at 17:26, Will Deacon <will.deacon@arm.com> wrote:
> > On Thu, Feb 25, 2016 at 05:12:01PM +0100, Ard Biesheuvel wrote:
> >> On 25 February 2016 at 17:07, Will Deacon <will.deacon@arm.com> wrote:
> >> > On Mon, Feb 01, 2016 at 02:09:31PM +0100, Ard Biesheuvel wrote:
> >> >> +struct plt_entry {
> >> >> +     /*
> >> >> +      * A program that conforms to the AArch64 Procedure Call Standard
> >> >> +      * (AAPCS64) must assume that a veneer that alters IP0 (x16) and/or
> >> >> +      * IP1 (x17) may be inserted at any branch instruction that is
> >> >> +      * exposed to a relocation that supports long branches. Since that
> >> >> +      * is exactly what we are dealing with here, we are free to use x16
> >> >> +      * as a scratch register in the PLT veneers.
> >> >> +      */
> >> >> +     __le32  mov0;   /* movn x16, #0x....                    */
> >> >> +     __le32  mov1;   /* movk x16, #0x...., lsl #16           */
> >> >> +     __le32  mov2;   /* movk x16, #0x...., lsl #32           */
> >> >> +     __le32  br;     /* br   x16                             */
> >> >> +};
> >> >
> >> > I'm worried about this code when CONFIG_ARM64_LSE_ATOMICS=y, but we don't
> >> > detect them on the CPU at runtime. In this case, all atomic operations
> >> > are moved out-of-line and called using a bl instruction from inline asm.
> >> >
> >> > The out-of-line code is compiled with magic GCC options
> >>
> >> Which options are those exactly?
> >
> > # Tell the compiler to treat all general purpose registers as
> > # callee-saved, which allows for efficient runtime patching of the bl
> > # instruction in the caller with an atomic instruction when supported by
> > # the CPU. Result and argument registers are handled correctly, based on
> > # the function prototype.
> > lib-$(CONFIG_ARM64_LSE_ATOMICS) += atomic_ll_sc.o
> > CFLAGS_atomic_ll_sc.o   := -fcall-used-x0 -ffixed-x1 -ffixed-x2         \
> >                    -ffixed-x3 -ffixed-x4 -ffixed-x5 -ffixed-x6          \
> >                    -ffixed-x7 -fcall-saved-x8 -fcall-saved-x9           \
> >                    -fcall-saved-x10 -fcall-saved-x11 -fcall-saved-x12   \
> >                    -fcall-saved-x13 -fcall-saved-x14 -fcall-saved-x15   \
> >                    -fcall-saved-x16 -fcall-saved-x17 -fcall-saved-x18
> >
> 
> Yikes. Is that safe?

It seems to work, and x86 uses a similar trick for it's hweight code.

> >> > to force the
> >> > explicit save/restore of all used registers (see arch/arm64/lib/Makefile),
> >> > otherwise we'd have to clutter the inline asm with constraints that
> >> > wouldn't be needed had we managed to patch the bl with an LSE atomic
> >> > instruction.
> >> >
> >> > If you're emitting a PLT, couldn't we end up with silent corruption of
> >> > x16 for modules using out-of-line atomics like this?
> >> >
> >>
> >> If you violate the AAPCS64 ABI, then obviously the claim above does not hold.
> >
> > Indeed, but this is what mainline is doing today and I'm not keen on
> > breaking it. One way to fix it would be to generate a different type of
> > plt for branches to the atomic functions that would use the stack
> > instead of x16.
> >
> 
> AFAIK, gcc never uses x18 (the platform register) so we may be able to
> use that instead. We'd need confirmation from the toolchain guys,
> though ...

In fact, a better thing to do is probably for the atomic code to
save/restore those register explicitly and then remove them from the
cflags above.

I'll try hacking something together...

Will

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Ard Biesheuvel]

On 25 February 2016 at 17:42, Will Deacon <will.deacon@arm.com> wrote:
> On Thu, Feb 25, 2016 at 05:33:25PM +0100, Ard Biesheuvel wrote:
>> On 25 February 2016 at 17:26, Will Deacon <will.deacon@arm.com> wrote:
>> > On Thu, Feb 25, 2016 at 05:12:01PM +0100, Ard Biesheuvel wrote:
>> >> On 25 February 2016 at 17:07, Will Deacon <will.deacon@arm.com> wrote:
>> >> > On Mon, Feb 01, 2016 at 02:09:31PM +0100, Ard Biesheuvel wrote:
>> >> >> +struct plt_entry {
>> >> >> +     /*
>> >> >> +      * A program that conforms to the AArch64 Procedure Call Standard
>> >> >> +      * (AAPCS64) must assume that a veneer that alters IP0 (x16) and/or
>> >> >> +      * IP1 (x17) may be inserted at any branch instruction that is
>> >> >> +      * exposed to a relocation that supports long branches. Since that
>> >> >> +      * is exactly what we are dealing with here, we are free to use x16
>> >> >> +      * as a scratch register in the PLT veneers.
>> >> >> +      */
>> >> >> +     __le32  mov0;   /* movn x16, #0x....                    */
>> >> >> +     __le32  mov1;   /* movk x16, #0x...., lsl #16           */
>> >> >> +     __le32  mov2;   /* movk x16, #0x...., lsl #32           */
>> >> >> +     __le32  br;     /* br   x16                             */
>> >> >> +};
>> >> >
>> >> > I'm worried about this code when CONFIG_ARM64_LSE_ATOMICS=y, but we don't
>> >> > detect them on the CPU at runtime. In this case, all atomic operations
>> >> > are moved out-of-line and called using a bl instruction from inline asm.
>> >> >
>> >> > The out-of-line code is compiled with magic GCC options
>> >>
>> >> Which options are those exactly?
>> >
>> > # Tell the compiler to treat all general purpose registers as
>> > # callee-saved, which allows for efficient runtime patching of the bl
>> > # instruction in the caller with an atomic instruction when supported by
>> > # the CPU. Result and argument registers are handled correctly, based on
>> > # the function prototype.
>> > lib-$(CONFIG_ARM64_LSE_ATOMICS) += atomic_ll_sc.o
>> > CFLAGS_atomic_ll_sc.o   := -fcall-used-x0 -ffixed-x1 -ffixed-x2         \
>> >                    -ffixed-x3 -ffixed-x4 -ffixed-x5 -ffixed-x6          \
>> >                    -ffixed-x7 -fcall-saved-x8 -fcall-saved-x9           \
>> >                    -fcall-saved-x10 -fcall-saved-x11 -fcall-saved-x12   \
>> >                    -fcall-saved-x13 -fcall-saved-x14 -fcall-saved-x15   \
>> >                    -fcall-saved-x16 -fcall-saved-x17 -fcall-saved-x18
>> >
>>
>> Yikes. Is that safe?
>
> It seems to work, and x86 uses a similar trick for it's hweight code.
>
>> >> > to force the
>> >> > explicit save/restore of all used registers (see arch/arm64/lib/Makefile),
>> >> > otherwise we'd have to clutter the inline asm with constraints that
>> >> > wouldn't be needed had we managed to patch the bl with an LSE atomic
>> >> > instruction.
>> >> >
>> >> > If you're emitting a PLT, couldn't we end up with silent corruption of
>> >> > x16 for modules using out-of-line atomics like this?
>> >> >
>> >>
>> >> If you violate the AAPCS64 ABI, then obviously the claim above does not hold.
>> >
>> > Indeed, but this is what mainline is doing today and I'm not keen on
>> > breaking it. One way to fix it would be to generate a different type of
>> > plt for branches to the atomic functions that would use the stack
>> > instead of x16.
>> >
>>
>> AFAIK, gcc never uses x18 (the platform register) so we may be able to
>> use that instead. We'd need confirmation from the toolchain guys,
>> though ...
>
> In fact, a better thing to do is probably for the atomic code to
> save/restore those register explicitly and then remove them from the
> cflags above.
>
> I'll try hacking something together...
>

That would be more correct, indeed. Also, the PLT can only use br
<reg> so it will always enter the callee with the preserved value
stacked, and so the callee needs to be modified in any case.

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Will Deacon]

On Thu, Feb 25, 2016 at 05:43:35PM +0100, Ard Biesheuvel wrote:
> On 25 February 2016 at 17:42, Will Deacon <will.deacon@arm.com> wrote:
> > On Thu, Feb 25, 2016 at 05:33:25PM +0100, Ard Biesheuvel wrote:
> >> AFAIK, gcc never uses x18 (the platform register) so we may be able to
> >> use that instead. We'd need confirmation from the toolchain guys,
> >> though ...
> >
> > In fact, a better thing to do is probably for the atomic code to
> > save/restore those register explicitly and then remove them from the
> > cflags above.
> >
> > I'll try hacking something together...
> >
> 
> That would be more correct, indeed. Also, the PLT can only use br
> <reg> so it will always enter the callee with the preserved value
> stacked, and so the callee needs to be modified in any case.

Wait -- why does the callee need to be modified here? It has no idea
about whether or not it was invoked via a PLT.

Also, can I rely on the PLT not clobbering the condition code?

Will

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Ard Biesheuvel]

On 25 February 2016 at 17:46, Will Deacon <will.deacon@arm.com> wrote:
> On Thu, Feb 25, 2016 at 05:43:35PM +0100, Ard Biesheuvel wrote:
>> On 25 February 2016 at 17:42, Will Deacon <will.deacon@arm.com> wrote:
>> > On Thu, Feb 25, 2016 at 05:33:25PM +0100, Ard Biesheuvel wrote:
>> >> AFAIK, gcc never uses x18 (the platform register) so we may be able to
>> >> use that instead. We'd need confirmation from the toolchain guys,
>> >> though ...
>> >
>> > In fact, a better thing to do is probably for the atomic code to
>> > save/restore those register explicitly and then remove them from the
>> > cflags above.
>> >
>> > I'll try hacking something together...
>> >
>>
>> That would be more correct, indeed. Also, the PLT can only use br
>> <reg> so it will always enter the callee with the preserved value
>> stacked, and so the callee needs to be modified in any case.
>
> Wait -- why does the callee need to be modified here? It has no idea
> about whether or not it was invoked via a PLT.
>

No, it doesn't, and that is exactly the problem: the PLT must end in a
br <reg> instruction because that is the only branch instruction with
unlimited range. That means you will always enter the callee with its
address in some register rather than the value that should have been
preserved. I don't see a way to manage that from the callee

> Also, can I rely on the PLT not clobbering the condition code?
>

For my particular implementation, yes.

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Ard Biesheuvel]

On 25 February 2016 at 17:49, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> On 25 February 2016 at 17:46, Will Deacon <will.deacon@arm.com> wrote:
>> On Thu, Feb 25, 2016 at 05:43:35PM +0100, Ard Biesheuvel wrote:
>>> On 25 February 2016 at 17:42, Will Deacon <will.deacon@arm.com> wrote:
>>> > On Thu, Feb 25, 2016 at 05:33:25PM +0100, Ard Biesheuvel wrote:
>>> >> AFAIK, gcc never uses x18 (the platform register) so we may be able to
>>> >> use that instead. We'd need confirmation from the toolchain guys,
>>> >> though ...
>>> >
>>> > In fact, a better thing to do is probably for the atomic code to
>>> > save/restore those register explicitly and then remove them from the
>>> > cflags above.
>>> >
>>> > I'll try hacking something together...
>>> >
>>>
>>> That would be more correct, indeed. Also, the PLT can only use br
>>> <reg> so it will always enter the callee with the preserved value
>>> stacked, and so the callee needs to be modified in any case.
>>
>> Wait -- why does the callee need to be modified here? It has no idea
>> about whether or not it was invoked via a PLT.
>>
>
> No, it doesn't, and that is exactly the problem: the PLT must end in a
> br <reg> instruction because that is the only branch instruction with
> unlimited range. That means you will always enter the callee with its
> address in some register rather than the value that should have been
> preserved. I don't see a way to manage that from the callee
>

*caller! dammit

>> Also, can I rely on the PLT not clobbering the condition code?
>>
>
> For my particular implementation, yes.

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Will Deacon]

On Thu, Feb 25, 2016 at 05:50:17PM +0100, Ard Biesheuvel wrote:
> On 25 February 2016 at 17:49, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> > On 25 February 2016 at 17:46, Will Deacon <will.deacon@arm.com> wrote:
> >> On Thu, Feb 25, 2016 at 05:43:35PM +0100, Ard Biesheuvel wrote:
> >>> On 25 February 2016 at 17:42, Will Deacon <will.deacon@arm.com> wrote:
> >>> > On Thu, Feb 25, 2016 at 05:33:25PM +0100, Ard Biesheuvel wrote:
> >>> >> AFAIK, gcc never uses x18 (the platform register) so we may be able to
> >>> >> use that instead. We'd need confirmation from the toolchain guys,
> >>> >> though ...
> >>> >
> >>> > In fact, a better thing to do is probably for the atomic code to
> >>> > save/restore those register explicitly and then remove them from the
> >>> > cflags above.
> >>> >
> >>> > I'll try hacking something together...
> >>> >
> >>>
> >>> That would be more correct, indeed. Also, the PLT can only use br
> >>> <reg> so it will always enter the callee with the preserved value
> >>> stacked, and so the callee needs to be modified in any case.
> >>
> >> Wait -- why does the callee need to be modified here? It has no idea
> >> about whether or not it was invoked via a PLT.
> >>
> >
> > No, it doesn't, and that is exactly the problem: the PLT must end in a
> > br <reg> instruction because that is the only branch instruction with
> > unlimited range. That means you will always enter the callee with its
> > address in some register rather than the value that should have been
> > preserved. I don't see a way to manage that from the callee
> >
> 
> *caller! dammit

The caller can do:

save x16, 17
bl plt
restore x16, x17

the plt will do:

get_addr_of_callee_into_x16_and_clobber_x17_or_something
br callee

then the callee will be compiled with all those weird options, but *not*
the ones specifying x16 and x17. That means it can happily use those guys
as scratch, because the caller will take care of them.

Will

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Ard Biesheuvel]

On 25 February 2016 at 17:56, Will Deacon <will.deacon@arm.com> wrote:
> On Thu, Feb 25, 2016 at 05:50:17PM +0100, Ard Biesheuvel wrote:
>> On 25 February 2016 at 17:49, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
>> > On 25 February 2016 at 17:46, Will Deacon <will.deacon@arm.com> wrote:
>> >> On Thu, Feb 25, 2016 at 05:43:35PM +0100, Ard Biesheuvel wrote:
>> >>> On 25 February 2016 at 17:42, Will Deacon <will.deacon@arm.com> wrote:
>> >>> > On Thu, Feb 25, 2016 at 05:33:25PM +0100, Ard Biesheuvel wrote:
>> >>> >> AFAIK, gcc never uses x18 (the platform register) so we may be able to
>> >>> >> use that instead. We'd need confirmation from the toolchain guys,
>> >>> >> though ...
>> >>> >
>> >>> > In fact, a better thing to do is probably for the atomic code to
>> >>> > save/restore those register explicitly and then remove them from the
>> >>> > cflags above.
>> >>> >
>> >>> > I'll try hacking something together...
>> >>> >
>> >>>
>> >>> That would be more correct, indeed. Also, the PLT can only use br
>> >>> <reg> so it will always enter the callee with the preserved value
>> >>> stacked, and so the callee needs to be modified in any case.
>> >>
>> >> Wait -- why does the callee need to be modified here? It has no idea
>> >> about whether or not it was invoked via a PLT.
>> >>
>> >
>> > No, it doesn't, and that is exactly the problem: the PLT must end in a
>> > br <reg> instruction because that is the only branch instruction with
>> > unlimited range. That means you will always enter the callee with its
>> > address in some register rather than the value that should have been
>> > preserved. I don't see a way to manage that from the callee
>> >
>>
>> *caller! dammit
>
> The caller can do:
>
> save x16, 17
> bl plt
> restore x16, x17
>

This is __LL_SC_CALL, right? So you just fold a stp/ldp in there?

> the plt will do:
>
> get_addr_of_callee_into_x16_and_clobber_x17_or_something
> br callee
>
> then the callee will be compiled with all those weird options, but *not*
> the ones specifying x16 and x17. That means it can happily use those guys
> as scratch, because the caller will take care of them.
>

Yes, that makes sense. But if you don't relax that restriction, you
only need the alternative __LL_SC_CALL for modules.

[PATCH v5sub2 1/8] arm64: add support for module PLTs

[Will Deacon]

On Thu, Feb 25, 2016 at 06:31:04PM +0100, Ard Biesheuvel wrote:
> On 25 February 2016 at 17:56, Will Deacon <will.deacon@arm.com> wrote:
> > the plt will do:
> >
> > get_addr_of_callee_into_x16_and_clobber_x17_or_something
> > br callee
> >
> > then the callee will be compiled with all those weird options, but *not*
> > the ones specifying x16 and x17. That means it can happily use those guys
> > as scratch, because the caller will take care of them.
> >
> 
> Yes, that makes sense. But if you don't relax that restriction, you
> only need the alternative __LL_SC_CALL for modules.

Totally untested patch below, but I'll give it a whirl tomorrow.

It looks a bit mental.

Will

--->8

diff --git a/arch/arm64/include/asm/atomic_lse.h b/arch/arm64/include/asm/atomic_lse.h
index 197e06afbf71..5b595b32ab40 100644
--- a/arch/arm64/include/asm/atomic_lse.h
+++ b/arch/arm64/include/asm/atomic_lse.h
@@ -33,6 +33,8 @@ static inline void atomic_andnot(int i, atomic_t *v)
 	register atomic_t *x1 asm ("x1") = v;
 
 	asm volatile(ARM64_LSE_ATOMIC_INSN(__LL_SC_ATOMIC(andnot),
+	"	nop\n"
+	"	nop\n"
 	"	stclr	%w[i], %[v]\n")
 	: [i] "+r" (w0), [v] "+Q" (v->counter)
 	: "r" (x1)
@@ -45,6 +47,8 @@ static inline void atomic_or(int i, atomic_t *v)
 	register atomic_t *x1 asm ("x1") = v;
 
 	asm volatile(ARM64_LSE_ATOMIC_INSN(__LL_SC_ATOMIC(or),
+	"	nop\n"
+	"	nop\n"
 	"	stset	%w[i], %[v]\n")
 	: [i] "+r" (w0), [v] "+Q" (v->counter)
 	: "r" (x1)
@@ -57,6 +61,8 @@ static inline void atomic_xor(int i, atomic_t *v)
 	register atomic_t *x1 asm ("x1") = v;
 
 	asm volatile(ARM64_LSE_ATOMIC_INSN(__LL_SC_ATOMIC(xor),
+	"	nop\n"
+	"	nop\n"
 	"	steor	%w[i], %[v]\n")
 	: [i] "+r" (w0), [v] "+Q" (v->counter)
 	: "r" (x1)
@@ -69,6 +75,8 @@ static inline void atomic_add(int i, atomic_t *v)
 	register atomic_t *x1 asm ("x1") = v;
 
 	asm volatile(ARM64_LSE_ATOMIC_INSN(__LL_SC_ATOMIC(add),
+	"	nop\n"
+	"	nop\n"
 	"	stadd	%w[i], %[v]\n")
 	: [i] "+r" (w0), [v] "+Q" (v->counter)
 	: "r" (x1)
@@ -83,9 +91,9 @@ static inline int atomic_add_return##name(int i, atomic_t *v)		\
 									\
 	asm volatile(ARM64_LSE_ATOMIC_INSN(				\
 	/* LL/SC */							\
-	"	nop\n"							\
 	__LL_SC_ATOMIC(add_return##name),				\
 	/* LSE atomics */						\
+	"	nop\n"							\
 	"	ldadd" #mb "	%w[i], w30, %[v]\n"			\
 	"	add	%w[i], %w[i], w30")				\
 	: [i] "+r" (w0), [v] "+Q" (v->counter)				\
@@ -109,9 +117,9 @@ static inline void atomic_and(int i, atomic_t *v)
 
 	asm volatile(ARM64_LSE_ATOMIC_INSN(
 	/* LL/SC */
-	"	nop\n"
 	__LL_SC_ATOMIC(and),
 	/* LSE atomics */
+	"	nop\n"
 	"	mvn	%w[i], %w[i]\n"
 	"	stclr	%w[i], %[v]")
 	: [i] "+r" (w0), [v] "+Q" (v->counter)
@@ -126,9 +134,9 @@ static inline void atomic_sub(int i, atomic_t *v)
 
 	asm volatile(ARM64_LSE_ATOMIC_INSN(
 	/* LL/SC */
-	"	nop\n"
 	__LL_SC_ATOMIC(sub),
 	/* LSE atomics */
+	"	nop\n"
 	"	neg	%w[i], %w[i]\n"
 	"	stadd	%w[i], %[v]")
 	: [i] "+r" (w0), [v] "+Q" (v->counter)
@@ -144,9 +152,7 @@ static inline int atomic_sub_return##name(int i, atomic_t *v)		\
 									\
 	asm volatile(ARM64_LSE_ATOMIC_INSN(				\
 	/* LL/SC */							\
-	"	nop\n"							\
-	__LL_SC_ATOMIC(sub_return##name)				\
-	"	nop",							\
+	__LL_SC_ATOMIC(sub_return##name),				\
 	/* LSE atomics */						\
 	"	neg	%w[i], %w[i]\n"					\
 	"	ldadd" #mb "	%w[i], w30, %[v]\n"			\
@@ -174,6 +180,8 @@ static inline void atomic64_andnot(long i, atomic64_t *v)
 	register atomic64_t *x1 asm ("x1") = v;
 
 	asm volatile(ARM64_LSE_ATOMIC_INSN(__LL_SC_ATOMIC64(andnot),
+	"	nop\n"
+	"	nop\n"
 	"	stclr	%[i], %[v]\n")
 	: [i] "+r" (x0), [v] "+Q" (v->counter)
 	: "r" (x1)
@@ -186,6 +194,8 @@ static inline void atomic64_or(long i, atomic64_t *v)
 	register atomic64_t *x1 asm ("x1") = v;
 
 	asm volatile(ARM64_LSE_ATOMIC_INSN(__LL_SC_ATOMIC64(or),
+	"	nop\n"
+	"	nop\n"
 	"	stset	%[i], %[v]\n")
 	: [i] "+r" (x0), [v] "+Q" (v->counter)
 	: "r" (x1)
@@ -198,6 +208,8 @@ static inline void atomic64_xor(long i, atomic64_t *v)
 	register atomic64_t *x1 asm ("x1") = v;
 
 	asm volatile(ARM64_LSE_ATOMIC_INSN(__LL_SC_ATOMIC64(xor),
+	"	nop\n"
+	"	nop\n"
 	"	steor	%[i], %[v]\n")
 	: [i] "+r" (x0), [v] "+Q" (v->counter)
 	: "r" (x1)
@@ -210,6 +222,8 @@ static inline void atomic64_add(long i, atomic64_t *v)
 	register atomic64_t *x1 asm ("x1") = v;
 
 	asm volatile(ARM64_LSE_ATOMIC_INSN(__LL_SC_ATOMIC64(add),
+	"	nop\n"
+	"	nop\n"
 	"	stadd	%[i], %[v]\n")
 	: [i] "+r" (x0), [v] "+Q" (v->counter)
 	: "r" (x1)
@@ -224,9 +238,9 @@ static inline long atomic64_add_return##name(long i, atomic64_t *v)	\
 									\
 	asm volatile(ARM64_LSE_ATOMIC_INSN(				\
 	/* LL/SC */							\
-	"	nop\n"							\
 	__LL_SC_ATOMIC64(add_return##name),				\
 	/* LSE atomics */						\
+	"	nop\n"							\
 	"	ldadd" #mb "	%[i], x30, %[v]\n"			\
 	"	add	%[i], %[i], x30")				\
 	: [i] "+r" (x0), [v] "+Q" (v->counter)				\
@@ -250,9 +264,9 @@ static inline void atomic64_and(long i, atomic64_t *v)
 
 	asm volatile(ARM64_LSE_ATOMIC_INSN(
 	/* LL/SC */
-	"	nop\n"
 	__LL_SC_ATOMIC64(and),
 	/* LSE atomics */
+	"	nop\n"
 	"	mvn	%[i], %[i]\n"
 	"	stclr	%[i], %[v]")
 	: [i] "+r" (x0), [v] "+Q" (v->counter)
@@ -267,9 +281,9 @@ static inline void atomic64_sub(long i, atomic64_t *v)
 
 	asm volatile(ARM64_LSE_ATOMIC_INSN(
 	/* LL/SC */
-	"	nop\n"
 	__LL_SC_ATOMIC64(sub),
 	/* LSE atomics */
+	"	nop\n"
 	"	neg	%[i], %[i]\n"
 	"	stadd	%[i], %[v]")
 	: [i] "+r" (x0), [v] "+Q" (v->counter)
@@ -285,9 +299,7 @@ static inline long atomic64_sub_return##name(long i, atomic64_t *v)	\
 									\
 	asm volatile(ARM64_LSE_ATOMIC_INSN(				\
 	/* LL/SC */							\
-	"	nop\n"							\
-	__LL_SC_ATOMIC64(sub_return##name)				\
-	"	nop",							\
+	__LL_SC_ATOMIC64(sub_return##name),				\
 	/* LSE atomics */						\
 	"	neg	%[i], %[i]\n"					\
 	"	ldadd" #mb "	%[i], x30, %[v]\n"			\
@@ -312,12 +324,10 @@ static inline long atomic64_dec_if_positive(atomic64_t *v)
 
 	asm volatile(ARM64_LSE_ATOMIC_INSN(
 	/* LL/SC */
-	"	nop\n"
 	__LL_SC_ATOMIC64(dec_if_positive)
 	"	nop\n"
 	"	nop\n"
 	"	nop\n"
-	"	nop\n"
 	"	nop",
 	/* LSE atomics */
 	"1:	ldr	x30, %[v]\n"
@@ -350,9 +360,7 @@ static inline unsigned long __cmpxchg_case_##name(volatile void *ptr,	\
 									\
 	asm volatile(ARM64_LSE_ATOMIC_INSN(				\
 	/* LL/SC */							\
-	"	nop\n"							\
-		__LL_SC_CMPXCHG(name)					\
-	"	nop",							\
+		__LL_SC_CMPXCHG(name),					\
 	/* LSE atomics */						\
 	"	mov	" #w "30, %" #w "[old]\n"			\
 	"	cas" #mb #sz "\t" #w "30, %" #w "[new], %[v]\n"		\
@@ -404,8 +412,6 @@ static inline long __cmpxchg_double##name(unsigned long old1,		\
 	asm volatile(ARM64_LSE_ATOMIC_INSN(				\
 	/* LL/SC */							\
 	"	nop\n"							\
-	"	nop\n"							\
-	"	nop\n"							\
 	__LL_SC_CMPXCHG_DBL(name),					\
 	/* LSE atomics */						\
 	"	casp" #mb "\t%[old1], %[old2], %[new1], %[new2], %[v]\n"\
diff --git a/arch/arm64/include/asm/lse.h b/arch/arm64/include/asm/lse.h
index 3de42d68611d..8252dc6b3046 100644
--- a/arch/arm64/include/asm/lse.h
+++ b/arch/arm64/include/asm/lse.h
@@ -25,7 +25,10 @@ __asm__(".arch_extension	lse");
 #define __LL_SC_EXPORT(x)	EXPORT_SYMBOL(__LL_SC_PREFIX(x))
 
 /* Macro for constructing calls to out-of-line ll/sc atomics */
-#define __LL_SC_CALL(op)	"bl\t" __stringify(__LL_SC_PREFIX(op)) "\n"
+#define __LL_SC_CALL(op)						\
+	"stp	x16, x17, [sp, #-16]!\n"				\
+	"bl\t" __stringify(__LL_SC_PREFIX(op)) "\n"			\
+	"ldp	x16, x17, [sp], #16\n"
 
 /* In-line patching at runtime */
 #define ARM64_LSE_ATOMIC_INSN(llsc, lse)				\
diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile
index 1a811ecf71da..c86b7909ef31 100644
--- a/arch/arm64/lib/Makefile
+++ b/arch/arm64/lib/Makefile
@@ -4,15 +4,16 @@ lib-y		:= bitops.o clear_user.o delay.o copy_from_user.o	\
 		   memcmp.o strcmp.o strncmp.o strlen.o strnlen.o	\
 		   strchr.o strrchr.o
 
-# Tell the compiler to treat all general purpose registers as
-# callee-saved, which allows for efficient runtime patching of the bl
-# instruction in the caller with an atomic instruction when supported by
-# the CPU. Result and argument registers are handled correctly, based on
-# the function prototype.
+# Tell the compiler to treat all general purpose registers (with the
+# exception of the IP registers, which are already handled by the caller
+# in case of a PLT) as callee-saved, which allows for efficient runtime
+# patching of the bl instruction in the caller with an atomic instruction
+# when supported by the CPU. Result and argument registers are handled
+# correctly, based on the function prototype.
 lib-$(CONFIG_ARM64_LSE_ATOMICS) += atomic_ll_sc.o
 CFLAGS_atomic_ll_sc.o	:= -fcall-used-x0 -ffixed-x1 -ffixed-x2		\
 		   -ffixed-x3 -ffixed-x4 -ffixed-x5 -ffixed-x6		\
 		   -ffixed-x7 -fcall-saved-x8 -fcall-saved-x9		\
 		   -fcall-saved-x10 -fcall-saved-x11 -fcall-saved-x12	\
 		   -fcall-saved-x13 -fcall-saved-x14 -fcall-saved-x15	\
-		   -fcall-saved-x16 -fcall-saved-x17 -fcall-saved-x18
+		   -fcall-saved-x18