[PATCH v2 00/11] arm64: vdso: getcpu() support

[PATCH v2 00/11] arm64: vdso: getcpu() support

[Mark Brown]

This series is a rebase of the previously posted getcpu() support with
some additional patches 5-10 added which try to do some cleanups and
clarifications of the vDSO code and extend it to multi-page support.
Those patches are currently drafts and haven't been fully tested or
considered, they're posted as there was some discussion of other
applications of the per-CPU data so it seemed useful to share this in
progress work.

Some applications, especially tracing ones, benefit from avoiding the
syscall overhead for getcpu() so it is common for architectures to have
vDSO implementations. Add one for arm64, using TPIDRRO_EL0 to pass a
pointer to per-CPU data rather than just store the immediate value in
order to allow for future extensibility.

It is questionable if something TPIDRRO_EL0 based is worthwhile at all
on current kernels, since v4.18 we have had support for restartable
sequences which can be used to provide a sched_getcpu() implementation
with generally better performance than the vDSO approach on
architectures which have that[1].  Work is ongoing to implement this for
glibc:

    https://lore.kernel.org/lkml/20200527185130.5604-3-mathieu.desnoyers@efficio
+s.com/

but is not yet merged and will need similar work for other userspaces.
The main advantages for the vDSO implementation are the node parameter
(though this is a static mapping to CPU number so could be looked up
separately when processing data if it's needed, it shouldn't need to be
in the hot path) and ease of implementation for users.

This is currently not compatible with KPTI due to the use of TPIDRRO_EL0
by the KPTI trampoline, this could be addressed by reinitializing that
system register in the return path but I have found it hard to justify
adding that overhead for all users for something that is essentially a
profiling optimization which is likely to get superceeded by a more
modern implementation - if there are other uses for the per-CPU data
then the balance might change here.

There is some overlap with an in flight patch series from Andrei Vagin
supporting time namespaces in the vDSO, there shouldn't be a fundamental
issue integrating the two serieses.

This builds on work done by Kristina Martsenko some time ago but is a
new implementation.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d7822b1e24f2df5df98c76f0e94a5416349ff759

v2:
 - Rebase on v5.8-rc3.
 - Add further cleanup patches & a first draft of multi-page support.

Mark Brown (11):
  arm64: vdso: Provide a define when building the vDSO
  arm64: vdso: Add per-CPU data
  arm64: vdso: Initialise the per-CPU vDSO data
  arm64: vdso: Add getcpu() implementation
  arm64: vdso: Remove union in declaration of the data store
  arm64: vdso: Document and verify alignment of vDSO text
  arm64: vdso: Rename vdso_pages to vdso_text_pages
  arm64: vdso: Simplify pagelist allocation
  arm64: vdso: Parameterise vDSO data length assumptions in code
  arm64: vdso: Support multiple pages of vDSO data
  selftests: vdso: Support arm64 in getcpu() test

 arch/arm64/include/asm/processor.h            |  12 +-
 arch/arm64/include/asm/vdso.h                 |  11 ++
 arch/arm64/include/asm/vdso/datapage.h        |  54 +++++++++
 arch/arm64/kernel/process.c                   |  26 +++-
 arch/arm64/kernel/vdso.c                      | 112 ++++++++++++------
 arch/arm64/kernel/vdso/Makefile               |   4 +-
 arch/arm64/kernel/vdso/vdso.lds.S             |   3 +-
 arch/arm64/kernel/vdso/vgetcpu.c              |  48 ++++++++
 .../testing/selftests/vDSO/vdso_test_getcpu.c |  10 ++
 9 files changed, 229 insertions(+), 51 deletions(-)
 create mode 100644 arch/arm64/include/asm/vdso/datapage.h
 create mode 100644 arch/arm64/kernel/vdso/vgetcpu.c


base-commit: 9ebcfadb0610322ac537dd7aa5d9cbc2b2894c68
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 01/11] arm64: vdso: Provide a define when building the vDSO

[Mark Brown]

Provide a define identifying if code is being built for the vDSO to help
with writing headers that are shared between the kernel and the vDSO.

Signed-off-by: Mark Brown <broonie@kernel.org>
---
 arch/arm64/kernel/vdso/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/kernel/vdso/Makefile b/arch/arm64/kernel/vdso/Makefile
index 45d5cfe46429..88cf0f0b91ed 100644
--- a/arch/arm64/kernel/vdso/Makefile
+++ b/arch/arm64/kernel/vdso/Makefile
@@ -28,7 +28,7 @@ ldflags-y := -shared -nostdlib -soname=linux-vdso.so.1 --hash-style=sysv	\
 	     $(btildflags-y) -T
 
 ccflags-y := -fno-common -fno-builtin -fno-stack-protector -ffixed-x18
-ccflags-y += -DDISABLE_BRANCH_PROFILING
+ccflags-y += -DDISABLE_BRANCH_PROFILING -D__VDSO__
 
 CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS) $(GCC_PLUGINS_CFLAGS)
 KBUILD_CFLAGS			+= $(DISABLE_LTO)
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 02/11] arm64: vdso: Add per-CPU data

[Mark Brown]

In order to support a vDSO getcpu() implementation add per-CPU data to
the vDSO data page. Do this by wrapping the generic vdso_data struct in
an arm64 specific one with an array of per-CPU data. The offset of the
per-CPU data applying to a CPU will be stored in TPIDRRO_EL0, this
allows us to get to the per-CPU data without doing any multiplications.

Since we currently only map a single data page for the vDSO but support
very large numbers of CPUs TPIDRRO may be set to zero for CPUs which don't
fit in the data page. This will also happen when KPTI is active since
kernel_ventry uses TPIDRRO_EL0 as a scratch register in that case, add a
comment to the code explaining this.

Acessors for the data are provided in the header since they will be needed
in multiple files and it seems neater to keep things together.

Signed-off-by: Mark Brown <broonie@kernel.org>
---
 arch/arm64/include/asm/processor.h     | 12 +-----
 arch/arm64/include/asm/vdso/datapage.h | 54 ++++++++++++++++++++++++++
 arch/arm64/kernel/process.c            | 26 ++++++++++++-
 arch/arm64/kernel/vdso.c               |  5 ++-
 4 files changed, 83 insertions(+), 14 deletions(-)
 create mode 100644 arch/arm64/include/asm/vdso/datapage.h

diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
index 240fe5e5b720..db7a804030b3 100644
--- a/arch/arm64/include/asm/processor.h
+++ b/arch/arm64/include/asm/processor.h
@@ -207,17 +207,7 @@ static inline void set_compat_ssbs_bit(struct pt_regs *regs)
 	regs->pstate |= PSR_AA32_SSBS_BIT;
 }
 
-static inline void start_thread(struct pt_regs *regs, unsigned long pc,
-				unsigned long sp)
-{
-	start_thread_common(regs, pc);
-	regs->pstate = PSR_MODE_EL0t;
-
-	if (arm64_get_ssbd_state() != ARM64_SSBD_FORCE_ENABLE)
-		set_ssbs_bit(regs);
-
-	regs->sp = sp;
-}
+void start_thread(struct pt_regs *regs, unsigned long pc, unsigned long sp);
 
 static inline bool is_ttbr0_addr(unsigned long addr)
 {
diff --git a/arch/arm64/include/asm/vdso/datapage.h b/arch/arm64/include/asm/vdso/datapage.h
new file mode 100644
index 000000000000..e88d97238c52
--- /dev/null
+++ b/arch/arm64/include/asm/vdso/datapage.h
@@ -0,0 +1,54 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 ARM Limited
+ */
+#ifndef __ASM_VDSO_DATAPAGE_H
+#define __ASM_VDSO_DATAPAGE_H
+
+#include <vdso/datapage.h>
+
+struct vdso_cpu_data {
+	unsigned int cpu;
+	unsigned int node;
+};
+
+struct arm64_vdso_data {
+	/* Must be first in struct, we cast to vdso_data */
+	struct vdso_data data[CS_BASES];
+	struct vdso_cpu_data cpu_data[];
+};
+
+#ifdef __VDSO__
+static inline struct vdso_cpu_data *__vdso_cpu_data(void)
+{
+	unsigned long offset;
+
+	asm volatile(
+		"mrs %0, tpidrro_el0\n"
+	: "=r" (offset)
+	:
+	: "cc");
+
+	if (offset)
+		return (void *)(_vdso_data) + offset;
+
+	return NULL;
+}
+#else
+static inline size_t vdso_cpu_offset(void)
+{
+	size_t offset, data_end;
+
+	offset = offsetof(struct arm64_vdso_data, cpu_data) +
+		smp_processor_id() * sizeof(struct vdso_cpu_data);
+	data_end = offset + sizeof(struct vdso_cpu_data) + 1;
+
+	/* We only map a single page for vDSO data currently */
+	if (data_end > PAGE_SIZE)
+		return 0;
+
+	return offset;
+}
+#endif
+
+#endif
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 6089638c7d43..b37fe0ceb1c9 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -55,6 +55,7 @@
 #include <asm/processor.h>
 #include <asm/pointer_auth.h>
 #include <asm/stacktrace.h>
+#include <asm/vdso/datapage.h>
 
 #if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_STACKPROTECTOR_PER_TASK)
 #include <linux/stackprotector.h>
@@ -309,6 +310,28 @@ void show_regs(struct pt_regs * regs)
 	dump_backtrace(regs, NULL, KERN_DEFAULT);
 }
 
+void start_thread(struct pt_regs *regs, unsigned long pc, unsigned long sp)
+{
+	start_thread_common(regs, pc);
+	regs->pstate = PSR_MODE_EL0t;
+
+	if (arm64_get_ssbd_state() != ARM64_SSBD_FORCE_ENABLE)
+		set_ssbs_bit(regs);
+
+	regs->sp = sp;
+
+	/*
+	 * Store the vDSO per-CPU offset if supported. Disable
+	 * preemption to make sure we read the CPU offset on the CPU
+	 * we write it on.
+	 */
+	if (!arm64_kernel_unmapped_at_el0()) {
+		preempt_disable();
+		write_sysreg(vdso_cpu_offset(), tpidrro_el0);
+		preempt_enable();
+	}
+}
+
 static void tls_thread_flush(void)
 {
 	write_sysreg(0, tpidr_el0);
@@ -452,7 +475,8 @@ static void tls_thread_switch(struct task_struct *next)
 	if (is_compat_thread(task_thread_info(next)))
 		write_sysreg(next->thread.uw.tp_value, tpidrro_el0);
 	else if (!arm64_kernel_unmapped_at_el0())
-		write_sysreg(0, tpidrro_el0);
+		/* Used as scratch in KPTI trampoline so don't set here. */
+		write_sysreg(vdso_cpu_offset(), tpidrro_el0);
 
 	write_sysreg(*task_user_tls(next), tpidr_el0);
 }
diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c
index e546df0efefb..9b117dcf9cb5 100644
--- a/arch/arm64/kernel/vdso.c
+++ b/arch/arm64/kernel/vdso.c
@@ -27,6 +27,7 @@
 #include <asm/cacheflush.h>
 #include <asm/signal32.h>
 #include <asm/vdso.h>
+#include <asm/vdso/datapage.h>
 
 extern char vdso_start[], vdso_end[];
 #ifdef CONFIG_COMPAT_VDSO
@@ -70,10 +71,10 @@ static struct vdso_abi_info vdso_info[] __ro_after_init = {
  * The vDSO data page.
  */
 static union {
-	struct vdso_data	data[CS_BASES];
+	struct arm64_vdso_data	data;
 	u8			page[PAGE_SIZE];
 } vdso_data_store __page_aligned_data;
-struct vdso_data *vdso_data = vdso_data_store.data;
+struct vdso_data *vdso_data = vdso_data_store.data.data;
 
 static int __vdso_remap(enum vdso_abi abi,
 			const struct vm_special_mapping *sm,
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 03/11] arm64: vdso: Initialise the per-CPU vDSO data

[Mark Brown]

Register with the CPU hotplug system to initialise the per-CPU data for
getcpu().

Signed-off-by: Mark Brown <broonie@kernel.org>
---
 arch/arm64/kernel/vdso.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c
index 9b117dcf9cb5..bdc18fc80eb3 100644
--- a/arch/arm64/kernel/vdso.c
+++ b/arch/arm64/kernel/vdso.c
@@ -9,6 +9,7 @@
 
 #include <linux/cache.h>
 #include <linux/clocksource.h>
+#include <linux/cpuhotplug.h>
 #include <linux/elf.h>
 #include <linux/err.h>
 #include <linux/errno.h>
@@ -18,6 +19,7 @@
 #include <linux/sched.h>
 #include <linux/signal.h>
 #include <linux/slab.h>
+#include <linux/smp.h>
 #include <linux/timekeeper_internal.h>
 #include <linux/vmalloc.h>
 #include <vdso/datapage.h>
@@ -358,6 +360,26 @@ int aarch32_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)
 }
 #endif /* CONFIG_COMPAT */
 
+static void vdso_cpu_init(void *p)
+{
+	struct arm64_vdso_data *data = (struct arm64_vdso_data *)vdso_data;
+	unsigned int cpu;
+
+	if (vdso_cpu_offset()) {
+		cpu = smp_processor_id();
+
+		data->cpu_data[cpu].cpu = cpu;
+		data->cpu_data[cpu].node = cpu_to_node(cpu);
+	}
+}
+
+static int vdso_cpu_online(unsigned int cpu)
+{
+	smp_call_function_single(cpu, vdso_cpu_init, NULL, 1);
+
+	return 0;
+}
+
 static int vdso_mremap(const struct vm_special_mapping *sm,
 		struct vm_area_struct *new_vma)
 {
@@ -384,6 +406,12 @@ static int __init vdso_init(void)
 	vdso_info[VDSO_ABI_AA64].dm = &aarch64_vdso_maps[AA64_MAP_VVAR];
 	vdso_info[VDSO_ABI_AA64].cm = &aarch64_vdso_maps[AA64_MAP_VDSO];
 
+	/*
+	 * Initialize per-CPU data, callback runs for all current and
+	 * future CPUs.
+	 */
+	cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "vdso", vdso_cpu_online, NULL);
+
 	return __vdso_init(VDSO_ABI_AA64);
 }
 arch_initcall(vdso_init);
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 04/11] arm64: vdso: Add getcpu() implementation

[Mark Brown]

Some applications, especially trace ones, benefit from avoiding the syscall
overhead on getcpu() calls so provide a vDSO implementation of it.

Signed-off-by: Mark Brown <broonie@kernel.org>
---
 arch/arm64/kernel/vdso/Makefile   |  2 +-
 arch/arm64/kernel/vdso/vdso.lds.S |  1 +
 arch/arm64/kernel/vdso/vgetcpu.c  | 48 +++++++++++++++++++++++++++++++
 3 files changed, 50 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/kernel/vdso/vgetcpu.c

diff --git a/arch/arm64/kernel/vdso/Makefile b/arch/arm64/kernel/vdso/Makefile
index 88cf0f0b91ed..ff350e69b8b6 100644
--- a/arch/arm64/kernel/vdso/Makefile
+++ b/arch/arm64/kernel/vdso/Makefile
@@ -11,7 +11,7 @@
 ARCH_REL_TYPE_ABS := R_AARCH64_JUMP_SLOT|R_AARCH64_GLOB_DAT|R_AARCH64_ABS64
 include $(srctree)/lib/vdso/Makefile
 
-obj-vdso := vgettimeofday.o note.o sigreturn.o
+obj-vdso := vgettimeofday.o note.o sigreturn.o vgetcpu.o
 
 # Build rules
 targets := $(obj-vdso) vdso.so vdso.so.dbg
diff --git a/arch/arm64/kernel/vdso/vdso.lds.S b/arch/arm64/kernel/vdso/vdso.lds.S
index 7ad2d3a0cd48..c2fd94f14b94 100644
--- a/arch/arm64/kernel/vdso/vdso.lds.S
+++ b/arch/arm64/kernel/vdso/vdso.lds.S
@@ -77,6 +77,7 @@ VERSION
 		__kernel_gettimeofday;
 		__kernel_clock_gettime;
 		__kernel_clock_getres;
+		__kernel_getcpu;
 	local: *;
 	};
 }
diff --git a/arch/arm64/kernel/vdso/vgetcpu.c b/arch/arm64/kernel/vdso/vgetcpu.c
new file mode 100644
index 000000000000..e8972e561e08
--- /dev/null
+++ b/arch/arm64/kernel/vdso/vgetcpu.c
@@ -0,0 +1,48 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ARM64 userspace implementations of getcpu()
+ *
+ * Copyright (C) 2020 ARM Limited
+ *
+ */
+
+#include <asm/unistd.h>
+#include <asm/vdso/datapage.h>
+
+struct getcpucache;
+
+static __always_inline
+int getcpu_fallback(unsigned int *_cpu, unsigned int *_node,
+		    struct getcpucache *_c)
+{
+	register unsigned int *cpu asm("x0") = _cpu;
+	register unsigned int *node asm("x1") = _node;
+	register struct getcpucache *c asm("x2") = _c;
+	register long ret asm ("x0");
+	register long nr asm("x8") = __NR_getcpu;
+
+	asm volatile(
+	"       svc #0\n"
+	: "=r" (ret)
+	: "r" (cpu), "r" (node), "r" (c), "r" (nr)
+	: "memory");
+
+	return ret;
+}
+
+int __kernel_getcpu(unsigned int *cpu, unsigned int *node,
+		    struct getcpucache *c)
+{
+	struct vdso_cpu_data *cpu_data = __vdso_cpu_data();
+
+	if (cpu_data) {
+		if (cpu)
+			*cpu = cpu_data->cpu;
+		if (node)
+			*node = cpu_data->node;
+
+		return 0;
+	}
+
+	return getcpu_fallback(cpu, node, c);
+}
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 05/11] arm64: vdso: Remove union in declaration of the data store

[Mark Brown]

Currently the vDSO data store is declared as part of a union with a page
of char for no documented reason, it *appears* that this has been done
for alignment purposes but we also have a __page_aligned_data annotation
so it's not 100% clear why and confusing when looking at the code.
Since it seems to be redundant with current toolchains remove the union.

Signed-off-by: Mark Brown <broonie@kernel.org>
---
 arch/arm64/kernel/vdso.c | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)

diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c
index bdc18fc80eb3..b05e4d5880af 100644
--- a/arch/arm64/kernel/vdso.c
+++ b/arch/arm64/kernel/vdso.c
@@ -69,14 +69,8 @@ static struct vdso_abi_info vdso_info[] __ro_after_init = {
 #endif /* CONFIG_COMPAT_VDSO */
 };
 
-/*
- * The vDSO data page.
- */
-static union {
-	struct arm64_vdso_data	data;
-	u8			page[PAGE_SIZE];
-} vdso_data_store __page_aligned_data;
-struct vdso_data *vdso_data = vdso_data_store.data.data;
+struct arm64_vdso_data vdso_data_store __page_aligned_data;
+struct vdso_data *vdso_data = vdso_data_store.data;
 
 static int __vdso_remap(enum vdso_abi abi,
 			const struct vm_special_mapping *sm,
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 06/11] arm64: vdso: Document and verify alignment of vDSO text

[Mark Brown]

We do no rounding when working out the number of vDSO text pages because
we ensure in vdso.S that the vDSO text is an exact number of pages long.
This is not immediately apparent when looking at the code that does the
vDSO mapping in vdso.c so add a comment explaining what's going on and
check that this assumption is true.

Ideally we'd have a BUILD_BUG_ON() but since the start and end come from
another compilation unit it needs to be a runtime check, add one which
prints a warning and errors out similarly to the existing check for the
vDSO not being an ELF object.

Signed-off-by: Mark Brown <broonie@kernel.org>
---
 arch/arm64/kernel/vdso.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c
index b05e4d5880af..95e69df42141 100644
--- a/arch/arm64/kernel/vdso.c
+++ b/arch/arm64/kernel/vdso.c
@@ -99,6 +99,15 @@ static int __vdso_init(enum vdso_abi abi)
 		return -EINVAL;
 	}
 
+	/*
+	 * We ensure that the vDSO text is page aligned and an exact
+	 * number of pages in vdso.S so don't need to round here.
+	 */
+	if (!PAGE_ALIGNED(vdso_info[abi].vdso_code_start) ||
+	    !PAGE_ALIGNED(vdso_info[abi].vdso_code_end)) {
+		pr_err("vDSO is not page aligned\n");
+		return -EINVAL;
+	}
 	vdso_info[abi].vdso_pages = (
 			vdso_info[abi].vdso_code_end -
 			vdso_info[abi].vdso_code_start) >>
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 07/11] arm64: vdso: Rename vdso_pages to vdso_text_pages

[Mark Brown]

In preparation for having multiple data pages rename the current vdso_pages
to vdso_text_pages.

Signed-off-by: Mark Brown <broonie@kernel.org>
---
 arch/arm64/kernel/vdso.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c
index 95e69df42141..032249f70460 100644
--- a/arch/arm64/kernel/vdso.c
+++ b/arch/arm64/kernel/vdso.c
@@ -47,7 +47,7 @@ struct vdso_abi_info {
 	const char *name;
 	const char *vdso_code_start;
 	const char *vdso_code_end;
-	unsigned long vdso_pages;
+	unsigned long vdso_text_pages;
 	/* Data Mapping */
 	struct vm_special_mapping *dm;
 	/* Code Mapping */
@@ -108,13 +108,13 @@ static int __vdso_init(enum vdso_abi abi)
 		pr_err("vDSO is not page aligned\n");
 		return -EINVAL;
 	}
-	vdso_info[abi].vdso_pages = (
+	vdso_info[abi].vdso_text_pages = (
 			vdso_info[abi].vdso_code_end -
 			vdso_info[abi].vdso_code_start) >>
 			PAGE_SHIFT;
 
 	/* Allocate the vDSO pagelist, plus a page for the data. */
-	vdso_pagelist = kcalloc(vdso_info[abi].vdso_pages + 1,
+	vdso_pagelist = kcalloc(vdso_info[abi].vdso_text_pages + 1,
 				sizeof(struct page *),
 				GFP_KERNEL);
 	if (vdso_pagelist == NULL)
@@ -127,7 +127,7 @@ static int __vdso_init(enum vdso_abi abi)
 	/* Grab the vDSO code pages. */
 	pfn = sym_to_pfn(vdso_info[abi].vdso_code_start);
 
-	for (i = 0; i < vdso_info[abi].vdso_pages; i++)
+	for (i = 0; i < vdso_info[abi].vdso_text_pages; i++)
 		vdso_pagelist[i + 1] = pfn_to_page(pfn + i);
 
 	vdso_info[abi].dm->pages = &vdso_pagelist[0];
@@ -145,7 +145,7 @@ static int __setup_additional_pages(enum vdso_abi abi,
 	unsigned long gp_flags = 0;
 	void *ret;
 
-	vdso_text_len = vdso_info[abi].vdso_pages << PAGE_SHIFT;
+	vdso_text_len = vdso_info[abi].vdso_text_pages << PAGE_SHIFT;
 	/* Be sure to map the data page */
 	vdso_mapping_len = vdso_text_len + PAGE_SIZE;
 
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 08/11] arm64: vdso: Simplify pagelist allocation

[Mark Brown]

Currently we allocate a single array for the struct page * for the vDSO
and then do the lookup for code and data in slightly different ways. Make
this more regular by factoring out the array allocation and struct page *
lookup into a function and using it for both. We don't currently rely on
everything being in one array so there should be no functional change.

Signed-off-by: Mark Brown <broonie@kernel.org>
---
 arch/arm64/kernel/vdso.c | 42 ++++++++++++++++++++--------------------
 1 file changed, 21 insertions(+), 21 deletions(-)

diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c
index 032249f70460..f78349faa6c4 100644
--- a/arch/arm64/kernel/vdso.c
+++ b/arch/arm64/kernel/vdso.c
@@ -88,12 +88,23 @@ static int __vdso_remap(enum vdso_abi abi,
 	return 0;
 }
 
-static int __vdso_init(enum vdso_abi abi)
+static struct page **vdso_get_pages(unsigned long pfn, size_t pages)
 {
+	struct page **pagelist;
 	int i;
-	struct page **vdso_pagelist;
-	unsigned long pfn;
 
+	pagelist = kcalloc(pages, sizeof(struct page *), GFP_KERNEL);
+	if (!pagelist)
+		return NULL;
+
+	for (i = 0; i < pages; i++)
+		pagelist[i] = pfn_to_page(pfn + i);
+
+	return pagelist;
+}
+
+static int __vdso_init(enum vdso_abi abi)
+{
 	if (memcmp(vdso_info[abi].vdso_code_start, "\177ELF", 4)) {
 		pr_err("vDSO is not a valid ELF object!\n");
 		return -EINVAL;
@@ -113,25 +124,14 @@ static int __vdso_init(enum vdso_abi abi)
 			vdso_info[abi].vdso_code_start) >>
 			PAGE_SHIFT;
 
-	/* Allocate the vDSO pagelist, plus a page for the data. */
-	vdso_pagelist = kcalloc(vdso_info[abi].vdso_text_pages + 1,
-				sizeof(struct page *),
-				GFP_KERNEL);
-	if (vdso_pagelist == NULL)
-		return -ENOMEM;
-
-	/* Grab the vDSO data page. */
-	vdso_pagelist[0] = phys_to_page(__pa_symbol(vdso_data));
-
+	vdso_info[abi].dm->pages =
+		vdso_get_pages(sym_to_pfn(vdso_data), 1);
+	vdso_info[abi].cm->pages =
+		vdso_get_pages(sym_to_pfn(vdso_info[abi].vdso_code_start),
+			       vdso_info[abi].vdso_text_pages);
 
-	/* Grab the vDSO code pages. */
-	pfn = sym_to_pfn(vdso_info[abi].vdso_code_start);
-
-	for (i = 0; i < vdso_info[abi].vdso_text_pages; i++)
-		vdso_pagelist[i + 1] = pfn_to_page(pfn + i);
-
-	vdso_info[abi].dm->pages = &vdso_pagelist[0];
-	vdso_info[abi].cm->pages = &vdso_pagelist[1];
+	if (!vdso_info[abi].cm->pages || !vdso_info[abi].dm->pages)
+		return -ENOMEM;
 
 	return 0;
 }
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 09/11] arm64: vdso: Parameterise vDSO data length assumptions in code

[Mark Brown]

In preparation for adding per-CPU data for the vDSO factor out the
assumptions about the vDSO having a single data page so that we use
a function vdso_data_size() to determine the actual length and a
vdso_data_pages to determine the number of pages mapped for data.
The actual data size and runtime behaviour are currently unaffected.

Signed-off-by: Mark Brown <broonie@kernel.org>
---
 arch/arm64/include/asm/vdso.h |  7 +++++++
 arch/arm64/kernel/vdso.c      | 18 ++++++++++++------
 2 files changed, 19 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/include/asm/vdso.h b/arch/arm64/include/asm/vdso.h
index 07468428fd29..c7edf7a7491f 100644
--- a/arch/arm64/include/asm/vdso.h
+++ b/arch/arm64/include/asm/vdso.h
@@ -12,6 +12,13 @@
  */
 #define VDSO_LBASE	0x0
 
+/*
+ * Hard code magic numbers since we need these in vdso.lds.S, there
+ * are BUILD_BUG_ON() checks in vdso.c to make sure these are correct.
+ */
+#define VDSO_BASE_DATA_SIZE 240
+#define VDSO_CS_BASES       2
+
 #ifndef __ASSEMBLY__
 
 #include <generated/vdso-offsets.h>
diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c
index f78349faa6c4..e1146f424e6f 100644
--- a/arch/arm64/kernel/vdso.c
+++ b/arch/arm64/kernel/vdso.c
@@ -48,6 +48,7 @@ struct vdso_abi_info {
 	const char *vdso_code_start;
 	const char *vdso_code_end;
 	unsigned long vdso_text_pages;
+	unsigned long vdso_data_pages;
 	/* Data Mapping */
 	struct vm_special_mapping *dm;
 	/* Code Mapping */
@@ -110,6 +111,8 @@ static int __vdso_init(enum vdso_abi abi)
 		return -EINVAL;
 	}
 
+	BUILD_BUG_ON(sizeof(struct vdso_data) != VDSO_BASE_DATA_SIZE);
+
 	/*
 	 * We ensure that the vDSO text is page aligned and an exact
 	 * number of pages in vdso.S so don't need to round here.
@@ -123,9 +126,11 @@ static int __vdso_init(enum vdso_abi abi)
 			vdso_info[abi].vdso_code_end -
 			vdso_info[abi].vdso_code_start) >>
 			PAGE_SHIFT;
+	vdso_info[abi].vdso_data_pages = VDSO_DATA_PAGES;
 
 	vdso_info[abi].dm->pages =
-		vdso_get_pages(sym_to_pfn(vdso_data), 1);
+		vdso_get_pages(sym_to_pfn(vdso_data),
+			       vdso_info[abi].vdso_data_pages);
 	vdso_info[abi].cm->pages =
 		vdso_get_pages(sym_to_pfn(vdso_info[abi].vdso_code_start),
 			       vdso_info[abi].vdso_text_pages);
@@ -141,13 +146,14 @@ static int __setup_additional_pages(enum vdso_abi abi,
 				    struct linux_binprm *bprm,
 				    int uses_interp)
 {
-	unsigned long vdso_base, vdso_text_len, vdso_mapping_len;
+	unsigned long vdso_base, vdso_mapping_len;
+	unsigned long vdso_text_len, vdso_data_len;
 	unsigned long gp_flags = 0;
 	void *ret;
 
 	vdso_text_len = vdso_info[abi].vdso_text_pages << PAGE_SHIFT;
-	/* Be sure to map the data page */
-	vdso_mapping_len = vdso_text_len + PAGE_SIZE;
+	vdso_data_len = vdso_info[abi].vdso_data_pages << PAGE_SHIFT;
+	vdso_mapping_len = vdso_text_len + vdso_data_len;
 
 	vdso_base = get_unmapped_area(NULL, 0, vdso_mapping_len, 0, 0);
 	if (IS_ERR_VALUE(vdso_base)) {
@@ -155,7 +161,7 @@ static int __setup_additional_pages(enum vdso_abi abi,
 		goto up_fail;
 	}
 
-	ret = _install_special_mapping(mm, vdso_base, PAGE_SIZE,
+	ret = _install_special_mapping(mm, vdso_base, vdso_data_len,
 				       VM_READ|VM_MAYREAD,
 				       vdso_info[abi].dm);
 	if (IS_ERR(ret))
@@ -164,7 +170,7 @@ static int __setup_additional_pages(enum vdso_abi abi,
 	if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL) && system_supports_bti())
 		gp_flags = VM_ARM64_BTI;
 
-	vdso_base += PAGE_SIZE;
+	vdso_base += vdso_data_len;
 	mm->context.vdso = (void *)vdso_base;
 	ret = _install_special_mapping(mm, vdso_base, vdso_text_len,
 				       VM_READ|VM_EXEC|gp_flags|
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 10/11] arm64: vdso: Support multiple pages of vDSO data

[Mark Brown]

In order to provide per-CPU data for systems with larger numbers of CPUs
we need multiple data pages for the vDSO, exactly how many will depend on
the page size and the maximum number of CPUs the kernel is configured to
support.

Since the vDSO references the data via a symbol defined in the linker
script we need to be able to reference the sizes of structures defined
in C which appears not to be something we have sensible support for.
Handle this by defining magic numbers and using BUILD_BUG_ON() which
checks that the numbers are accurate, this is rather distasteful but
works. Currently the patch duplicates CS_BASES, with the hard coding of
struct sizes it felt like more trouble than it was worth to move it into
an assembler clean header.

Signed-off-by: Mark Brown <broonie@kernel.org>
---
 arch/arm64/include/asm/vdso.h     | 4 ++++
 arch/arm64/kernel/vdso.c          | 2 ++
 arch/arm64/kernel/vdso/vdso.lds.S | 2 +-
 3 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/vdso.h b/arch/arm64/include/asm/vdso.h
index c7edf7a7491f..21e35d79022a 100644
--- a/arch/arm64/include/asm/vdso.h
+++ b/arch/arm64/include/asm/vdso.h
@@ -18,6 +18,10 @@
  */
 #define VDSO_BASE_DATA_SIZE 240
 #define VDSO_CS_BASES       2
+#define VDSO_PER_CPU_SIZE   8
+#define VDSO_DATA_SIZE      ((VDSO_PER_CPU_SIZE * CONFIG_NR_CPUS)	\
+			     + (VDSO_BASE_DATA_SIZE * VDSO_CS_BASES))
+#define VDSO_DATA_PAGES     ((VDSO_DATA_SIZE / PAGE_SIZE) + 1)
 
 #ifndef __ASSEMBLY__
 
diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c
index e1146f424e6f..b0899591d91e 100644
--- a/arch/arm64/kernel/vdso.c
+++ b/arch/arm64/kernel/vdso.c
@@ -112,6 +112,8 @@ static int __vdso_init(enum vdso_abi abi)
 	}
 
 	BUILD_BUG_ON(sizeof(struct vdso_data) != VDSO_BASE_DATA_SIZE);
+	BUILD_BUG_ON(CS_BASES != VDSO_CS_BASES);
+	BUILD_BUG_ON(sizeof(struct vdso_cpu_data) != VDSO_PER_CPU_SIZE);
 
 	/*
 	 * We ensure that the vDSO text is page aligned and an exact
diff --git a/arch/arm64/kernel/vdso/vdso.lds.S b/arch/arm64/kernel/vdso/vdso.lds.S
index c2fd94f14b94..7ee43b149bf9 100644
--- a/arch/arm64/kernel/vdso/vdso.lds.S
+++ b/arch/arm64/kernel/vdso/vdso.lds.S
@@ -17,7 +17,7 @@ OUTPUT_ARCH(aarch64)
 
 SECTIONS
 {
-	PROVIDE(_vdso_data = . - PAGE_SIZE);
+	PROVIDE(_vdso_data = . - (VDSO_DATA_PAGES * PAGE_SIZE));
 	. = VDSO_LBASE + SIZEOF_HEADERS;
 
 	.hash		: { *(.hash) }			:text
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v2 11/11] selftests: vdso: Support arm64 in getcpu() test

[Mark Brown]

arm64 exports the vDSO ABI with a version of LINUX_2.6.39 and symbols
prefixed with __kernel rather than __vdso. Update the getcpu() test to
handle this.

Signed-off-by: Mark Brown <broonie@kernel.org>
---
 tools/testing/selftests/vDSO/vdso_test_getcpu.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/tools/testing/selftests/vDSO/vdso_test_getcpu.c b/tools/testing/selftests/vDSO/vdso_test_getcpu.c
index fc25ede131b8..4aeb65012b81 100644
--- a/tools/testing/selftests/vDSO/vdso_test_getcpu.c
+++ b/tools/testing/selftests/vDSO/vdso_test_getcpu.c
@@ -14,8 +14,18 @@
 #include "../kselftest.h"
 #include "parse_vdso.h"
 
+/*
+ * ARM64's vDSO exports its getcpu() implementation with a different
+ * name and version from other architectures, so we need to handle it
+ * as a special case.
+ */
+#if defined(__aarch64__)
+const char *version = "LINUX_2.6.39";
+const char *name = "__kernel_getcpu";
+#else
 const char *version = "LINUX_2.6";
 const char *name = "__vdso_getcpu";
+#endif
 
 struct getcpu_cache;
 typedef long (*getcpu_t)(unsigned int *, unsigned int *,
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel