* [PATCH v2 0/9] arm64: optimize virt_to_page and page_address
@ 2016-02-29 14:44 ` Ard Biesheuvel
0 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel, catalin.marinas, will.deacon, mark.rutland
Cc: linux-mm, akpm, nios2-dev, lftan, jonas, linux, Ard Biesheuvel
Apologies for the wall of text. This is a followup to 'restrict virt_to_page
to linear region (instead of __pa)' [1], posted on the 24th of this month.
This series applies onto current arm64/for-next/core (5be8b70af1ca 'arm64:
lse: deal with clobbered IP registers after branch via PLT') with
arm64/fixes/core (dfd55ad85e4a 'arm64: vmemmap: use virtual projection of
linear region') merged into it. Complete branch can be found here [2].
In current -next, the implementations of virt_to_page and its converse
[lowmem_]page_address resolve to
virt_to_page
6f0: b6300180 tbz x0, #38, 720 <bar+0x30>
6f4: 90000001 adrp x1, 0 <memstart_addr>
6f8: 92409400 and x0, x0, #0x3fffffffff
6fc: f9400021 ldr x1, [x1]
700: 8b000020 add x0, x1, x0
704: 370001e1 tbnz w1, #0, 740 <bar+0x50>
708: d34cfc00 lsr x0, x0, #12
70c: d2dff7c1 mov x1, #0xffbe00000000
710: f2ffffe1 movk x1, #0xffff, lsl #48
714: 8b001820 add x0, x1, x0, lsl #6
718: d65f03c0 ret
71c: d503201f nop
720: 90000001 adrp x1, 0 <init_pgd>
724: f9400021 ldr x1, [x1]
728: cb010000 sub x0, x0, x1
72c: d2dff7c1 mov x1, #0xffbe00000000
730: d34cfc00 lsr x0, x0, #12
734: f2ffffe1 movk x1, #0xffff, lsl #48
738: 8b001820 add x0, x1, x0, lsl #6
73c: d65f03c0 ret
740: d4210000 brk #0x800
page_address:
6c0: 90000002 adrp x2, 0 <memstart_addr>
6c4: d2c00841 mov x1, #0x4200000000
6c8: 8b010000 add x0, x0, x1
6cc: 9346fc01 asr x1, x0, #6
6d0: f9400040 ldr x0, [x2]
6d4: d374cc21 lsl x1, x1, #12
6d8: 37000080 tbnz w0, #0, 6e8 <foo+0x28>
6dc: cb000020 sub x0, x1, x0
6e0: b25a6400 orr x0, x0, #0xffffffc000000000
6e4: d65f03c0 ret
6e8: d4210000 brk #0x800
Disappointingly, after cherrypicking commit dfd55ad85e ("arm64: vmemmap:
use virtual projection of linear region") from arm64/fixes/core, which
should make this translation independent of the physical start of RAM,
things don't improve much, and even get slightly worse:
virt_to_page
6f8: b6300180 tbz x0, #38, 728 <bar+0x30>
6fc: 90000001 adrp x1, 0 <memstart_addr>
700: 92409400 and x0, x0, #0x3fffffffff
704: f9400021 ldr x1, [x1]
708: 8b000020 add x0, x1, x0
70c: 37000261 tbnz w1, #0, 758 <bar+0x60>
710: d34cfc00 lsr x0, x0, #12
714: d2dff7c2 mov x2, #0xffbe00000000
718: cb813000 sub x0, x0, x1, asr #12
71c: f2ffffe2 movk x2, #0xffff, lsl #48
720: 8b001840 add x0, x2, x0, lsl #6
724: d65f03c0 ret
728: 90000002 adrp x2, 0 <init_pgd>
72c: 90000001 adrp x1, 0 <memstart_addr>
730: f9400042 ldr x2, [x2]
734: f9400021 ldr x1, [x1]
738: cb020000 sub x0, x0, x2
73c: d2dff7c2 mov x2, #0xffbe00000000
740: d34cfc00 lsr x0, x0, #12
744: f2ffffe2 movk x2, #0xffff, lsl #48
748: cb813000 sub x0, x0, x1, asr #12
74c: 8b001840 add x0, x2, x0, lsl #6
750: d65f03c0 ret
754: d503201f nop
758: d4210000 brk #0x800
page_address:
6c0: 90000002 adrp x2, 0 <memstart_addr>
6c4: d2c00841 mov x1, #0x4200000000
6c8: f9400043 ldr x3, [x2]
6cc: 934cfc62 asr x2, x3, #12
6d0: 8b021800 add x0, x0, x2, lsl #6
6d4: 8b010001 add x1, x0, x1
6d8: 9346fc21 asr x1, x1, #6
6dc: d374cc21 lsl x1, x1, #12
6e0: 37000083 tbnz w3, #0, 6f0 <foo+0x30>
6e4: cb030020 sub x0, x1, x3
6e8: b25a6400 orr x0, x0, #0xffffffc000000000
6ec: d65f03c0 ret
6f0: d4210000 brk #0x800
The thing to note here is that the expression is evaluated in a way that
does not allow the compiler to eliminate the read of memstart_addr,
presumably since it is unaware that its value is aligned to PAGE_SIZE, and
that shifting it down and up again by 12 bits produces the exact same value.
So let's give the compiler a hand here. First of all, let's reimplement
virt_to_page() (patch #6) so that it explicitly translates without taking
the physical placement into account. This results in the virt_to_page()
translation to only work correctly for addresses above PAGE_OFFSET, but
this is a reasonable restriction to impose, even if it means a couple of
incorrect uses need to be fixed (patches #1 to #4). If we also, in patch #5,
move the vmemmap region right below the linear region (which guarantees that
the region is always aligned to a power-of-2 upper bound of its size, which
means we can treat VMEMMAP_START as a bitmask rather than an offset), we end
up with
virt_to_page
6d0: d34c9400 ubfx x0, x0, #12, #26
6d4: d2dff7c1 mov x1, #0xffbe00000000
6d8: f2ffffe1 movk x1, #0xffff, lsl #48
6dc: aa001820 orr x0, x1, x0, lsl #6
6e0: d65f03c0 ret
In the same way, we can get page_address to look like this
page_address:
6c0: d37a7c00 ubfiz x0, x0, #6, #32
6c4: b25a6400 orr x0, x0, #0xffffffc000000000
6c8: d65f03c0 ret
However, in this case, we need to slightly refactor the implementation of
lowmem_page_paddress(), since it performs an explicit page-to-pa-to-va
translation, rather than going through an opaque arch-defined definition
of page_to_virt. (patches #7 to #9)
[1] http://thread.gmane.org/gmane.linux.ports.arm.kernel/481327
[2] https://git.linaro.org/people/ard.biesheuvel/linux-arm.git/shortlog/refs/heads/arm64-vmemmap
(git url git://git.linaro.org/people/ard.biesheuvel/linux-arm.git arm64-vmemmap)
Ard Biesheuvel (9):
arm64: vdso: avoid virt_to_page() translations on kernel symbols
arm64: mm: free __init memory via the linear mapping
arm64: mm: avoid virt_to_page() translation for the zero page
arm64: insn: avoid virt_to_page() translations on core kernel symbols
arm64: mm: move vmemmap region right below the linear region
arm64: mm: restrict virt_to_page() to the linear mapping
nios2: use correct void* return type for page_to_virt()
openrisc: drop wrongly typed definition of page_to_virt()
mm: replace open coded page to virt conversion with page_to_virt()
arch/arm64/include/asm/memory.h | 30 ++++++++++++++++++--
arch/arm64/include/asm/pgtable.h | 13 +++------
arch/arm64/kernel/insn.c | 2 +-
arch/arm64/kernel/vdso.c | 7 +++--
arch/arm64/mm/dump.c | 16 +++++------
arch/arm64/mm/init.c | 19 +++++++++----
arch/nios2/include/asm/io.h | 1 -
arch/nios2/include/asm/page.h | 2 +-
arch/nios2/include/asm/pgtable.h | 2 +-
arch/openrisc/include/asm/page.h | 2 --
include/linux/mm.h | 6 +++-
11 files changed, 65 insertions(+), 35 deletions(-)
--
2.5.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH v2 0/9] arm64: optimize virt_to_page and page_address
@ 2016-02-29 14:44 ` Ard Biesheuvel
0 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel
Apologies for the wall of text. This is a followup to 'restrict virt_to_page
to linear region (instead of __pa)' [1], posted on the 24th of this month.
This series applies onto current arm64/for-next/core (5be8b70af1ca 'arm64:
lse: deal with clobbered IP registers after branch via PLT') with
arm64/fixes/core (dfd55ad85e4a 'arm64: vmemmap: use virtual projection of
linear region') merged into it. Complete branch can be found here [2].
In current -next, the implementations of virt_to_page and its converse
[lowmem_]page_address resolve to
virt_to_page
6f0: b6300180 tbz x0, #38, 720 <bar+0x30>
6f4: 90000001 adrp x1, 0 <memstart_addr>
6f8: 92409400 and x0, x0, #0x3fffffffff
6fc: f9400021 ldr x1, [x1]
700: 8b000020 add x0, x1, x0
704: 370001e1 tbnz w1, #0, 740 <bar+0x50>
708: d34cfc00 lsr x0, x0, #12
70c: d2dff7c1 mov x1, #0xffbe00000000
710: f2ffffe1 movk x1, #0xffff, lsl #48
714: 8b001820 add x0, x1, x0, lsl #6
718: d65f03c0 ret
71c: d503201f nop
720: 90000001 adrp x1, 0 <init_pgd>
724: f9400021 ldr x1, [x1]
728: cb010000 sub x0, x0, x1
72c: d2dff7c1 mov x1, #0xffbe00000000
730: d34cfc00 lsr x0, x0, #12
734: f2ffffe1 movk x1, #0xffff, lsl #48
738: 8b001820 add x0, x1, x0, lsl #6
73c: d65f03c0 ret
740: d4210000 brk #0x800
page_address:
6c0: 90000002 adrp x2, 0 <memstart_addr>
6c4: d2c00841 mov x1, #0x4200000000
6c8: 8b010000 add x0, x0, x1
6cc: 9346fc01 asr x1, x0, #6
6d0: f9400040 ldr x0, [x2]
6d4: d374cc21 lsl x1, x1, #12
6d8: 37000080 tbnz w0, #0, 6e8 <foo+0x28>
6dc: cb000020 sub x0, x1, x0
6e0: b25a6400 orr x0, x0, #0xffffffc000000000
6e4: d65f03c0 ret
6e8: d4210000 brk #0x800
Disappointingly, after cherrypicking commit dfd55ad85e ("arm64: vmemmap:
use virtual projection of linear region") from arm64/fixes/core, which
should make this translation independent of the physical start of RAM,
things don't improve much, and even get slightly worse:
virt_to_page
6f8: b6300180 tbz x0, #38, 728 <bar+0x30>
6fc: 90000001 adrp x1, 0 <memstart_addr>
700: 92409400 and x0, x0, #0x3fffffffff
704: f9400021 ldr x1, [x1]
708: 8b000020 add x0, x1, x0
70c: 37000261 tbnz w1, #0, 758 <bar+0x60>
710: d34cfc00 lsr x0, x0, #12
714: d2dff7c2 mov x2, #0xffbe00000000
718: cb813000 sub x0, x0, x1, asr #12
71c: f2ffffe2 movk x2, #0xffff, lsl #48
720: 8b001840 add x0, x2, x0, lsl #6
724: d65f03c0 ret
728: 90000002 adrp x2, 0 <init_pgd>
72c: 90000001 adrp x1, 0 <memstart_addr>
730: f9400042 ldr x2, [x2]
734: f9400021 ldr x1, [x1]
738: cb020000 sub x0, x0, x2
73c: d2dff7c2 mov x2, #0xffbe00000000
740: d34cfc00 lsr x0, x0, #12
744: f2ffffe2 movk x2, #0xffff, lsl #48
748: cb813000 sub x0, x0, x1, asr #12
74c: 8b001840 add x0, x2, x0, lsl #6
750: d65f03c0 ret
754: d503201f nop
758: d4210000 brk #0x800
page_address:
6c0: 90000002 adrp x2, 0 <memstart_addr>
6c4: d2c00841 mov x1, #0x4200000000
6c8: f9400043 ldr x3, [x2]
6cc: 934cfc62 asr x2, x3, #12
6d0: 8b021800 add x0, x0, x2, lsl #6
6d4: 8b010001 add x1, x0, x1
6d8: 9346fc21 asr x1, x1, #6
6dc: d374cc21 lsl x1, x1, #12
6e0: 37000083 tbnz w3, #0, 6f0 <foo+0x30>
6e4: cb030020 sub x0, x1, x3
6e8: b25a6400 orr x0, x0, #0xffffffc000000000
6ec: d65f03c0 ret
6f0: d4210000 brk #0x800
The thing to note here is that the expression is evaluated in a way that
does not allow the compiler to eliminate the read of memstart_addr,
presumably since it is unaware that its value is aligned to PAGE_SIZE, and
that shifting it down and up again by 12 bits produces the exact same value.
So let's give the compiler a hand here. First of all, let's reimplement
virt_to_page() (patch #6) so that it explicitly translates without taking
the physical placement into account. This results in the virt_to_page()
translation to only work correctly for addresses above PAGE_OFFSET, but
this is a reasonable restriction to impose, even if it means a couple of
incorrect uses need to be fixed (patches #1 to #4). If we also, in patch #5,
move the vmemmap region right below the linear region (which guarantees that
the region is always aligned to a power-of-2 upper bound of its size, which
means we can treat VMEMMAP_START as a bitmask rather than an offset), we end
up with
virt_to_page
6d0: d34c9400 ubfx x0, x0, #12, #26
6d4: d2dff7c1 mov x1, #0xffbe00000000
6d8: f2ffffe1 movk x1, #0xffff, lsl #48
6dc: aa001820 orr x0, x1, x0, lsl #6
6e0: d65f03c0 ret
In the same way, we can get page_address to look like this
page_address:
6c0: d37a7c00 ubfiz x0, x0, #6, #32
6c4: b25a6400 orr x0, x0, #0xffffffc000000000
6c8: d65f03c0 ret
However, in this case, we need to slightly refactor the implementation of
lowmem_page_paddress(), since it performs an explicit page-to-pa-to-va
translation, rather than going through an opaque arch-defined definition
of page_to_virt. (patches #7 to #9)
[1] http://thread.gmane.org/gmane.linux.ports.arm.kernel/481327
[2] https://git.linaro.org/people/ard.biesheuvel/linux-arm.git/shortlog/refs/heads/arm64-vmemmap
(git url git://git.linaro.org/people/ard.biesheuvel/linux-arm.git arm64-vmemmap)
Ard Biesheuvel (9):
arm64: vdso: avoid virt_to_page() translations on kernel symbols
arm64: mm: free __init memory via the linear mapping
arm64: mm: avoid virt_to_page() translation for the zero page
arm64: insn: avoid virt_to_page() translations on core kernel symbols
arm64: mm: move vmemmap region right below the linear region
arm64: mm: restrict virt_to_page() to the linear mapping
nios2: use correct void* return type for page_to_virt()
openrisc: drop wrongly typed definition of page_to_virt()
mm: replace open coded page to virt conversion with page_to_virt()
arch/arm64/include/asm/memory.h | 30 ++++++++++++++++++--
arch/arm64/include/asm/pgtable.h | 13 +++------
arch/arm64/kernel/insn.c | 2 +-
arch/arm64/kernel/vdso.c | 7 +++--
arch/arm64/mm/dump.c | 16 +++++------
arch/arm64/mm/init.c | 19 +++++++++----
arch/nios2/include/asm/io.h | 1 -
arch/nios2/include/asm/page.h | 2 +-
arch/nios2/include/asm/pgtable.h | 2 +-
arch/openrisc/include/asm/page.h | 2 --
include/linux/mm.h | 6 +++-
11 files changed, 65 insertions(+), 35 deletions(-)
--
2.5.0
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH v2 1/9] arm64: vdso: avoid virt_to_page() translations on kernel symbols
2016-02-29 14:44 ` Ard Biesheuvel
@ 2016-02-29 14:44 ` Ard Biesheuvel
-1 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel, catalin.marinas, will.deacon, mark.rutland
Cc: linux-mm, akpm, nios2-dev, lftan, jonas, linux, Ard Biesheuvel
The translation performed by virt_to_page() is only valid for linear
addresses, and kernel symbols are no longer in the linear mapping.
So perform the __pa() translation explicitly, which does the right
thing in either case, and only then translate to a struct page offset.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/arm64/kernel/vdso.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c
index 97bc68f4c689..fb3c17f031aa 100644
--- a/arch/arm64/kernel/vdso.c
+++ b/arch/arm64/kernel/vdso.c
@@ -131,11 +131,12 @@ static int __init vdso_init(void)
return -ENOMEM;
/* Grab the vDSO data page. */
- vdso_pagelist[0] = virt_to_page(vdso_data);
+ vdso_pagelist[0] = pfn_to_page(__pa(vdso_data) >> PAGE_SHIFT);
/* Grab the vDSO code pages. */
- for (i = 0; i < vdso_pages; i++)
- vdso_pagelist[i + 1] = virt_to_page(&vdso_start + i * PAGE_SIZE);
+ vdso_pagelist[1] = pfn_to_page(__pa(&vdso_start) >> PAGE_SHIFT);
+ for (i = 1; i < vdso_pages; i++)
+ vdso_pagelist[i + 1] = vdso_pagelist[1] + i;
/* Populate the special mapping structures */
vdso_spec[0] = (struct vm_special_mapping) {
--
2.5.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 1/9] arm64: vdso: avoid virt_to_page() translations on kernel symbols
@ 2016-02-29 14:44 ` Ard Biesheuvel
0 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel
The translation performed by virt_to_page() is only valid for linear
addresses, and kernel symbols are no longer in the linear mapping.
So perform the __pa() translation explicitly, which does the right
thing in either case, and only then translate to a struct page offset.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/arm64/kernel/vdso.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c
index 97bc68f4c689..fb3c17f031aa 100644
--- a/arch/arm64/kernel/vdso.c
+++ b/arch/arm64/kernel/vdso.c
@@ -131,11 +131,12 @@ static int __init vdso_init(void)
return -ENOMEM;
/* Grab the vDSO data page. */
- vdso_pagelist[0] = virt_to_page(vdso_data);
+ vdso_pagelist[0] = pfn_to_page(__pa(vdso_data) >> PAGE_SHIFT);
/* Grab the vDSO code pages. */
- for (i = 0; i < vdso_pages; i++)
- vdso_pagelist[i + 1] = virt_to_page(&vdso_start + i * PAGE_SIZE);
+ vdso_pagelist[1] = pfn_to_page(__pa(&vdso_start) >> PAGE_SHIFT);
+ for (i = 1; i < vdso_pages; i++)
+ vdso_pagelist[i + 1] = vdso_pagelist[1] + i;
/* Populate the special mapping structures */
vdso_spec[0] = (struct vm_special_mapping) {
--
2.5.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 2/9] arm64: mm: free __init memory via the linear mapping
2016-02-29 14:44 ` Ard Biesheuvel
@ 2016-02-29 14:44 ` Ard Biesheuvel
-1 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel, catalin.marinas, will.deacon, mark.rutland
Cc: linux-mm, akpm, nios2-dev, lftan, jonas, linux, Ard Biesheuvel
The implementation of free_initmem_default() expects __init_begin
and __init_end to be covered by the linear mapping, which is no
longer the case. So open code it instead, using addresses that are
explicitly translated from kernel virtual to linear virtual.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/arm64/mm/init.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index a08153c99ffa..cebad6cceda8 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -416,7 +416,8 @@ void __init mem_init(void)
void free_initmem(void)
{
- free_initmem_default(0);
+ free_reserved_area(__va(__pa(__init_begin)), __va(__pa(__init_end)),
+ 0, "unused kernel");
fixup_init();
}
--
2.5.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 2/9] arm64: mm: free __init memory via the linear mapping
@ 2016-02-29 14:44 ` Ard Biesheuvel
0 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel
The implementation of free_initmem_default() expects __init_begin
and __init_end to be covered by the linear mapping, which is no
longer the case. So open code it instead, using addresses that are
explicitly translated from kernel virtual to linear virtual.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/arm64/mm/init.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index a08153c99ffa..cebad6cceda8 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -416,7 +416,8 @@ void __init mem_init(void)
void free_initmem(void)
{
- free_initmem_default(0);
+ free_reserved_area(__va(__pa(__init_begin)), __va(__pa(__init_end)),
+ 0, "unused kernel");
fixup_init();
}
--
2.5.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 3/9] arm64: mm: avoid virt_to_page() translation for the zero page
2016-02-29 14:44 ` Ard Biesheuvel
@ 2016-02-29 14:44 ` Ard Biesheuvel
-1 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel, catalin.marinas, will.deacon, mark.rutland
Cc: linux-mm, akpm, nios2-dev, lftan, jonas, linux, Ard Biesheuvel
The zero page is statically allocated, so grab its struct page pointer
without using virt_to_page(), which will be restricted to the linear
mapping later.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/arm64/include/asm/pgtable.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 7d0a9eab1133..af80e437d075 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -57,7 +57,7 @@ extern void __pgd_error(const char *file, int line, unsigned long val);
* for zero-mapped memory areas etc..
*/
extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)];
-#define ZERO_PAGE(vaddr) virt_to_page(empty_zero_page)
+#define ZERO_PAGE(vaddr) pfn_to_page(__pa(empty_zero_page) >> PAGE_SHIFT)
#define pte_ERROR(pte) __pte_error(__FILE__, __LINE__, pte_val(pte))
--
2.5.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 3/9] arm64: mm: avoid virt_to_page() translation for the zero page
@ 2016-02-29 14:44 ` Ard Biesheuvel
0 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel
The zero page is statically allocated, so grab its struct page pointer
without using virt_to_page(), which will be restricted to the linear
mapping later.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/arm64/include/asm/pgtable.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 7d0a9eab1133..af80e437d075 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -57,7 +57,7 @@ extern void __pgd_error(const char *file, int line, unsigned long val);
* for zero-mapped memory areas etc..
*/
extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)];
-#define ZERO_PAGE(vaddr) virt_to_page(empty_zero_page)
+#define ZERO_PAGE(vaddr) pfn_to_page(__pa(empty_zero_page) >> PAGE_SHIFT)
#define pte_ERROR(pte) __pte_error(__FILE__, __LINE__, pte_val(pte))
--
2.5.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 4/9] arm64: insn: avoid virt_to_page() translations on core kernel symbols
2016-02-29 14:44 ` Ard Biesheuvel
@ 2016-02-29 14:44 ` Ard Biesheuvel
-1 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel, catalin.marinas, will.deacon, mark.rutland
Cc: linux-mm, akpm, nios2-dev, lftan, jonas, linux, Ard Biesheuvel
Before restricting virt_to_page() to the linear mapping, ensure that
the text patching code does not use it to resolve references into the
core kernel text, which is mapped in the vmalloc area.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/arm64/kernel/insn.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index 7371455160e5..219d3a5df142 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -96,7 +96,7 @@ static void __kprobes *patch_map(void *addr, int fixmap)
if (module && IS_ENABLED(CONFIG_DEBUG_SET_MODULE_RONX))
page = vmalloc_to_page(addr);
else if (!module && IS_ENABLED(CONFIG_DEBUG_RODATA))
- page = virt_to_page(addr);
+ page = pfn_to_page(__pa(addr) >> PAGE_SHIFT);
else
return addr;
--
2.5.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 4/9] arm64: insn: avoid virt_to_page() translations on core kernel symbols
@ 2016-02-29 14:44 ` Ard Biesheuvel
0 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel
Before restricting virt_to_page() to the linear mapping, ensure that
the text patching code does not use it to resolve references into the
core kernel text, which is mapped in the vmalloc area.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/arm64/kernel/insn.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index 7371455160e5..219d3a5df142 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -96,7 +96,7 @@ static void __kprobes *patch_map(void *addr, int fixmap)
if (module && IS_ENABLED(CONFIG_DEBUG_SET_MODULE_RONX))
page = vmalloc_to_page(addr);
else if (!module && IS_ENABLED(CONFIG_DEBUG_RODATA))
- page = virt_to_page(addr);
+ page = pfn_to_page(__pa(addr) >> PAGE_SHIFT);
else
return addr;
--
2.5.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 5/9] arm64: mm: move vmemmap region right below the linear region
2016-02-29 14:44 ` Ard Biesheuvel
@ 2016-02-29 14:44 ` Ard Biesheuvel
-1 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel, catalin.marinas, will.deacon, mark.rutland
Cc: linux-mm, akpm, nios2-dev, lftan, jonas, linux, Ard Biesheuvel
This moves the vmemmap region right below PAGE_OFFSET, aka the start
of the linear region, and redefines its size to be a power of two.
Due to the placement of PAGE_OFFSET in the middle of the address space,
whose size is a power of two as well, this guarantees that virt to
page conversions and vice versa can be implemented efficiently, by
masking and shifting rather than ordinary arithmetic.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/arm64/include/asm/memory.h | 18 +++++++++++++++++-
arch/arm64/include/asm/pgtable.h | 11 +++--------
arch/arm64/mm/dump.c | 16 ++++++++--------
arch/arm64/mm/init.c | 16 +++++++++++-----
4 files changed, 39 insertions(+), 22 deletions(-)
diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h
index 12f8a00fb3f1..8a2ab195ca77 100644
--- a/arch/arm64/include/asm/memory.h
+++ b/arch/arm64/include/asm/memory.h
@@ -40,6 +40,21 @@
#define PCI_IO_SIZE SZ_16M
/*
+ * Log2 of the upper bound of the size of a struct page. Used for sizing
+ * the vmemmap region only, does not affect actual memory footprint.
+ * We don't use sizeof(struct page) directly since taking its size here
+ * requires its definition to be available at this point in the inclusion
+ * chain, and it may not be a power of 2 in the first place.
+ */
+#define STRUCT_PAGE_MAX_SHIFT 6
+
+/*
+ * VMEMMAP_SIZE - allows the whole linear region to be covered by
+ * a struct page array
+ */
+#define VMEMMAP_SIZE (UL(1) << (VA_BITS - PAGE_SHIFT - 1 + STRUCT_PAGE_MAX_SHIFT))
+
+/*
* PAGE_OFFSET - the virtual address of the start of the kernel image (top
* (VA_BITS - 1))
* VA_BITS - the maximum number of bits for virtual addresses.
@@ -54,7 +69,8 @@
#define MODULES_END (MODULES_VADDR + MODULES_VSIZE)
#define MODULES_VADDR (VA_START + KASAN_SHADOW_SIZE)
#define MODULES_VSIZE (SZ_128M)
-#define PCI_IO_END (PAGE_OFFSET - SZ_2M)
+#define VMEMMAP_START (PAGE_OFFSET - VMEMMAP_SIZE)
+#define PCI_IO_END (VMEMMAP_START - SZ_2M)
#define PCI_IO_START (PCI_IO_END - PCI_IO_SIZE)
#define FIXADDR_TOP (PCI_IO_START - SZ_2M)
#define TASK_SIZE_64 (UL(1) << VA_BITS)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index af80e437d075..fdea729f1fbf 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -24,20 +24,15 @@
#include <asm/pgtable-prot.h>
/*
- * VMALLOC and SPARSEMEM_VMEMMAP ranges.
+ * VMALLOC range.
*
- * VMEMAP_SIZE: allows the whole linear region to be covered by a struct page array
- * (rounded up to PUD_SIZE).
* VMALLOC_START: beginning of the kernel vmalloc space
- * VMALLOC_END: extends to the available space below vmmemmap, PCI I/O space,
- * fixed mappings and modules
+ * VMALLOC_END: extends to the available space below vmmemmap, PCI I/O space
+ * and fixed mappings
*/
-#define VMEMMAP_SIZE ALIGN((1UL << (VA_BITS - PAGE_SHIFT - 1)) * sizeof(struct page), PUD_SIZE)
-
#define VMALLOC_START (MODULES_END)
#define VMALLOC_END (PAGE_OFFSET - PUD_SIZE - VMEMMAP_SIZE - SZ_64K)
-#define VMEMMAP_START (VMALLOC_END + SZ_64K)
#define vmemmap ((struct page *)VMEMMAP_START - (memstart_addr >> PAGE_SHIFT))
#define FIRST_USER_ADDRESS 0UL
diff --git a/arch/arm64/mm/dump.c b/arch/arm64/mm/dump.c
index f9271cb2f5e3..a21f47421b0c 100644
--- a/arch/arm64/mm/dump.c
+++ b/arch/arm64/mm/dump.c
@@ -37,14 +37,14 @@ enum address_markers_idx {
MODULES_END_NR,
VMALLOC_START_NR,
VMALLOC_END_NR,
-#ifdef CONFIG_SPARSEMEM_VMEMMAP
- VMEMMAP_START_NR,
- VMEMMAP_END_NR,
-#endif
FIXADDR_START_NR,
FIXADDR_END_NR,
PCI_START_NR,
PCI_END_NR,
+#ifdef CONFIG_SPARSEMEM_VMEMMAP
+ VMEMMAP_START_NR,
+ VMEMMAP_END_NR,
+#endif
KERNEL_SPACE_NR,
};
@@ -53,14 +53,14 @@ static struct addr_marker address_markers[] = {
{ MODULES_END, "Modules end" },
{ VMALLOC_START, "vmalloc() Area" },
{ VMALLOC_END, "vmalloc() End" },
-#ifdef CONFIG_SPARSEMEM_VMEMMAP
- { 0, "vmemmap start" },
- { 0, "vmemmap end" },
-#endif
{ FIXADDR_START, "Fixmap start" },
{ FIXADDR_TOP, "Fixmap end" },
{ PCI_IO_START, "PCI I/O start" },
{ PCI_IO_END, "PCI I/O end" },
+#ifdef CONFIG_SPARSEMEM_VMEMMAP
+ { 0, "vmemmap start" },
+ { 0, "vmemmap end" },
+#endif
{ PAGE_OFFSET, "Linear Mapping" },
{ -1, NULL },
};
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index cebad6cceda8..9cfe94b41b54 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -365,13 +365,13 @@ void __init mem_init(void)
" .text : 0x%p" " - 0x%p" " (%6ld KB)\n"
" .rodata : 0x%p" " - 0x%p" " (%6ld KB)\n"
" .data : 0x%p" " - 0x%p" " (%6ld KB)\n"
+ " fixed : 0x%16lx - 0x%16lx (%6ld KB)\n"
+ " PCI I/O : 0x%16lx - 0x%16lx (%6ld MB)\n"
#ifdef CONFIG_SPARSEMEM_VMEMMAP
" vmemmap : 0x%16lx - 0x%16lx (%6ld GB maximum)\n"
" 0x%16lx - 0x%16lx (%6ld MB actual)\n"
#endif
- " fixed : 0x%16lx - 0x%16lx (%6ld KB)\n"
- " PCI I/O : 0x%16lx - 0x%16lx (%6ld MB)\n"
- " memory : 0x%16lx - 0x%16lx (%6ld MB)\n",
+ " memory : 0x%16lx - 0x%16lx (%6ld MB)\n",
#ifdef CONFIG_KASAN
MLG(KASAN_SHADOW_START, KASAN_SHADOW_END),
#endif
@@ -381,14 +381,14 @@ void __init mem_init(void)
MLK_ROUNDUP(_text, __start_rodata),
MLK_ROUNDUP(__start_rodata, _etext),
MLK_ROUNDUP(_sdata, _edata),
+ MLK(FIXADDR_START, FIXADDR_TOP),
+ MLM(PCI_IO_START, PCI_IO_END),
#ifdef CONFIG_SPARSEMEM_VMEMMAP
MLG(VMEMMAP_START,
VMEMMAP_START + VMEMMAP_SIZE),
MLM((unsigned long)phys_to_page(memblock_start_of_DRAM()),
(unsigned long)virt_to_page(high_memory)),
#endif
- MLK(FIXADDR_START, FIXADDR_TOP),
- MLM(PCI_IO_START, PCI_IO_END),
MLM(__phys_to_virt(memblock_start_of_DRAM()),
(unsigned long)high_memory));
@@ -404,6 +404,12 @@ void __init mem_init(void)
BUILD_BUG_ON(TASK_SIZE_32 > TASK_SIZE_64);
#endif
+ /*
+ * Make sure we chose the upper bound of sizeof(struct page)
+ * correctly.
+ */
+ BUILD_BUG_ON(sizeof(struct page) > (1 << STRUCT_PAGE_MAX_SHIFT));
+
if (PAGE_SIZE >= 16384 && get_num_physpages() <= 128) {
extern int sysctl_overcommit_memory;
/*
--
2.5.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 5/9] arm64: mm: move vmemmap region right below the linear region
@ 2016-02-29 14:44 ` Ard Biesheuvel
0 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel
This moves the vmemmap region right below PAGE_OFFSET, aka the start
of the linear region, and redefines its size to be a power of two.
Due to the placement of PAGE_OFFSET in the middle of the address space,
whose size is a power of two as well, this guarantees that virt to
page conversions and vice versa can be implemented efficiently, by
masking and shifting rather than ordinary arithmetic.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/arm64/include/asm/memory.h | 18 +++++++++++++++++-
arch/arm64/include/asm/pgtable.h | 11 +++--------
arch/arm64/mm/dump.c | 16 ++++++++--------
arch/arm64/mm/init.c | 16 +++++++++++-----
4 files changed, 39 insertions(+), 22 deletions(-)
diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h
index 12f8a00fb3f1..8a2ab195ca77 100644
--- a/arch/arm64/include/asm/memory.h
+++ b/arch/arm64/include/asm/memory.h
@@ -40,6 +40,21 @@
#define PCI_IO_SIZE SZ_16M
/*
+ * Log2 of the upper bound of the size of a struct page. Used for sizing
+ * the vmemmap region only, does not affect actual memory footprint.
+ * We don't use sizeof(struct page) directly since taking its size here
+ * requires its definition to be available at this point in the inclusion
+ * chain, and it may not be a power of 2 in the first place.
+ */
+#define STRUCT_PAGE_MAX_SHIFT 6
+
+/*
+ * VMEMMAP_SIZE - allows the whole linear region to be covered by
+ * a struct page array
+ */
+#define VMEMMAP_SIZE (UL(1) << (VA_BITS - PAGE_SHIFT - 1 + STRUCT_PAGE_MAX_SHIFT))
+
+/*
* PAGE_OFFSET - the virtual address of the start of the kernel image (top
* (VA_BITS - 1))
* VA_BITS - the maximum number of bits for virtual addresses.
@@ -54,7 +69,8 @@
#define MODULES_END (MODULES_VADDR + MODULES_VSIZE)
#define MODULES_VADDR (VA_START + KASAN_SHADOW_SIZE)
#define MODULES_VSIZE (SZ_128M)
-#define PCI_IO_END (PAGE_OFFSET - SZ_2M)
+#define VMEMMAP_START (PAGE_OFFSET - VMEMMAP_SIZE)
+#define PCI_IO_END (VMEMMAP_START - SZ_2M)
#define PCI_IO_START (PCI_IO_END - PCI_IO_SIZE)
#define FIXADDR_TOP (PCI_IO_START - SZ_2M)
#define TASK_SIZE_64 (UL(1) << VA_BITS)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index af80e437d075..fdea729f1fbf 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -24,20 +24,15 @@
#include <asm/pgtable-prot.h>
/*
- * VMALLOC and SPARSEMEM_VMEMMAP ranges.
+ * VMALLOC range.
*
- * VMEMAP_SIZE: allows the whole linear region to be covered by a struct page array
- * (rounded up to PUD_SIZE).
* VMALLOC_START: beginning of the kernel vmalloc space
- * VMALLOC_END: extends to the available space below vmmemmap, PCI I/O space,
- * fixed mappings and modules
+ * VMALLOC_END: extends to the available space below vmmemmap, PCI I/O space
+ * and fixed mappings
*/
-#define VMEMMAP_SIZE ALIGN((1UL << (VA_BITS - PAGE_SHIFT - 1)) * sizeof(struct page), PUD_SIZE)
-
#define VMALLOC_START (MODULES_END)
#define VMALLOC_END (PAGE_OFFSET - PUD_SIZE - VMEMMAP_SIZE - SZ_64K)
-#define VMEMMAP_START (VMALLOC_END + SZ_64K)
#define vmemmap ((struct page *)VMEMMAP_START - (memstart_addr >> PAGE_SHIFT))
#define FIRST_USER_ADDRESS 0UL
diff --git a/arch/arm64/mm/dump.c b/arch/arm64/mm/dump.c
index f9271cb2f5e3..a21f47421b0c 100644
--- a/arch/arm64/mm/dump.c
+++ b/arch/arm64/mm/dump.c
@@ -37,14 +37,14 @@ enum address_markers_idx {
MODULES_END_NR,
VMALLOC_START_NR,
VMALLOC_END_NR,
-#ifdef CONFIG_SPARSEMEM_VMEMMAP
- VMEMMAP_START_NR,
- VMEMMAP_END_NR,
-#endif
FIXADDR_START_NR,
FIXADDR_END_NR,
PCI_START_NR,
PCI_END_NR,
+#ifdef CONFIG_SPARSEMEM_VMEMMAP
+ VMEMMAP_START_NR,
+ VMEMMAP_END_NR,
+#endif
KERNEL_SPACE_NR,
};
@@ -53,14 +53,14 @@ static struct addr_marker address_markers[] = {
{ MODULES_END, "Modules end" },
{ VMALLOC_START, "vmalloc() Area" },
{ VMALLOC_END, "vmalloc() End" },
-#ifdef CONFIG_SPARSEMEM_VMEMMAP
- { 0, "vmemmap start" },
- { 0, "vmemmap end" },
-#endif
{ FIXADDR_START, "Fixmap start" },
{ FIXADDR_TOP, "Fixmap end" },
{ PCI_IO_START, "PCI I/O start" },
{ PCI_IO_END, "PCI I/O end" },
+#ifdef CONFIG_SPARSEMEM_VMEMMAP
+ { 0, "vmemmap start" },
+ { 0, "vmemmap end" },
+#endif
{ PAGE_OFFSET, "Linear Mapping" },
{ -1, NULL },
};
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index cebad6cceda8..9cfe94b41b54 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -365,13 +365,13 @@ void __init mem_init(void)
" .text : 0x%p" " - 0x%p" " (%6ld KB)\n"
" .rodata : 0x%p" " - 0x%p" " (%6ld KB)\n"
" .data : 0x%p" " - 0x%p" " (%6ld KB)\n"
+ " fixed : 0x%16lx - 0x%16lx (%6ld KB)\n"
+ " PCI I/O : 0x%16lx - 0x%16lx (%6ld MB)\n"
#ifdef CONFIG_SPARSEMEM_VMEMMAP
" vmemmap : 0x%16lx - 0x%16lx (%6ld GB maximum)\n"
" 0x%16lx - 0x%16lx (%6ld MB actual)\n"
#endif
- " fixed : 0x%16lx - 0x%16lx (%6ld KB)\n"
- " PCI I/O : 0x%16lx - 0x%16lx (%6ld MB)\n"
- " memory : 0x%16lx - 0x%16lx (%6ld MB)\n",
+ " memory : 0x%16lx - 0x%16lx (%6ld MB)\n",
#ifdef CONFIG_KASAN
MLG(KASAN_SHADOW_START, KASAN_SHADOW_END),
#endif
@@ -381,14 +381,14 @@ void __init mem_init(void)
MLK_ROUNDUP(_text, __start_rodata),
MLK_ROUNDUP(__start_rodata, _etext),
MLK_ROUNDUP(_sdata, _edata),
+ MLK(FIXADDR_START, FIXADDR_TOP),
+ MLM(PCI_IO_START, PCI_IO_END),
#ifdef CONFIG_SPARSEMEM_VMEMMAP
MLG(VMEMMAP_START,
VMEMMAP_START + VMEMMAP_SIZE),
MLM((unsigned long)phys_to_page(memblock_start_of_DRAM()),
(unsigned long)virt_to_page(high_memory)),
#endif
- MLK(FIXADDR_START, FIXADDR_TOP),
- MLM(PCI_IO_START, PCI_IO_END),
MLM(__phys_to_virt(memblock_start_of_DRAM()),
(unsigned long)high_memory));
@@ -404,6 +404,12 @@ void __init mem_init(void)
BUILD_BUG_ON(TASK_SIZE_32 > TASK_SIZE_64);
#endif
+ /*
+ * Make sure we chose the upper bound of sizeof(struct page)
+ * correctly.
+ */
+ BUILD_BUG_ON(sizeof(struct page) > (1 << STRUCT_PAGE_MAX_SHIFT));
+
if (PAGE_SIZE >= 16384 && get_num_physpages() <= 128) {
extern int sysctl_overcommit_memory;
/*
--
2.5.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 6/9] arm64: mm: restrict virt_to_page() to the linear mapping
2016-02-29 14:44 ` Ard Biesheuvel
@ 2016-02-29 14:44 ` Ard Biesheuvel
-1 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel, catalin.marinas, will.deacon, mark.rutland
Cc: linux-mm, akpm, nios2-dev, lftan, jonas, linux, Ard Biesheuvel
The mm layer makes heavy use of virt_to_page(), which translates from
virtual addresses to offsets in the struct page array using an intermediate
translation to physical addresses. However, these physical translations
are based on the actual placement of physical memory, which can only be
discovered at runtime. This means virt_to_page() translations involve a
global PHYS_OFFSET variable, and hence a memory access.
Now that the vmemmap region has been redefined to cover the linear region
rather than the entire physical address space, we no longer need to perform
a virtual-to-physical translation in the implementation of virt_to_page(),
which means we can get rid of the memory access. Since VMEMMAP_START is
guaranteed to be aligned to a power-of-two upper bound of the size of the
vmemmap region, we can also treat VMEMMAP_START as a mask rather than an
offset.
This restricts virt_to_page() translations to the linear region, so
redefine virt_addr_valid() as well.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/arm64/include/asm/memory.h | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h
index 8a2ab195ca77..f412f502ccdd 100644
--- a/arch/arm64/include/asm/memory.h
+++ b/arch/arm64/include/asm/memory.h
@@ -208,9 +208,19 @@ static inline void *phys_to_virt(phys_addr_t x)
*/
#define ARCH_PFN_OFFSET ((unsigned long)PHYS_PFN_OFFSET)
+#ifndef CONFIG_SPARSEMEM_VMEMMAP
#define virt_to_page(kaddr) pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
-#define virt_addr_valid(kaddr) pfn_valid(__pa(kaddr) >> PAGE_SHIFT)
+#define virt_addr_valid(kaddr) pfn_valid(__pa(kaddr) >> PAGE_SHIFT)
+#else
+#define __virt_to_pgoff(kaddr) (((u64)(kaddr) & ~PAGE_OFFSET) / PAGE_SIZE * sizeof(struct page))
+#define __page_to_voff(kaddr) (((u64)(page) & ~VMEMMAP_START) * PAGE_SIZE / sizeof(struct page))
+
+#define page_to_virt(page) ((void *)((__page_to_voff(page)) | PAGE_OFFSET))
+#define virt_to_page(vaddr) ((struct page *)((__virt_to_pgoff(vaddr)) | VMEMMAP_START))
+#define virt_addr_valid(kaddr) pfn_valid((((u64)(kaddr) & ~PAGE_OFFSET) \
+ + PHYS_OFFSET) >> PAGE_SHIFT)
+#endif
#endif
#include <asm-generic/memory_model.h>
--
2.5.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 6/9] arm64: mm: restrict virt_to_page() to the linear mapping
@ 2016-02-29 14:44 ` Ard Biesheuvel
0 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel
The mm layer makes heavy use of virt_to_page(), which translates from
virtual addresses to offsets in the struct page array using an intermediate
translation to physical addresses. However, these physical translations
are based on the actual placement of physical memory, which can only be
discovered at runtime. This means virt_to_page() translations involve a
global PHYS_OFFSET variable, and hence a memory access.
Now that the vmemmap region has been redefined to cover the linear region
rather than the entire physical address space, we no longer need to perform
a virtual-to-physical translation in the implementation of virt_to_page(),
which means we can get rid of the memory access. Since VMEMMAP_START is
guaranteed to be aligned to a power-of-two upper bound of the size of the
vmemmap region, we can also treat VMEMMAP_START as a mask rather than an
offset.
This restricts virt_to_page() translations to the linear region, so
redefine virt_addr_valid() as well.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/arm64/include/asm/memory.h | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h
index 8a2ab195ca77..f412f502ccdd 100644
--- a/arch/arm64/include/asm/memory.h
+++ b/arch/arm64/include/asm/memory.h
@@ -208,9 +208,19 @@ static inline void *phys_to_virt(phys_addr_t x)
*/
#define ARCH_PFN_OFFSET ((unsigned long)PHYS_PFN_OFFSET)
+#ifndef CONFIG_SPARSEMEM_VMEMMAP
#define virt_to_page(kaddr) pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
-#define virt_addr_valid(kaddr) pfn_valid(__pa(kaddr) >> PAGE_SHIFT)
+#define virt_addr_valid(kaddr) pfn_valid(__pa(kaddr) >> PAGE_SHIFT)
+#else
+#define __virt_to_pgoff(kaddr) (((u64)(kaddr) & ~PAGE_OFFSET) / PAGE_SIZE * sizeof(struct page))
+#define __page_to_voff(kaddr) (((u64)(page) & ~VMEMMAP_START) * PAGE_SIZE / sizeof(struct page))
+
+#define page_to_virt(page) ((void *)((__page_to_voff(page)) | PAGE_OFFSET))
+#define virt_to_page(vaddr) ((struct page *)((__virt_to_pgoff(vaddr)) | VMEMMAP_START))
+#define virt_addr_valid(kaddr) pfn_valid((((u64)(kaddr) & ~PAGE_OFFSET) \
+ + PHYS_OFFSET) >> PAGE_SHIFT)
+#endif
#endif
#include <asm-generic/memory_model.h>
--
2.5.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 7/9] nios2: use correct void* return type for page_to_virt()
2016-02-29 14:44 ` Ard Biesheuvel
@ 2016-02-29 14:44 ` Ard Biesheuvel
-1 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel, catalin.marinas, will.deacon, mark.rutland
Cc: linux-mm, akpm, nios2-dev, lftan, jonas, linux, Ard Biesheuvel
To align with other architectures, the expression procuded by expanding
the macro page_to_virt() should be of type void*, since it returns a
virtual address. Fix that, and also fix up an instance where page_to_virt
was expected to return 'unsigned long', and drop another instance that was
entirely unused (page_to_bus)
Cc: linux-mm@kvack.org
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Ley Foon Tan <lftan@altera.com>
Cc: nios2-dev@lists.rocketboards.org
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/nios2/include/asm/io.h | 1 -
arch/nios2/include/asm/page.h | 2 +-
arch/nios2/include/asm/pgtable.h | 2 +-
3 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/arch/nios2/include/asm/io.h b/arch/nios2/include/asm/io.h
index c5a62da22cd2..ce072ba0f8dd 100644
--- a/arch/nios2/include/asm/io.h
+++ b/arch/nios2/include/asm/io.h
@@ -50,7 +50,6 @@ static inline void iounmap(void __iomem *addr)
/* Pages to physical address... */
#define page_to_phys(page) virt_to_phys(page_to_virt(page))
-#define page_to_bus(page) page_to_virt(page)
/* Macros used for converting between virtual and physical mappings. */
#define phys_to_virt(vaddr) \
diff --git a/arch/nios2/include/asm/page.h b/arch/nios2/include/asm/page.h
index 4b32d6fd9d98..c1683f51ad0f 100644
--- a/arch/nios2/include/asm/page.h
+++ b/arch/nios2/include/asm/page.h
@@ -84,7 +84,7 @@ extern struct page *mem_map;
((void *)((unsigned long)(x) + PAGE_OFFSET - PHYS_OFFSET))
#define page_to_virt(page) \
- ((((page) - mem_map) << PAGE_SHIFT) + PAGE_OFFSET)
+ ((void *)(((page) - mem_map) << PAGE_SHIFT) + PAGE_OFFSET)
# define pfn_to_kaddr(pfn) __va((pfn) << PAGE_SHIFT)
# define pfn_valid(pfn) ((pfn) >= ARCH_PFN_OFFSET && \
diff --git a/arch/nios2/include/asm/pgtable.h b/arch/nios2/include/asm/pgtable.h
index a213e8c9aad0..298393c3cb42 100644
--- a/arch/nios2/include/asm/pgtable.h
+++ b/arch/nios2/include/asm/pgtable.h
@@ -209,7 +209,7 @@ static inline void set_pte(pte_t *ptep, pte_t pteval)
static inline void set_pte_at(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pteval)
{
- unsigned long paddr = page_to_virt(pte_page(pteval));
+ unsigned long paddr = (unsigned long)page_to_virt(pte_page(pteval));
flush_dcache_range(paddr, paddr + PAGE_SIZE);
set_pte(ptep, pteval);
--
2.5.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 7/9] nios2: use correct void* return type for page_to_virt()
@ 2016-02-29 14:44 ` Ard Biesheuvel
0 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel
To align with other architectures, the expression procuded by expanding
the macro page_to_virt() should be of type void*, since it returns a
virtual address. Fix that, and also fix up an instance where page_to_virt
was expected to return 'unsigned long', and drop another instance that was
entirely unused (page_to_bus)
Cc: linux-mm at kvack.org
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Ley Foon Tan <lftan@altera.com>
Cc: nios2-dev at lists.rocketboards.org
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/nios2/include/asm/io.h | 1 -
arch/nios2/include/asm/page.h | 2 +-
arch/nios2/include/asm/pgtable.h | 2 +-
3 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/arch/nios2/include/asm/io.h b/arch/nios2/include/asm/io.h
index c5a62da22cd2..ce072ba0f8dd 100644
--- a/arch/nios2/include/asm/io.h
+++ b/arch/nios2/include/asm/io.h
@@ -50,7 +50,6 @@ static inline void iounmap(void __iomem *addr)
/* Pages to physical address... */
#define page_to_phys(page) virt_to_phys(page_to_virt(page))
-#define page_to_bus(page) page_to_virt(page)
/* Macros used for converting between virtual and physical mappings. */
#define phys_to_virt(vaddr) \
diff --git a/arch/nios2/include/asm/page.h b/arch/nios2/include/asm/page.h
index 4b32d6fd9d98..c1683f51ad0f 100644
--- a/arch/nios2/include/asm/page.h
+++ b/arch/nios2/include/asm/page.h
@@ -84,7 +84,7 @@ extern struct page *mem_map;
((void *)((unsigned long)(x) + PAGE_OFFSET - PHYS_OFFSET))
#define page_to_virt(page) \
- ((((page) - mem_map) << PAGE_SHIFT) + PAGE_OFFSET)
+ ((void *)(((page) - mem_map) << PAGE_SHIFT) + PAGE_OFFSET)
# define pfn_to_kaddr(pfn) __va((pfn) << PAGE_SHIFT)
# define pfn_valid(pfn) ((pfn) >= ARCH_PFN_OFFSET && \
diff --git a/arch/nios2/include/asm/pgtable.h b/arch/nios2/include/asm/pgtable.h
index a213e8c9aad0..298393c3cb42 100644
--- a/arch/nios2/include/asm/pgtable.h
+++ b/arch/nios2/include/asm/pgtable.h
@@ -209,7 +209,7 @@ static inline void set_pte(pte_t *ptep, pte_t pteval)
static inline void set_pte_at(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pteval)
{
- unsigned long paddr = page_to_virt(pte_page(pteval));
+ unsigned long paddr = (unsigned long)page_to_virt(pte_page(pteval));
flush_dcache_range(paddr, paddr + PAGE_SIZE);
set_pte(ptep, pteval);
--
2.5.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 8/9] openrisc: drop wrongly typed definition of page_to_virt()
2016-02-29 14:44 ` Ard Biesheuvel
@ 2016-02-29 14:44 ` Ard Biesheuvel
-1 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel, catalin.marinas, will.deacon, mark.rutland
Cc: linux-mm, akpm, nios2-dev, lftan, jonas, linux, Ard Biesheuvel
To align with generic code and other architectures that expect the macro
page_to_virt to produce an expression whose type is 'void*', drop the
arch specific definition, which is never referenced anyway.
Cc: linux-mm@kvack.org
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Jonas Bonn <jonas@southpole.se>
Cc: linux@lists.openrisc.net
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/openrisc/include/asm/page.h | 2 --
1 file changed, 2 deletions(-)
diff --git a/arch/openrisc/include/asm/page.h b/arch/openrisc/include/asm/page.h
index 108906f991d6..1976f7272b1f 100644
--- a/arch/openrisc/include/asm/page.h
+++ b/arch/openrisc/include/asm/page.h
@@ -84,8 +84,6 @@ typedef struct page *pgtable_t;
#define virt_to_page(addr) \
(mem_map + (((unsigned long)(addr)-PAGE_OFFSET) >> PAGE_SHIFT))
-#define page_to_virt(page) \
- ((((page) - mem_map) << PAGE_SHIFT) + PAGE_OFFSET)
#define page_to_phys(page) ((dma_addr_t)page_to_pfn(page) << PAGE_SHIFT)
--
2.5.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 8/9] openrisc: drop wrongly typed definition of page_to_virt()
@ 2016-02-29 14:44 ` Ard Biesheuvel
0 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel
To align with generic code and other architectures that expect the macro
page_to_virt to produce an expression whose type is 'void*', drop the
arch specific definition, which is never referenced anyway.
Cc: linux-mm at kvack.org
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Jonas Bonn <jonas@southpole.se>
Cc: linux at lists.openrisc.net
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
arch/openrisc/include/asm/page.h | 2 --
1 file changed, 2 deletions(-)
diff --git a/arch/openrisc/include/asm/page.h b/arch/openrisc/include/asm/page.h
index 108906f991d6..1976f7272b1f 100644
--- a/arch/openrisc/include/asm/page.h
+++ b/arch/openrisc/include/asm/page.h
@@ -84,8 +84,6 @@ typedef struct page *pgtable_t;
#define virt_to_page(addr) \
(mem_map + (((unsigned long)(addr)-PAGE_OFFSET) >> PAGE_SHIFT))
-#define page_to_virt(page) \
- ((((page) - mem_map) << PAGE_SHIFT) + PAGE_OFFSET)
#define page_to_phys(page) ((dma_addr_t)page_to_pfn(page) << PAGE_SHIFT)
--
2.5.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 9/9] mm: replace open coded page to virt conversion with page_to_virt()
2016-02-29 14:44 ` Ard Biesheuvel
@ 2016-02-29 14:44 ` Ard Biesheuvel
-1 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel, catalin.marinas, will.deacon, mark.rutland
Cc: linux-mm, akpm, nios2-dev, lftan, jonas, linux, Ard Biesheuvel
The open coded conversion from struct page address to virtual address in
lowmem_page_address() involves an intermediate conversion step to pfn
number/physical address. Since the placement of the struct page array
relative to the linear mapping may be completely independent from the
placement of physical RAM (as is that case for arm64 after commit
dfd55ad85e 'arm64: vmemmap: use virtual projection of linear region'),
the conversion to physical address and back again should factor out of
the equation, but unfortunately, the shifting and pointer arithmetic
involved prevent this from happening, and the resulting calculation
essentially subtracts the address of the start of physical memory and
adds it back again, in a way that prevents the compiler from optimizing
it away.
Since the start of physical memory is not a build time constant on arm64,
the resulting conversion involves an unnecessary memory access, which
we would like to get rid of. So replace the open coded conversion with
a call to page_to_virt(), and use the open coded conversion as its
default definition, to be overriden by the architecture, if desired.
The existing arch specific definitions of page_to_virt are all equivalent
to this default definition, so by itself this patch is a no-op.
Cc: linux-mm@kvack.org
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
include/linux/mm.h | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 516e14944339..82cb74a3b158 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -71,6 +71,10 @@ extern int mmap_rnd_compat_bits __read_mostly;
#define __pa_symbol(x) __pa(RELOC_HIDE((unsigned long)(x), 0))
#endif
+#ifndef page_to_virt
+#define page_to_virt(x) __va(PFN_PHYS(page_to_pfn(x)))
+#endif
+
/*
* To prevent common memory management code establishing
* a zero page mapping on a read fault.
@@ -927,7 +931,7 @@ static inline void set_page_memcg(struct page *page, struct mem_cgroup *memcg)
static __always_inline void *lowmem_page_address(const struct page *page)
{
- return __va(PFN_PHYS(page_to_pfn(page)));
+ return page_to_virt(page);
}
#if defined(CONFIG_HIGHMEM) && !defined(WANT_PAGE_VIRTUAL)
--
2.5.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH v2 9/9] mm: replace open coded page to virt conversion with page_to_virt()
@ 2016-02-29 14:44 ` Ard Biesheuvel
0 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-02-29 14:44 UTC (permalink / raw)
To: linux-arm-kernel
The open coded conversion from struct page address to virtual address in
lowmem_page_address() involves an intermediate conversion step to pfn
number/physical address. Since the placement of the struct page array
relative to the linear mapping may be completely independent from the
placement of physical RAM (as is that case for arm64 after commit
dfd55ad85e 'arm64: vmemmap: use virtual projection of linear region'),
the conversion to physical address and back again should factor out of
the equation, but unfortunately, the shifting and pointer arithmetic
involved prevent this from happening, and the resulting calculation
essentially subtracts the address of the start of physical memory and
adds it back again, in a way that prevents the compiler from optimizing
it away.
Since the start of physical memory is not a build time constant on arm64,
the resulting conversion involves an unnecessary memory access, which
we would like to get rid of. So replace the open coded conversion with
a call to page_to_virt(), and use the open coded conversion as its
default definition, to be overriden by the architecture, if desired.
The existing arch specific definitions of page_to_virt are all equivalent
to this default definition, so by itself this patch is a no-op.
Cc: linux-mm at kvack.org
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
include/linux/mm.h | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 516e14944339..82cb74a3b158 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -71,6 +71,10 @@ extern int mmap_rnd_compat_bits __read_mostly;
#define __pa_symbol(x) __pa(RELOC_HIDE((unsigned long)(x), 0))
#endif
+#ifndef page_to_virt
+#define page_to_virt(x) __va(PFN_PHYS(page_to_pfn(x)))
+#endif
+
/*
* To prevent common memory management code establishing
* a zero page mapping on a read fault.
@@ -927,7 +931,7 @@ static inline void set_page_memcg(struct page *page, struct mem_cgroup *memcg)
static __always_inline void *lowmem_page_address(const struct page *page)
{
- return __va(PFN_PHYS(page_to_pfn(page)));
+ return page_to_virt(page);
}
#if defined(CONFIG_HIGHMEM) && !defined(WANT_PAGE_VIRTUAL)
--
2.5.0
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [PATCH v2 5/9] arm64: mm: move vmemmap region right below the linear region
2016-02-29 14:44 ` Ard Biesheuvel
@ 2016-03-01 15:39 ` Catalin Marinas
-1 siblings, 0 replies; 26+ messages in thread
From: Catalin Marinas @ 2016-03-01 15:39 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: linux-arm-kernel, will.deacon, mark.rutland, jonas, linux-mm,
nios2-dev, linux, lftan, akpm
On Mon, Feb 29, 2016 at 03:44:40PM +0100, Ard Biesheuvel wrote:
> @@ -404,6 +404,12 @@ void __init mem_init(void)
> BUILD_BUG_ON(TASK_SIZE_32 > TASK_SIZE_64);
> #endif
>
> + /*
> + * Make sure we chose the upper bound of sizeof(struct page)
> + * correctly.
> + */
> + BUILD_BUG_ON(sizeof(struct page) > (1 << STRUCT_PAGE_MAX_SHIFT));
Since with the vmemmap fix you already assume that PAGE_OFFSET is half
of the VA space, we should add another check on PAGE_OFFSET !=
UL(0xffffffffffffffff) << (VA_BITS - 1), just in case someone thinks
they could map a bit of extra RAM without going for a larger VA.
--
Catalin
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH v2 5/9] arm64: mm: move vmemmap region right below the linear region
@ 2016-03-01 15:39 ` Catalin Marinas
0 siblings, 0 replies; 26+ messages in thread
From: Catalin Marinas @ 2016-03-01 15:39 UTC (permalink / raw)
To: linux-arm-kernel
On Mon, Feb 29, 2016 at 03:44:40PM +0100, Ard Biesheuvel wrote:
> @@ -404,6 +404,12 @@ void __init mem_init(void)
> BUILD_BUG_ON(TASK_SIZE_32 > TASK_SIZE_64);
> #endif
>
> + /*
> + * Make sure we chose the upper bound of sizeof(struct page)
> + * correctly.
> + */
> + BUILD_BUG_ON(sizeof(struct page) > (1 << STRUCT_PAGE_MAX_SHIFT));
Since with the vmemmap fix you already assume that PAGE_OFFSET is half
of the VA space, we should add another check on PAGE_OFFSET !=
UL(0xffffffffffffffff) << (VA_BITS - 1), just in case someone thinks
they could map a bit of extra RAM without going for a larger VA.
--
Catalin
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH v2 5/9] arm64: mm: move vmemmap region right below the linear region
2016-03-01 15:39 ` Catalin Marinas
@ 2016-03-01 15:43 ` Ard Biesheuvel
-1 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-03-01 15:43 UTC (permalink / raw)
To: Catalin Marinas
Cc: linux-arm-kernel, Will Deacon, Mark Rutland, Jonas Bonn,
linux-mm, nios2-dev, linux, lftan, Andrew Morton
On 1 March 2016 at 16:39, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Mon, Feb 29, 2016 at 03:44:40PM +0100, Ard Biesheuvel wrote:
>> @@ -404,6 +404,12 @@ void __init mem_init(void)
>> BUILD_BUG_ON(TASK_SIZE_32 > TASK_SIZE_64);
>> #endif
>>
>> + /*
>> + * Make sure we chose the upper bound of sizeof(struct page)
>> + * correctly.
>> + */
>> + BUILD_BUG_ON(sizeof(struct page) > (1 << STRUCT_PAGE_MAX_SHIFT));
>
> Since with the vmemmap fix you already assume that PAGE_OFFSET is half
> of the VA space, we should add another check on PAGE_OFFSET !=
> UL(0xffffffffffffffff) << (VA_BITS - 1), just in case someone thinks
> they could map a bit of extra RAM without going for a larger VA.
>
Indeed. The __pa() check only checks a single bit, so it must be split
exactly in half, unless we want to revisit that in the future (if
__pa() is no longer on a hot path after changes like these).
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH v2 5/9] arm64: mm: move vmemmap region right below the linear region
@ 2016-03-01 15:43 ` Ard Biesheuvel
0 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-03-01 15:43 UTC (permalink / raw)
To: linux-arm-kernel
On 1 March 2016 at 16:39, Catalin Marinas <catalin.marinas@arm.com> wrote:
> On Mon, Feb 29, 2016 at 03:44:40PM +0100, Ard Biesheuvel wrote:
>> @@ -404,6 +404,12 @@ void __init mem_init(void)
>> BUILD_BUG_ON(TASK_SIZE_32 > TASK_SIZE_64);
>> #endif
>>
>> + /*
>> + * Make sure we chose the upper bound of sizeof(struct page)
>> + * correctly.
>> + */
>> + BUILD_BUG_ON(sizeof(struct page) > (1 << STRUCT_PAGE_MAX_SHIFT));
>
> Since with the vmemmap fix you already assume that PAGE_OFFSET is half
> of the VA space, we should add another check on PAGE_OFFSET !=
> UL(0xffffffffffffffff) << (VA_BITS - 1), just in case someone thinks
> they could map a bit of extra RAM without going for a larger VA.
>
Indeed. The __pa() check only checks a single bit, so it must be split
exactly in half, unless we want to revisit that in the future (if
__pa() is no longer on a hot path after changes like these).
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH v2 0/9] arm64: optimize virt_to_page and page_address
@ 2016-03-30 14:45 ` Ard Biesheuvel
0 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-03-30 14:45 UTC (permalink / raw)
To: linux-arm-kernel, catalin.marinas, will.deacon, linux-mm, akpm,
nios2-dev, lftan, jonas, linux
Cc: mark.rutland, steve.capper, Ard Biesheuvel
Apologies for the wall of text. This is a followup to 'restrict virt_to_page
to linear region (instead of __pa)' [1], posted on the 24th of February.
This v2 series applies onto v4.6-rc1 with the series 'arm64: memstart_addr
alignment and vmemmap offset fixes' [2] applied on top.
Only minor changes since v1, primarily replacing 'x >> PAGE_SHIFT' instances
with PHYS_PFN, and rebasing onto the latest upstream.
When building the arm64 defconfig kernel, which has CONFIG_SPARSEMEM_VMEMMAP
enabled, the implementations of virt_to_page and its converse
[lowmem_]page_address resolve to
virt_to_page
6f8: b6300180 tbz x0, #38, 728 <bar+0x30>
6fc: 90000001 adrp x1, 0 <memstart_addr>
700: 92409400 and x0, x0, #0x3fffffffff
704: f9400021 ldr x1, [x1]
708: 8b000020 add x0, x1, x0
70c: 37000261 tbnz w1, #0, 758 <bar+0x60>
710: d34cfc00 lsr x0, x0, #12
714: d2dff7c2 mov x2, #0xffbe00000000
718: cb813000 sub x0, x0, x1, asr #12
71c: f2ffffe2 movk x2, #0xffff, lsl #48
720: 8b001840 add x0, x2, x0, lsl #6
724: d65f03c0 ret
728: 90000002 adrp x2, 0 <init_pgd>
72c: 90000001 adrp x1, 0 <memstart_addr>
730: f9400042 ldr x2, [x2]
734: f9400021 ldr x1, [x1]
738: cb020000 sub x0, x0, x2
73c: d2dff7c2 mov x2, #0xffbe00000000
740: d34cfc00 lsr x0, x0, #12
744: f2ffffe2 movk x2, #0xffff, lsl #48
748: cb813000 sub x0, x0, x1, asr #12
74c: 8b001840 add x0, x2, x0, lsl #6
750: d65f03c0 ret
754: d503201f nop
758: d4210000 brk #0x800
page_address:
6c0: 90000002 adrp x2, 0 <memstart_addr>
6c4: d2c00841 mov x1, #0x4200000000
6c8: f9400043 ldr x3, [x2]
6cc: 934cfc62 asr x2, x3, #12
6d0: 8b021800 add x0, x0, x2, lsl #6
6d4: 8b010001 add x1, x0, x1
6d8: 9346fc21 asr x1, x1, #6
6dc: d374cc21 lsl x1, x1, #12
6e0: 37000083 tbnz w3, #0, 6f0 <foo+0x30>
6e4: cb030020 sub x0, x1, x3
6e8: b25a6400 orr x0, x0, #0xffffffc000000000
6ec: d65f03c0 ret
6f0: d4210000 brk #0x800
Disappointingly, even though this translation is independent of the physical
start of RAM since commit dfd55ad85e ("arm64: vmemmap: use virtual projection
of linear region"), the expression is evaluated in a way that does not allow
the compiler to eliminate the read of memstart_addr, presumably since it is
unaware that its value is aligned to PAGE_SIZE, and that shifting it down and
up again by PAGE_SHIFT bits produces the exact same value.
So let's give the compiler a hand here. First of all, let's reimplement
virt_to_page() (patch #6) so that it explicitly translates without taking
the physical placement into account. This results in the virt_to_page()
translation to only work correctly for addresses above PAGE_OFFSET, but
this is a reasonable restriction to impose, even if it means a couple of
incorrect uses need to be fixed (patches #1 to #4). If we also, in patch #5,
move the vmemmap region right below the linear region (which guarantees that
the region is always aligned to a power-of-2 upper bound of its size, which
means we can treat VMEMMAP_START as a bitmask rather than an offset), we end
up with
virt_to_page
6d0: d34c9400 ubfx x0, x0, #12, #26
6d4: d2dff7c1 mov x1, #0xffbe00000000
6d8: f2ffffe1 movk x1, #0xffff, lsl #48
6dc: aa001820 orr x0, x1, x0, lsl #6
6e0: d65f03c0 ret
In the same way, we can get page_address to look like this
page_address:
6c0: d37a7c00 ubfiz x0, x0, #6, #32
6c4: b25a6400 orr x0, x0, #0xffffffc000000000
6c8: d65f03c0 ret
However, in this case, we need to slightly refactor the implementation of
lowmem_page_paddress(), since it performs an explicit page-to-pa-to-va
translation, rather than going through an opaque arch-defined definition
of page_to_virt. (patches #7 to #9)
[1] http://thread.gmane.org/gmane.linux.ports.arm.kernel/481327
[2] http://thread.gmane.org/gmane.linux.ports.arm.kernel/488876
Ard Biesheuvel (9):
arm64: vdso: avoid virt_to_page() translations on kernel symbols
arm64: mm: free __init memory via the linear mapping
arm64: mm: avoid virt_to_page() translation for the zero page
arm64: insn: avoid virt_to_page() translations on core kernel symbols
arm64: mm: move vmemmap region right below the linear region
arm64: mm: restrict virt_to_page() to the linear mapping
nios2: use correct void* return type for page_to_virt()
openrisc: drop wrongly typed definition of page_to_virt()
mm: replace open coded page to virt conversion with page_to_virt()
arch/arm64/include/asm/memory.h | 30 ++++++++++++++++++--
arch/arm64/include/asm/pgtable.h | 13 +++------
arch/arm64/kernel/insn.c | 2 +-
arch/arm64/kernel/vdso.c | 4 +--
arch/arm64/mm/dump.c | 16 +++++------
arch/arm64/mm/init.c | 17 +++++++----
arch/nios2/include/asm/io.h | 1 -
arch/nios2/include/asm/page.h | 2 +-
arch/nios2/include/asm/pgtable.h | 2 +-
arch/openrisc/include/asm/page.h | 2 --
include/linux/mm.h | 6 +++-
11 files changed, 62 insertions(+), 33 deletions(-)
--
2.5.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH v2 0/9] arm64: optimize virt_to_page and page_address
@ 2016-03-30 14:45 ` Ard Biesheuvel
0 siblings, 0 replies; 26+ messages in thread
From: Ard Biesheuvel @ 2016-03-30 14:45 UTC (permalink / raw)
To: linux-arm-kernel
Apologies for the wall of text. This is a followup to 'restrict virt_to_page
to linear region (instead of __pa)' [1], posted on the 24th of February.
This v2 series applies onto v4.6-rc1 with the series 'arm64: memstart_addr
alignment and vmemmap offset fixes' [2] applied on top.
Only minor changes since v1, primarily replacing 'x >> PAGE_SHIFT' instances
with PHYS_PFN, and rebasing onto the latest upstream.
When building the arm64 defconfig kernel, which has CONFIG_SPARSEMEM_VMEMMAP
enabled, the implementations of virt_to_page and its converse
[lowmem_]page_address resolve to
virt_to_page
6f8: b6300180 tbz x0, #38, 728 <bar+0x30>
6fc: 90000001 adrp x1, 0 <memstart_addr>
700: 92409400 and x0, x0, #0x3fffffffff
704: f9400021 ldr x1, [x1]
708: 8b000020 add x0, x1, x0
70c: 37000261 tbnz w1, #0, 758 <bar+0x60>
710: d34cfc00 lsr x0, x0, #12
714: d2dff7c2 mov x2, #0xffbe00000000
718: cb813000 sub x0, x0, x1, asr #12
71c: f2ffffe2 movk x2, #0xffff, lsl #48
720: 8b001840 add x0, x2, x0, lsl #6
724: d65f03c0 ret
728: 90000002 adrp x2, 0 <init_pgd>
72c: 90000001 adrp x1, 0 <memstart_addr>
730: f9400042 ldr x2, [x2]
734: f9400021 ldr x1, [x1]
738: cb020000 sub x0, x0, x2
73c: d2dff7c2 mov x2, #0xffbe00000000
740: d34cfc00 lsr x0, x0, #12
744: f2ffffe2 movk x2, #0xffff, lsl #48
748: cb813000 sub x0, x0, x1, asr #12
74c: 8b001840 add x0, x2, x0, lsl #6
750: d65f03c0 ret
754: d503201f nop
758: d4210000 brk #0x800
page_address:
6c0: 90000002 adrp x2, 0 <memstart_addr>
6c4: d2c00841 mov x1, #0x4200000000
6c8: f9400043 ldr x3, [x2]
6cc: 934cfc62 asr x2, x3, #12
6d0: 8b021800 add x0, x0, x2, lsl #6
6d4: 8b010001 add x1, x0, x1
6d8: 9346fc21 asr x1, x1, #6
6dc: d374cc21 lsl x1, x1, #12
6e0: 37000083 tbnz w3, #0, 6f0 <foo+0x30>
6e4: cb030020 sub x0, x1, x3
6e8: b25a6400 orr x0, x0, #0xffffffc000000000
6ec: d65f03c0 ret
6f0: d4210000 brk #0x800
Disappointingly, even though this translation is independent of the physical
start of RAM since commit dfd55ad85e ("arm64: vmemmap: use virtual projection
of linear region"), the expression is evaluated in a way that does not allow
the compiler to eliminate the read of memstart_addr, presumably since it is
unaware that its value is aligned to PAGE_SIZE, and that shifting it down and
up again by PAGE_SHIFT bits produces the exact same value.
So let's give the compiler a hand here. First of all, let's reimplement
virt_to_page() (patch #6) so that it explicitly translates without taking
the physical placement into account. This results in the virt_to_page()
translation to only work correctly for addresses above PAGE_OFFSET, but
this is a reasonable restriction to impose, even if it means a couple of
incorrect uses need to be fixed (patches #1 to #4). If we also, in patch #5,
move the vmemmap region right below the linear region (which guarantees that
the region is always aligned to a power-of-2 upper bound of its size, which
means we can treat VMEMMAP_START as a bitmask rather than an offset), we end
up with
virt_to_page
6d0: d34c9400 ubfx x0, x0, #12, #26
6d4: d2dff7c1 mov x1, #0xffbe00000000
6d8: f2ffffe1 movk x1, #0xffff, lsl #48
6dc: aa001820 orr x0, x1, x0, lsl #6
6e0: d65f03c0 ret
In the same way, we can get page_address to look like this
page_address:
6c0: d37a7c00 ubfiz x0, x0, #6, #32
6c4: b25a6400 orr x0, x0, #0xffffffc000000000
6c8: d65f03c0 ret
However, in this case, we need to slightly refactor the implementation of
lowmem_page_paddress(), since it performs an explicit page-to-pa-to-va
translation, rather than going through an opaque arch-defined definition
of page_to_virt. (patches #7 to #9)
[1] http://thread.gmane.org/gmane.linux.ports.arm.kernel/481327
[2] http://thread.gmane.org/gmane.linux.ports.arm.kernel/488876
Ard Biesheuvel (9):
arm64: vdso: avoid virt_to_page() translations on kernel symbols
arm64: mm: free __init memory via the linear mapping
arm64: mm: avoid virt_to_page() translation for the zero page
arm64: insn: avoid virt_to_page() translations on core kernel symbols
arm64: mm: move vmemmap region right below the linear region
arm64: mm: restrict virt_to_page() to the linear mapping
nios2: use correct void* return type for page_to_virt()
openrisc: drop wrongly typed definition of page_to_virt()
mm: replace open coded page to virt conversion with page_to_virt()
arch/arm64/include/asm/memory.h | 30 ++++++++++++++++++--
arch/arm64/include/asm/pgtable.h | 13 +++------
arch/arm64/kernel/insn.c | 2 +-
arch/arm64/kernel/vdso.c | 4 +--
arch/arm64/mm/dump.c | 16 +++++------
arch/arm64/mm/init.c | 17 +++++++----
arch/nios2/include/asm/io.h | 1 -
arch/nios2/include/asm/page.h | 2 +-
arch/nios2/include/asm/pgtable.h | 2 +-
arch/openrisc/include/asm/page.h | 2 --
include/linux/mm.h | 6 +++-
11 files changed, 62 insertions(+), 33 deletions(-)
--
2.5.0
^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2016-03-30 14:46 UTC | newest]
Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-02-29 14:44 [PATCH v2 0/9] arm64: optimize virt_to_page and page_address Ard Biesheuvel
2016-02-29 14:44 ` Ard Biesheuvel
2016-02-29 14:44 ` [PATCH v2 1/9] arm64: vdso: avoid virt_to_page() translations on kernel symbols Ard Biesheuvel
2016-02-29 14:44 ` Ard Biesheuvel
2016-02-29 14:44 ` [PATCH v2 2/9] arm64: mm: free __init memory via the linear mapping Ard Biesheuvel
2016-02-29 14:44 ` Ard Biesheuvel
2016-02-29 14:44 ` [PATCH v2 3/9] arm64: mm: avoid virt_to_page() translation for the zero page Ard Biesheuvel
2016-02-29 14:44 ` Ard Biesheuvel
2016-02-29 14:44 ` [PATCH v2 4/9] arm64: insn: avoid virt_to_page() translations on core kernel symbols Ard Biesheuvel
2016-02-29 14:44 ` Ard Biesheuvel
2016-02-29 14:44 ` [PATCH v2 5/9] arm64: mm: move vmemmap region right below the linear region Ard Biesheuvel
2016-02-29 14:44 ` Ard Biesheuvel
2016-03-01 15:39 ` Catalin Marinas
2016-03-01 15:39 ` Catalin Marinas
2016-03-01 15:43 ` Ard Biesheuvel
2016-03-01 15:43 ` Ard Biesheuvel
2016-02-29 14:44 ` [PATCH v2 6/9] arm64: mm: restrict virt_to_page() to the linear mapping Ard Biesheuvel
2016-02-29 14:44 ` Ard Biesheuvel
2016-02-29 14:44 ` [PATCH v2 7/9] nios2: use correct void* return type for page_to_virt() Ard Biesheuvel
2016-02-29 14:44 ` Ard Biesheuvel
2016-02-29 14:44 ` [PATCH v2 8/9] openrisc: drop wrongly typed definition of page_to_virt() Ard Biesheuvel
2016-02-29 14:44 ` Ard Biesheuvel
2016-02-29 14:44 ` [PATCH v2 9/9] mm: replace open coded page to virt conversion with page_to_virt() Ard Biesheuvel
2016-02-29 14:44 ` Ard Biesheuvel
2016-03-30 14:45 [PATCH v2 0/9] arm64: optimize virt_to_page and page_address Ard Biesheuvel
2016-03-30 14:45 ` Ard Biesheuvel
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.