* [PATCH v10 00/18] arm64: MMU enabled kexec relocation
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Changelog:
v10:
- Addressed a lot of comments form James Morse and from Marc Zyngier
- Added review-by's
- Synchronized with mainline
v9: - 9 patches from previous series landed in upstream, so now series
is smaller
- Added two patches from James Morse to address idmap issues for machines
with high physical addresses.
- Addressed comments from Selin Dag about compiling issues. He also tested
my series and got similar performance results: ~60 ms instead of ~580 ms
with an initramfs size of ~120MB.
v8:
- Synced with mainline to keep series up-to-date
v7:
-- Addressed comments from James Morse
- arm64: hibernate: pass the allocated pgdp to ttbr0
Removed "Fixes" tag, and added Added Reviewed-by: James Morse
- arm64: hibernate: check pgd table allocation
Sent out as a standalone patch so it can be sent to stable
Series applies on mainline + this patch
- arm64: hibernate: add trans_pgd public functions
Remove second allocation of tmp_pg_dir in swsusp_arch_resume
Added Reviewed-by: James Morse <james.morse@arm.com>
- arm64: kexec: move relocation function setup and clean up
Fixed typo in commit log
Changed kern_reloc to phys_addr_t types.
Added explanation why kern_reloc is needed.
Split into four patches:
arm64: kexec: make dtb_mem always enabled
arm64: kexec: remove unnecessary debug prints
arm64: kexec: call kexec_image_info only once
arm64: kexec: move relocation function setup
- arm64: kexec: add expandable argument to relocation function
Changed types of new arguments from unsigned long to phys_addr_t.
Changed offset prefix to KEXEC_*
Split into four patches:
arm64: kexec: cpu_soft_restart change argument types
arm64: kexec: arm64_relocate_new_kernel clean-ups
arm64: kexec: arm64_relocate_new_kernel don't use x0 as temp
arm64: kexec: add expandable argument to relocation function
- arm64: kexec: configure trans_pgd page table for kexec
Added invalid entries into EL2 vector table
Removed KEXEC_EL2_VECTOR_TABLE_SIZE and KEXEC_EL2_VECTOR_TABLE_OFFSET
Copy relocation functions and table into separate pages
Changed types in kern_reloc_arg.
Split into three patches:
arm64: kexec: offset for relocation function
arm64: kexec: kexec EL2 vectors
arm64: kexec: configure trans_pgd page table for kexec
- arm64: kexec: enable MMU during kexec relocation
Split into two patches:
arm64: kexec: enable MMU during kexec relocation
arm64: kexec: remove head from relocation argument
v6:
- Sync with mainline tip
- Added Acked's from Dave Young
v5:
- Addressed comments from Matthias Brugger: added review-by's, improved
comments, and made cleanups to swsusp_arch_resume() in addition to
create_safe_exec_page().
- Synced with mainline tip.
v4:
- Addressed comments from James Morse.
- Split "check pgd table allocation" into two patches, and moved to
the beginning of series for simpler backport of the fixes.
Added "Fixes:" tags to commit logs.
- Changed "arm64, hibernate:" to "arm64: hibernate:"
- Added Reviewed-by's
- Moved "add PUD_SECT_RDONLY" earlier in series to be with other
clean-ups
- Added "Derived from:" to arch/arm64/mm/trans_pgd.c
- Removed "flags" from trans_info
- Changed .trans_alloc_page assumption to return zeroed page.
- Simplify changes to trans_pgd_map_page(), by keeping the old
code.
- Simplify changes to trans_pgd_create_copy, by keeping the old
code.
- Removed: "add trans_pgd_create_empty"
- replace init_mm with NULL, and keep using non "__" version of
populate functions.
v3:
- Split changes to create_safe_exec_page() into several patches for
easier review as request by Mark Rutland. This is why this series
has 3 more patches.
- Renamed trans_table to tans_pgd as agreed with Mark. The header
comment in trans_pgd.c explains that trans stands for
transitional page tables. Meaning they are used in transition
between two kernels.
v2:
- Fixed hibernate bug reported by James Morse
- Addressed comments from James Morse:
* More incremental changes to trans_table
* Removed TRANS_FORCEMAP
* Added kexec reboot data for image with 380M in size.
Enable MMU during kexec relocation in order to improve reboot performance.
If kexec functionality is used for a fast system update, with a minimal
downtime, the relocation of kernel + initramfs takes a significant portion
of reboot.
The reason for slow relocation is because it is done without MMU, and thus
not benefiting from D-Cache.
Performance data
----------------
For this experiment, the size of kernel plus initramfs is small, only 25M.
If initramfs was larger, than the improvements would be greater, as time
spent in relocation is proportional to the size of relocation.
Previously:
kernel shutdown 0.022131328s
relocation 0.440510736s
kernel startup 0.294706768s
Relocation was taking: 58.2% of reboot time
Now:
kernel shutdown 0.032066576s
relocation 0.022158152s
kernel startup 0.296055880s
Now: Relocation takes 6.3% of reboot time
Total reboot is x2.16 times faster.
With bigger userland (fitImage 380M), the reboot time is improved by 3.57s,
and is reduced from 3.9s down to 0.33s
Previous approaches and discussions
-----------------------------------
v9: https://lore.kernel.org/lkml/20200326032420.27220-1-pasha.tatashin@soleen.com
v8: https://lore.kernel.org/lkml/20191204155938.2279686-1-pasha.tatashin@soleen.com
v7: https://lore.kernel.org/lkml/20191016200034.1342308-1-pasha.tatashin@soleen.com
v6: https://lore.kernel.org/lkml/20191004185234.31471-1-pasha.tatashin@soleen.com
v5: https://lore.kernel.org/lkml/20190923203427.294286-1-pasha.tatashin@soleen.com
v4: https://lore.kernel.org/lkml/20190909181221.309510-1-pasha.tatashin@soleen.com
v3: https://lore.kernel.org/lkml/20190821183204.23576-1-pasha.tatashin@soleen.com
v2: https://lore.kernel.org/lkml/20190817024629.26611-1-pasha.tatashin@soleen.com
v1: https://lore.kernel.org/lkml/20190801152439.11363-1-pasha.tatashin@soleen.com
Older approaches:
https://lore.kernel.org/lkml/20190709182014.16052-1-pasha.tatashin@soleen.com
reserve space for kexec to avoid relocation, involves changes to generic code
to optimize a problem that exists on arm64 only:
https://lore.kernel.org/lkml/20190716165641.6990-1-pasha.tatashin@soleen.com
The first attempt to enable MMU, some bugs that prevented performance
improvement. The page tables unnecessary configured idmap for the whole
physical space.
https://lore.kernel.org/lkml/20190731153857.4045-1-pasha.tatashin@soleen.com
No linear copy, bug with EL2 reboots.
James Morse (2):
arm64: mm: Always update TCR_EL1 from __cpu_set_tcr_t0sz()
arm64: trans_pgd: hibernate: idmap the single page that holds the copy
page routines
Pavel Tatashin (16):
arm64: kexec: make dtb_mem always enabled
arm64: hibernate: variable pudp is used instead of pd4dp
arm64: hibernate: move page handling function to new trans_pgd.c
arm64: trans_pgd: make trans_pgd_map_page generic
arm64: trans_pgd: pass allocator trans_pgd_create_copy
arm64: trans_pgd: pass NULL instead of init_mm to *_populate functions
arm64: kexec: move relocation function setup
arm64: kexec: call kexec_image_info only once
arm64: kexec: arm64_relocate_new_kernel clean-ups and optimizations
arm64: kexec: arm64_relocate_new_kernel don't use x0 as temp
arm64: kexec: add expandable argument to relocation function
arm64: kexec: use ld script for relocation function
arm64: kexec: kexec may require EL2 vectors
arm64: kexec: configure trans_pgd page table for kexec
arm64: kexec: enable MMU during kexec relocation
arm64: kexec: remove head from relocation argument
arch/arm64/Kconfig | 4 +
arch/arm64/include/asm/kexec.h | 42 +++-
arch/arm64/include/asm/mmu_context.h | 7 +-
arch/arm64/include/asm/sections.h | 1 +
arch/arm64/include/asm/trans_pgd.h | 39 ++++
arch/arm64/kernel/asm-offsets.c | 15 ++
arch/arm64/kernel/cpu-reset.S | 11 +-
arch/arm64/kernel/cpu-reset.h | 8 +-
arch/arm64/kernel/hibernate.c | 271 +++-------------------
arch/arm64/kernel/machine_kexec.c | 182 +++++++++++----
arch/arm64/kernel/relocate_kernel.S | 218 ++++++++++--------
arch/arm64/kernel/vmlinux.lds.S | 19 ++
arch/arm64/mm/Makefile | 1 +
arch/arm64/mm/trans_pgd.c | 324 +++++++++++++++++++++++++++
14 files changed, 749 insertions(+), 393 deletions(-)
create mode 100644 arch/arm64/include/asm/trans_pgd.h
create mode 100644 arch/arm64/mm/trans_pgd.c
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 84+ messages in thread
* [PATCH v10 00/18] arm64: MMU enabled kexec relocation
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Changelog:
v10:
- Addressed a lot of comments form James Morse and from Marc Zyngier
- Added review-by's
- Synchronized with mainline
v9: - 9 patches from previous series landed in upstream, so now series
is smaller
- Added two patches from James Morse to address idmap issues for machines
with high physical addresses.
- Addressed comments from Selin Dag about compiling issues. He also tested
my series and got similar performance results: ~60 ms instead of ~580 ms
with an initramfs size of ~120MB.
v8:
- Synced with mainline to keep series up-to-date
v7:
-- Addressed comments from James Morse
- arm64: hibernate: pass the allocated pgdp to ttbr0
Removed "Fixes" tag, and added Added Reviewed-by: James Morse
- arm64: hibernate: check pgd table allocation
Sent out as a standalone patch so it can be sent to stable
Series applies on mainline + this patch
- arm64: hibernate: add trans_pgd public functions
Remove second allocation of tmp_pg_dir in swsusp_arch_resume
Added Reviewed-by: James Morse <james.morse@arm.com>
- arm64: kexec: move relocation function setup and clean up
Fixed typo in commit log
Changed kern_reloc to phys_addr_t types.
Added explanation why kern_reloc is needed.
Split into four patches:
arm64: kexec: make dtb_mem always enabled
arm64: kexec: remove unnecessary debug prints
arm64: kexec: call kexec_image_info only once
arm64: kexec: move relocation function setup
- arm64: kexec: add expandable argument to relocation function
Changed types of new arguments from unsigned long to phys_addr_t.
Changed offset prefix to KEXEC_*
Split into four patches:
arm64: kexec: cpu_soft_restart change argument types
arm64: kexec: arm64_relocate_new_kernel clean-ups
arm64: kexec: arm64_relocate_new_kernel don't use x0 as temp
arm64: kexec: add expandable argument to relocation function
- arm64: kexec: configure trans_pgd page table for kexec
Added invalid entries into EL2 vector table
Removed KEXEC_EL2_VECTOR_TABLE_SIZE and KEXEC_EL2_VECTOR_TABLE_OFFSET
Copy relocation functions and table into separate pages
Changed types in kern_reloc_arg.
Split into three patches:
arm64: kexec: offset for relocation function
arm64: kexec: kexec EL2 vectors
arm64: kexec: configure trans_pgd page table for kexec
- arm64: kexec: enable MMU during kexec relocation
Split into two patches:
arm64: kexec: enable MMU during kexec relocation
arm64: kexec: remove head from relocation argument
v6:
- Sync with mainline tip
- Added Acked's from Dave Young
v5:
- Addressed comments from Matthias Brugger: added review-by's, improved
comments, and made cleanups to swsusp_arch_resume() in addition to
create_safe_exec_page().
- Synced with mainline tip.
v4:
- Addressed comments from James Morse.
- Split "check pgd table allocation" into two patches, and moved to
the beginning of series for simpler backport of the fixes.
Added "Fixes:" tags to commit logs.
- Changed "arm64, hibernate:" to "arm64: hibernate:"
- Added Reviewed-by's
- Moved "add PUD_SECT_RDONLY" earlier in series to be with other
clean-ups
- Added "Derived from:" to arch/arm64/mm/trans_pgd.c
- Removed "flags" from trans_info
- Changed .trans_alloc_page assumption to return zeroed page.
- Simplify changes to trans_pgd_map_page(), by keeping the old
code.
- Simplify changes to trans_pgd_create_copy, by keeping the old
code.
- Removed: "add trans_pgd_create_empty"
- replace init_mm with NULL, and keep using non "__" version of
populate functions.
v3:
- Split changes to create_safe_exec_page() into several patches for
easier review as request by Mark Rutland. This is why this series
has 3 more patches.
- Renamed trans_table to tans_pgd as agreed with Mark. The header
comment in trans_pgd.c explains that trans stands for
transitional page tables. Meaning they are used in transition
between two kernels.
v2:
- Fixed hibernate bug reported by James Morse
- Addressed comments from James Morse:
* More incremental changes to trans_table
* Removed TRANS_FORCEMAP
* Added kexec reboot data for image with 380M in size.
Enable MMU during kexec relocation in order to improve reboot performance.
If kexec functionality is used for a fast system update, with a minimal
downtime, the relocation of kernel + initramfs takes a significant portion
of reboot.
The reason for slow relocation is because it is done without MMU, and thus
not benefiting from D-Cache.
Performance data
----------------
For this experiment, the size of kernel plus initramfs is small, only 25M.
If initramfs was larger, than the improvements would be greater, as time
spent in relocation is proportional to the size of relocation.
Previously:
kernel shutdown 0.022131328s
relocation 0.440510736s
kernel startup 0.294706768s
Relocation was taking: 58.2% of reboot time
Now:
kernel shutdown 0.032066576s
relocation 0.022158152s
kernel startup 0.296055880s
Now: Relocation takes 6.3% of reboot time
Total reboot is x2.16 times faster.
With bigger userland (fitImage 380M), the reboot time is improved by 3.57s,
and is reduced from 3.9s down to 0.33s
Previous approaches and discussions
-----------------------------------
v9: https://lore.kernel.org/lkml/20200326032420.27220-1-pasha.tatashin@soleen.com
v8: https://lore.kernel.org/lkml/20191204155938.2279686-1-pasha.tatashin@soleen.com
v7: https://lore.kernel.org/lkml/20191016200034.1342308-1-pasha.tatashin@soleen.com
v6: https://lore.kernel.org/lkml/20191004185234.31471-1-pasha.tatashin@soleen.com
v5: https://lore.kernel.org/lkml/20190923203427.294286-1-pasha.tatashin@soleen.com
v4: https://lore.kernel.org/lkml/20190909181221.309510-1-pasha.tatashin@soleen.com
v3: https://lore.kernel.org/lkml/20190821183204.23576-1-pasha.tatashin@soleen.com
v2: https://lore.kernel.org/lkml/20190817024629.26611-1-pasha.tatashin@soleen.com
v1: https://lore.kernel.org/lkml/20190801152439.11363-1-pasha.tatashin@soleen.com
Older approaches:
https://lore.kernel.org/lkml/20190709182014.16052-1-pasha.tatashin@soleen.com
reserve space for kexec to avoid relocation, involves changes to generic code
to optimize a problem that exists on arm64 only:
https://lore.kernel.org/lkml/20190716165641.6990-1-pasha.tatashin@soleen.com
The first attempt to enable MMU, some bugs that prevented performance
improvement. The page tables unnecessary configured idmap for the whole
physical space.
https://lore.kernel.org/lkml/20190731153857.4045-1-pasha.tatashin@soleen.com
No linear copy, bug with EL2 reboots.
James Morse (2):
arm64: mm: Always update TCR_EL1 from __cpu_set_tcr_t0sz()
arm64: trans_pgd: hibernate: idmap the single page that holds the copy
page routines
Pavel Tatashin (16):
arm64: kexec: make dtb_mem always enabled
arm64: hibernate: variable pudp is used instead of pd4dp
arm64: hibernate: move page handling function to new trans_pgd.c
arm64: trans_pgd: make trans_pgd_map_page generic
arm64: trans_pgd: pass allocator trans_pgd_create_copy
arm64: trans_pgd: pass NULL instead of init_mm to *_populate functions
arm64: kexec: move relocation function setup
arm64: kexec: call kexec_image_info only once
arm64: kexec: arm64_relocate_new_kernel clean-ups and optimizations
arm64: kexec: arm64_relocate_new_kernel don't use x0 as temp
arm64: kexec: add expandable argument to relocation function
arm64: kexec: use ld script for relocation function
arm64: kexec: kexec may require EL2 vectors
arm64: kexec: configure trans_pgd page table for kexec
arm64: kexec: enable MMU during kexec relocation
arm64: kexec: remove head from relocation argument
arch/arm64/Kconfig | 4 +
arch/arm64/include/asm/kexec.h | 42 +++-
arch/arm64/include/asm/mmu_context.h | 7 +-
arch/arm64/include/asm/sections.h | 1 +
arch/arm64/include/asm/trans_pgd.h | 39 ++++
arch/arm64/kernel/asm-offsets.c | 15 ++
arch/arm64/kernel/cpu-reset.S | 11 +-
arch/arm64/kernel/cpu-reset.h | 8 +-
arch/arm64/kernel/hibernate.c | 271 +++-------------------
arch/arm64/kernel/machine_kexec.c | 182 +++++++++++----
arch/arm64/kernel/relocate_kernel.S | 218 ++++++++++--------
arch/arm64/kernel/vmlinux.lds.S | 19 ++
arch/arm64/mm/Makefile | 1 +
arch/arm64/mm/trans_pgd.c | 324 +++++++++++++++++++++++++++
14 files changed, 749 insertions(+), 393 deletions(-)
create mode 100644 arch/arm64/include/asm/trans_pgd.h
create mode 100644 arch/arm64/mm/trans_pgd.c
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 84+ messages in thread
* [PATCH v10 01/18] arm64: kexec: make dtb_mem always enabled
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-25 19:19 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Currently, dtb_mem is enabled only when CONFIG_KEXEC_FILE is
enabled. This adds ugly ifdefs to c files.
Always enabled dtb_mem, when it is not used, it is NULL.
Change the dtb_mem to phys_addr_t, as it is a physical address.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: James Morse <james.morse@arm.com>
---
arch/arm64/include/asm/kexec.h | 4 ++--
arch/arm64/kernel/machine_kexec.c | 6 +-----
2 files changed, 3 insertions(+), 7 deletions(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index d24b527e8c00..61530ec3a9b1 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -90,18 +90,18 @@ static inline void crash_prepare_suspend(void) {}
static inline void crash_post_resume(void) {}
#endif
-#ifdef CONFIG_KEXEC_FILE
#define ARCH_HAS_KIMAGE_ARCH
struct kimage_arch {
void *dtb;
- unsigned long dtb_mem;
+ phys_addr_t dtb_mem;
/* Core ELF header buffer */
void *elf_headers;
unsigned long elf_headers_mem;
unsigned long elf_headers_sz;
};
+#ifdef CONFIG_KEXEC_FILE
extern const struct kexec_file_ops kexec_image_ops;
struct kimage;
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index a0b144cfaea7..8096a6aa1d49 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -204,11 +204,7 @@ void machine_kexec(struct kimage *kimage)
* In kexec_file case, the kernel starts directly without purgatory.
*/
cpu_soft_restart(reboot_code_buffer_phys, kimage->head, kimage->start,
-#ifdef CONFIG_KEXEC_FILE
- kimage->arch.dtb_mem);
-#else
- 0);
-#endif
+ kimage->arch.dtb_mem);
BUG(); /* Should never get here. */
}
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 01/18] arm64: kexec: make dtb_mem always enabled
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Currently, dtb_mem is enabled only when CONFIG_KEXEC_FILE is
enabled. This adds ugly ifdefs to c files.
Always enabled dtb_mem, when it is not used, it is NULL.
Change the dtb_mem to phys_addr_t, as it is a physical address.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: James Morse <james.morse@arm.com>
---
arch/arm64/include/asm/kexec.h | 4 ++--
arch/arm64/kernel/machine_kexec.c | 6 +-----
2 files changed, 3 insertions(+), 7 deletions(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index d24b527e8c00..61530ec3a9b1 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -90,18 +90,18 @@ static inline void crash_prepare_suspend(void) {}
static inline void crash_post_resume(void) {}
#endif
-#ifdef CONFIG_KEXEC_FILE
#define ARCH_HAS_KIMAGE_ARCH
struct kimage_arch {
void *dtb;
- unsigned long dtb_mem;
+ phys_addr_t dtb_mem;
/* Core ELF header buffer */
void *elf_headers;
unsigned long elf_headers_mem;
unsigned long elf_headers_sz;
};
+#ifdef CONFIG_KEXEC_FILE
extern const struct kexec_file_ops kexec_image_ops;
struct kimage;
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index a0b144cfaea7..8096a6aa1d49 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -204,11 +204,7 @@ void machine_kexec(struct kimage *kimage)
* In kexec_file case, the kernel starts directly without purgatory.
*/
cpu_soft_restart(reboot_code_buffer_phys, kimage->head, kimage->start,
-#ifdef CONFIG_KEXEC_FILE
- kimage->arch.dtb_mem);
-#else
- 0);
-#endif
+ kimage->arch.dtb_mem);
BUG(); /* Should never get here. */
}
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 01/18] arm64: kexec: make dtb_mem always enabled
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Currently, dtb_mem is enabled only when CONFIG_KEXEC_FILE is
enabled. This adds ugly ifdefs to c files.
Always enabled dtb_mem, when it is not used, it is NULL.
Change the dtb_mem to phys_addr_t, as it is a physical address.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: James Morse <james.morse@arm.com>
---
arch/arm64/include/asm/kexec.h | 4 ++--
arch/arm64/kernel/machine_kexec.c | 6 +-----
2 files changed, 3 insertions(+), 7 deletions(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index d24b527e8c00..61530ec3a9b1 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -90,18 +90,18 @@ static inline void crash_prepare_suspend(void) {}
static inline void crash_post_resume(void) {}
#endif
-#ifdef CONFIG_KEXEC_FILE
#define ARCH_HAS_KIMAGE_ARCH
struct kimage_arch {
void *dtb;
- unsigned long dtb_mem;
+ phys_addr_t dtb_mem;
/* Core ELF header buffer */
void *elf_headers;
unsigned long elf_headers_mem;
unsigned long elf_headers_sz;
};
+#ifdef CONFIG_KEXEC_FILE
extern const struct kexec_file_ops kexec_image_ops;
struct kimage;
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index a0b144cfaea7..8096a6aa1d49 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -204,11 +204,7 @@ void machine_kexec(struct kimage *kimage)
* In kexec_file case, the kernel starts directly without purgatory.
*/
cpu_soft_restart(reboot_code_buffer_phys, kimage->head, kimage->start,
-#ifdef CONFIG_KEXEC_FILE
- kimage->arch.dtb_mem);
-#else
- 0);
-#endif
+ kimage->arch.dtb_mem);
BUG(); /* Should never get here. */
}
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 02/18] arm64: hibernate: variable pudp is used instead of pd4dp
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-25 19:19 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
There should be p4dp used when p4d page is allocated.
This is not a functional issue, but for the logical correctness this
should be fixed.
Fixes: e9f6376858b9 ("arm64: add support for folded p4d page tables")
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/kernel/hibernate.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 9c9f47e9f7f4..0a54d81c90f9 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -190,10 +190,10 @@ static int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
pgdp = pgd_offset_pgd(trans_pgd, dst_addr);
if (pgd_none(READ_ONCE(*pgdp))) {
- pudp = (void *)get_safe_page(GFP_ATOMIC);
- if (!pudp)
+ p4dp = (void *)get_safe_page(GFP_ATOMIC);
+ if (!pgdp)
return -ENOMEM;
- pgd_populate(&init_mm, pgdp, pudp);
+ pgd_populate(&init_mm, pgdp, p4dp);
}
p4dp = p4d_offset(pgdp, dst_addr);
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 02/18] arm64: hibernate: variable pudp is used instead of pd4dp
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
There should be p4dp used when p4d page is allocated.
This is not a functional issue, but for the logical correctness this
should be fixed.
Fixes: e9f6376858b9 ("arm64: add support for folded p4d page tables")
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/kernel/hibernate.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 9c9f47e9f7f4..0a54d81c90f9 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -190,10 +190,10 @@ static int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
pgdp = pgd_offset_pgd(trans_pgd, dst_addr);
if (pgd_none(READ_ONCE(*pgdp))) {
- pudp = (void *)get_safe_page(GFP_ATOMIC);
- if (!pudp)
+ p4dp = (void *)get_safe_page(GFP_ATOMIC);
+ if (!pgdp)
return -ENOMEM;
- pgd_populate(&init_mm, pgdp, pudp);
+ pgd_populate(&init_mm, pgdp, p4dp);
}
p4dp = p4d_offset(pgdp, dst_addr);
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 02/18] arm64: hibernate: variable pudp is used instead of pd4dp
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
There should be p4dp used when p4d page is allocated.
This is not a functional issue, but for the logical correctness this
should be fixed.
Fixes: e9f6376858b9 ("arm64: add support for folded p4d page tables")
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/kernel/hibernate.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 9c9f47e9f7f4..0a54d81c90f9 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -190,10 +190,10 @@ static int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
pgdp = pgd_offset_pgd(trans_pgd, dst_addr);
if (pgd_none(READ_ONCE(*pgdp))) {
- pudp = (void *)get_safe_page(GFP_ATOMIC);
- if (!pudp)
+ p4dp = (void *)get_safe_page(GFP_ATOMIC);
+ if (!pgdp)
return -ENOMEM;
- pgd_populate(&init_mm, pgdp, pudp);
+ pgd_populate(&init_mm, pgdp, p4dp);
}
p4dp = p4d_offset(pgdp, dst_addr);
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 03/18] arm64: hibernate: move page handling function to new trans_pgd.c
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-25 19:19 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Now, that we abstracted the required functions move them to a new home.
Later, we will generalize these function in order to be useful outside
of hibernation.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: James Morse <james.morse@arm.com>
---
arch/arm64/Kconfig | 4 +
arch/arm64/include/asm/trans_pgd.h | 21 +++
arch/arm64/kernel/hibernate.c | 228 +-------------------------
arch/arm64/mm/Makefile | 1 +
arch/arm64/mm/trans_pgd.c | 250 +++++++++++++++++++++++++++++
5 files changed, 277 insertions(+), 227 deletions(-)
create mode 100644 arch/arm64/include/asm/trans_pgd.h
create mode 100644 arch/arm64/mm/trans_pgd.c
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index f39568b28ec1..fc0ed9d6e011 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1132,6 +1132,10 @@ config CRASH_DUMP
For more details see Documentation/admin-guide/kdump/kdump.rst
+config TRANS_TABLE
+ def_bool y
+ depends on HIBERNATION
+
config XEN_DOM0
def_bool y
depends on XEN
diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/trans_pgd.h
new file mode 100644
index 000000000000..23153c13d1ce
--- /dev/null
+++ b/arch/arm64/include/asm/trans_pgd.h
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Copyright (c) 2020, Microsoft Corporation.
+ * Pavel Tatashin <pasha.tatashin@soleen.com>
+ */
+
+#ifndef _ASM_TRANS_TABLE_H
+#define _ASM_TRANS_TABLE_H
+
+#include <linux/bits.h>
+#include <linux/types.h>
+#include <asm/pgtable-types.h>
+
+int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
+ unsigned long end);
+
+int trans_pgd_map_page(pgd_t *trans_pgd, void *page, unsigned long dst_addr,
+ pgprot_t pgprot);
+
+#endif /* _ASM_TRANS_TABLE_H */
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 0a54d81c90f9..4a38662f0d90 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -16,7 +16,6 @@
#define pr_fmt(x) "hibernate: " x
#include <linux/cpu.h>
#include <linux/kvm_host.h>
-#include <linux/mm.h>
#include <linux/pm.h>
#include <linux/sched.h>
#include <linux/suspend.h>
@@ -31,13 +30,12 @@
#include <asm/memory.h>
#include <asm/mmu_context.h>
#include <asm/mte.h>
-#include <asm/pgalloc.h>
-#include <asm/pgtable-hwdef.h>
#include <asm/sections.h>
#include <asm/smp.h>
#include <asm/smp_plat.h>
#include <asm/suspend.h>
#include <asm/sysreg.h>
+#include <asm/trans_pgd.h>
#include <asm/virt.h>
/*
@@ -178,54 +176,6 @@ int arch_hibernation_header_restore(void *addr)
}
EXPORT_SYMBOL(arch_hibernation_header_restore);
-static int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
- unsigned long dst_addr,
- pgprot_t pgprot)
-{
- pgd_t *pgdp;
- p4d_t *p4dp;
- pud_t *pudp;
- pmd_t *pmdp;
- pte_t *ptep;
-
- pgdp = pgd_offset_pgd(trans_pgd, dst_addr);
- if (pgd_none(READ_ONCE(*pgdp))) {
- p4dp = (void *)get_safe_page(GFP_ATOMIC);
- if (!pgdp)
- return -ENOMEM;
- pgd_populate(&init_mm, pgdp, p4dp);
- }
-
- p4dp = p4d_offset(pgdp, dst_addr);
- if (p4d_none(READ_ONCE(*p4dp))) {
- pudp = (void *)get_safe_page(GFP_ATOMIC);
- if (!pudp)
- return -ENOMEM;
- p4d_populate(&init_mm, p4dp, pudp);
- }
-
- pudp = pud_offset(p4dp, dst_addr);
- if (pud_none(READ_ONCE(*pudp))) {
- pmdp = (void *)get_safe_page(GFP_ATOMIC);
- if (!pmdp)
- return -ENOMEM;
- pud_populate(&init_mm, pudp, pmdp);
- }
-
- pmdp = pmd_offset(pudp, dst_addr);
- if (pmd_none(READ_ONCE(*pmdp))) {
- ptep = (void *)get_safe_page(GFP_ATOMIC);
- if (!ptep)
- return -ENOMEM;
- pmd_populate_kernel(&init_mm, pmdp, ptep);
- }
-
- ptep = pte_offset_kernel(pmdp, dst_addr);
- set_pte(ptep, pfn_pte(virt_to_pfn(page), PAGE_KERNEL_EXEC));
-
- return 0;
-}
-
/*
* Copies length bytes, starting at src_start into an new page,
* perform cache maintenance, then maps it at the specified address low
@@ -462,182 +412,6 @@ int swsusp_arch_suspend(void)
return ret;
}
-static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr)
-{
- pte_t pte = READ_ONCE(*src_ptep);
-
- if (pte_valid(pte)) {
- /*
- * Resume will overwrite areas that may be marked
- * read only (code, rodata). Clear the RDONLY bit from
- * the temporary mappings we use during restore.
- */
- set_pte(dst_ptep, pte_mkwrite(pte));
- } else if (debug_pagealloc_enabled() && !pte_none(pte)) {
- /*
- * debug_pagealloc will removed the PTE_VALID bit if
- * the page isn't in use by the resume kernel. It may have
- * been in use by the original kernel, in which case we need
- * to put it back in our copy to do the restore.
- *
- * Before marking this entry valid, check the pfn should
- * be mapped.
- */
- BUG_ON(!pfn_valid(pte_pfn(pte)));
-
- set_pte(dst_ptep, pte_mkpresent(pte_mkwrite(pte)));
- }
-}
-
-static int copy_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long start,
- unsigned long end)
-{
- pte_t *src_ptep;
- pte_t *dst_ptep;
- unsigned long addr = start;
-
- dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC);
- if (!dst_ptep)
- return -ENOMEM;
- pmd_populate_kernel(&init_mm, dst_pmdp, dst_ptep);
- dst_ptep = pte_offset_kernel(dst_pmdp, start);
-
- src_ptep = pte_offset_kernel(src_pmdp, start);
- do {
- _copy_pte(dst_ptep, src_ptep, addr);
- } while (dst_ptep++, src_ptep++, addr += PAGE_SIZE, addr != end);
-
- return 0;
-}
-
-static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
- unsigned long end)
-{
- pmd_t *src_pmdp;
- pmd_t *dst_pmdp;
- unsigned long next;
- unsigned long addr = start;
-
- if (pud_none(READ_ONCE(*dst_pudp))) {
- dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC);
- if (!dst_pmdp)
- return -ENOMEM;
- pud_populate(&init_mm, dst_pudp, dst_pmdp);
- }
- dst_pmdp = pmd_offset(dst_pudp, start);
-
- src_pmdp = pmd_offset(src_pudp, start);
- do {
- pmd_t pmd = READ_ONCE(*src_pmdp);
-
- next = pmd_addr_end(addr, end);
- if (pmd_none(pmd))
- continue;
- if (pmd_table(pmd)) {
- if (copy_pte(dst_pmdp, src_pmdp, addr, next))
- return -ENOMEM;
- } else {
- set_pmd(dst_pmdp,
- __pmd(pmd_val(pmd) & ~PMD_SECT_RDONLY));
- }
- } while (dst_pmdp++, src_pmdp++, addr = next, addr != end);
-
- return 0;
-}
-
-static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
- unsigned long end)
-{
- pud_t *dst_pudp;
- pud_t *src_pudp;
- unsigned long next;
- unsigned long addr = start;
-
- if (p4d_none(READ_ONCE(*dst_p4dp))) {
- dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC);
- if (!dst_pudp)
- return -ENOMEM;
- p4d_populate(&init_mm, dst_p4dp, dst_pudp);
- }
- dst_pudp = pud_offset(dst_p4dp, start);
-
- src_pudp = pud_offset(src_p4dp, start);
- do {
- pud_t pud = READ_ONCE(*src_pudp);
-
- next = pud_addr_end(addr, end);
- if (pud_none(pud))
- continue;
- if (pud_table(pud)) {
- if (copy_pmd(dst_pudp, src_pudp, addr, next))
- return -ENOMEM;
- } else {
- set_pud(dst_pudp,
- __pud(pud_val(pud) & ~PUD_SECT_RDONLY));
- }
- } while (dst_pudp++, src_pudp++, addr = next, addr != end);
-
- return 0;
-}
-
-static int copy_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long start,
- unsigned long end)
-{
- p4d_t *dst_p4dp;
- p4d_t *src_p4dp;
- unsigned long next;
- unsigned long addr = start;
-
- dst_p4dp = p4d_offset(dst_pgdp, start);
- src_p4dp = p4d_offset(src_pgdp, start);
- do {
- next = p4d_addr_end(addr, end);
- if (p4d_none(READ_ONCE(*src_p4dp)))
- continue;
- if (copy_pud(dst_p4dp, src_p4dp, addr, next))
- return -ENOMEM;
- } while (dst_p4dp++, src_p4dp++, addr = next, addr != end);
-
- return 0;
-}
-
-static int copy_page_tables(pgd_t *dst_pgdp, unsigned long start,
- unsigned long end)
-{
- unsigned long next;
- unsigned long addr = start;
- pgd_t *src_pgdp = pgd_offset_k(start);
-
- dst_pgdp = pgd_offset_pgd(dst_pgdp, start);
- do {
- next = pgd_addr_end(addr, end);
- if (pgd_none(READ_ONCE(*src_pgdp)))
- continue;
- if (copy_p4d(dst_pgdp, src_pgdp, addr, next))
- return -ENOMEM;
- } while (dst_pgdp++, src_pgdp++, addr = next, addr != end);
-
- return 0;
-}
-
-static int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
- unsigned long end)
-{
- int rc;
- pgd_t *trans_pgd = (pgd_t *)get_safe_page(GFP_ATOMIC);
-
- if (!trans_pgd) {
- pr_err("Failed to allocate memory for temporary page tables.\n");
- return -ENOMEM;
- }
-
- rc = copy_page_tables(trans_pgd, start, end);
- if (!rc)
- *dst_pgdp = trans_pgd;
-
- return rc;
-}
-
/*
* Setup then Resume from the hibernate image using swsusp_arch_suspend_exit().
*
diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile
index 5ead3c3de3b6..77222d92667a 100644
--- a/arch/arm64/mm/Makefile
+++ b/arch/arm64/mm/Makefile
@@ -6,6 +6,7 @@ obj-y := dma-mapping.o extable.o fault.o init.o \
obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o
obj-$(CONFIG_PTDUMP_CORE) += ptdump.o
obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o
+obj-$(CONFIG_TRANS_TABLE) += trans_pgd.o
obj-$(CONFIG_NUMA) += numa.o
obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o
obj-$(CONFIG_ARM64_MTE) += mteswap.o
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
new file mode 100644
index 000000000000..e048d1f5c912
--- /dev/null
+++ b/arch/arm64/mm/trans_pgd.c
@@ -0,0 +1,250 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * Transitional page tables for kexec and hibernate
+ *
+ * This file derived from: arch/arm64/kernel/hibernate.c
+ *
+ * Copyright (c) 2020, Microsoft Corporation.
+ * Pavel Tatashin <pasha.tatashin@soleen.com>
+ *
+ */
+
+/*
+ * Transitional tables are used during system transferring from one world to
+ * another: such as during hibernate restore, and kexec reboots. During these
+ * phases one cannot rely on page table not being overwritten. This is because
+ * hibernate and kexec can overwrite the current page tables during transition.
+ */
+
+#include <asm/trans_pgd.h>
+#include <asm/pgalloc.h>
+#include <asm/pgtable.h>
+#include <linux/suspend.h>
+#include <linux/bug.h>
+#include <linux/mm.h>
+#include <linux/mmzone.h>
+
+static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr)
+{
+ pte_t pte = READ_ONCE(*src_ptep);
+
+ if (pte_valid(pte)) {
+ /*
+ * Resume will overwrite areas that may be marked
+ * read only (code, rodata). Clear the RDONLY bit from
+ * the temporary mappings we use during restore.
+ */
+ set_pte(dst_ptep, pte_mkwrite(pte));
+ } else if (debug_pagealloc_enabled() && !pte_none(pte)) {
+ /*
+ * debug_pagealloc will removed the PTE_VALID bit if
+ * the page isn't in use by the resume kernel. It may have
+ * been in use by the original kernel, in which case we need
+ * to put it back in our copy to do the restore.
+ *
+ * Before marking this entry valid, check the pfn should
+ * be mapped.
+ */
+ BUG_ON(!pfn_valid(pte_pfn(pte)));
+
+ set_pte(dst_ptep, pte_mkpresent(pte_mkwrite(pte)));
+ }
+}
+
+static int copy_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long start,
+ unsigned long end)
+{
+ pte_t *src_ptep;
+ pte_t *dst_ptep;
+ unsigned long addr = start;
+
+ dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC);
+ if (!dst_ptep)
+ return -ENOMEM;
+ pmd_populate_kernel(&init_mm, dst_pmdp, dst_ptep);
+ dst_ptep = pte_offset_kernel(dst_pmdp, start);
+
+ src_ptep = pte_offset_kernel(src_pmdp, start);
+ do {
+ _copy_pte(dst_ptep, src_ptep, addr);
+ } while (dst_ptep++, src_ptep++, addr += PAGE_SIZE, addr != end);
+
+ return 0;
+}
+
+static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
+ unsigned long end)
+{
+ pmd_t *src_pmdp;
+ pmd_t *dst_pmdp;
+ unsigned long next;
+ unsigned long addr = start;
+
+ if (pud_none(READ_ONCE(*dst_pudp))) {
+ dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC);
+ if (!dst_pmdp)
+ return -ENOMEM;
+ pud_populate(&init_mm, dst_pudp, dst_pmdp);
+ }
+ dst_pmdp = pmd_offset(dst_pudp, start);
+
+ src_pmdp = pmd_offset(src_pudp, start);
+ do {
+ pmd_t pmd = READ_ONCE(*src_pmdp);
+
+ next = pmd_addr_end(addr, end);
+ if (pmd_none(pmd))
+ continue;
+ if (pmd_table(pmd)) {
+ if (copy_pte(dst_pmdp, src_pmdp, addr, next))
+ return -ENOMEM;
+ } else {
+ set_pmd(dst_pmdp,
+ __pmd(pmd_val(pmd) & ~PMD_SECT_RDONLY));
+ }
+ } while (dst_pmdp++, src_pmdp++, addr = next, addr != end);
+
+ return 0;
+}
+
+static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
+ unsigned long end)
+{
+ pud_t *dst_pudp;
+ pud_t *src_pudp;
+ unsigned long next;
+ unsigned long addr = start;
+
+ if (p4d_none(READ_ONCE(*dst_p4dp))) {
+ dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC);
+ if (!dst_pudp)
+ return -ENOMEM;
+ p4d_populate(&init_mm, dst_p4dp, dst_pudp);
+ }
+ dst_pudp = pud_offset(dst_p4dp, start);
+
+ src_pudp = pud_offset(src_p4dp, start);
+ do {
+ pud_t pud = READ_ONCE(*src_pudp);
+
+ next = pud_addr_end(addr, end);
+ if (pud_none(pud))
+ continue;
+ if (pud_table(pud)) {
+ if (copy_pmd(dst_pudp, src_pudp, addr, next))
+ return -ENOMEM;
+ } else {
+ set_pud(dst_pudp,
+ __pud(pud_val(pud) & ~PUD_SECT_RDONLY));
+ }
+ } while (dst_pudp++, src_pudp++, addr = next, addr != end);
+
+ return 0;
+}
+
+static int copy_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long start,
+ unsigned long end)
+{
+ p4d_t *dst_p4dp;
+ p4d_t *src_p4dp;
+ unsigned long next;
+ unsigned long addr = start;
+
+ dst_p4dp = p4d_offset(dst_pgdp, start);
+ src_p4dp = p4d_offset(src_pgdp, start);
+ do {
+ next = p4d_addr_end(addr, end);
+ if (p4d_none(READ_ONCE(*src_p4dp)))
+ continue;
+ if (copy_pud(dst_p4dp, src_p4dp, addr, next))
+ return -ENOMEM;
+ } while (dst_p4dp++, src_p4dp++, addr = next, addr != end);
+
+ return 0;
+}
+
+static int copy_page_tables(pgd_t *dst_pgdp, unsigned long start,
+ unsigned long end)
+{
+ unsigned long next;
+ unsigned long addr = start;
+ pgd_t *src_pgdp = pgd_offset_k(start);
+
+ dst_pgdp = pgd_offset_pgd(dst_pgdp, start);
+ do {
+ next = pgd_addr_end(addr, end);
+ if (pgd_none(READ_ONCE(*src_pgdp)))
+ continue;
+ if (copy_p4d(dst_pgdp, src_pgdp, addr, next))
+ return -ENOMEM;
+ } while (dst_pgdp++, src_pgdp++, addr = next, addr != end);
+
+ return 0;
+}
+
+int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
+ unsigned long end)
+{
+ int rc;
+ pgd_t *trans_pgd = (pgd_t *)get_safe_page(GFP_ATOMIC);
+
+ if (!trans_pgd) {
+ pr_err("Failed to allocate memory for temporary page tables.\n");
+ return -ENOMEM;
+ }
+
+ rc = copy_page_tables(trans_pgd, start, end);
+ if (!rc)
+ *dst_pgdp = trans_pgd;
+
+ return rc;
+}
+
+int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
+ unsigned long dst_addr,
+ pgprot_t pgprot)
+{
+ pgd_t *pgdp;
+ p4d_t *p4dp;
+ pud_t *pudp;
+ pmd_t *pmdp;
+ pte_t *ptep;
+
+ pgdp = pgd_offset_pgd(trans_pgd, dst_addr);
+ if (pgd_none(READ_ONCE(*pgdp))) {
+ p4dp = (void *)get_safe_page(GFP_ATOMIC);
+ if (!pgdp)
+ return -ENOMEM;
+ pgd_populate(&init_mm, pgdp, p4dp);
+ }
+
+ p4dp = p4d_offset(pgdp, dst_addr);
+ if (p4d_none(READ_ONCE(*p4dp))) {
+ pudp = (void *)get_safe_page(GFP_ATOMIC);
+ if (!pudp)
+ return -ENOMEM;
+ p4d_populate(&init_mm, p4dp, pudp);
+ }
+
+ pudp = pud_offset(p4dp, dst_addr);
+ if (pud_none(READ_ONCE(*pudp))) {
+ pmdp = (void *)get_safe_page(GFP_ATOMIC);
+ if (!pmdp)
+ return -ENOMEM;
+ pud_populate(&init_mm, pudp, pmdp);
+ }
+
+ pmdp = pmd_offset(pudp, dst_addr);
+ if (pmd_none(READ_ONCE(*pmdp))) {
+ ptep = (void *)get_safe_page(GFP_ATOMIC);
+ if (!ptep)
+ return -ENOMEM;
+ pmd_populate_kernel(&init_mm, pmdp, ptep);
+ }
+
+ ptep = pte_offset_kernel(pmdp, dst_addr);
+ set_pte(ptep, pfn_pte(virt_to_pfn(page), PAGE_KERNEL_EXEC));
+
+ return 0;
+}
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 03/18] arm64: hibernate: move page handling function to new trans_pgd.c
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Now, that we abstracted the required functions move them to a new home.
Later, we will generalize these function in order to be useful outside
of hibernation.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: James Morse <james.morse@arm.com>
---
arch/arm64/Kconfig | 4 +
arch/arm64/include/asm/trans_pgd.h | 21 +++
arch/arm64/kernel/hibernate.c | 228 +-------------------------
arch/arm64/mm/Makefile | 1 +
arch/arm64/mm/trans_pgd.c | 250 +++++++++++++++++++++++++++++
5 files changed, 277 insertions(+), 227 deletions(-)
create mode 100644 arch/arm64/include/asm/trans_pgd.h
create mode 100644 arch/arm64/mm/trans_pgd.c
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index f39568b28ec1..fc0ed9d6e011 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1132,6 +1132,10 @@ config CRASH_DUMP
For more details see Documentation/admin-guide/kdump/kdump.rst
+config TRANS_TABLE
+ def_bool y
+ depends on HIBERNATION
+
config XEN_DOM0
def_bool y
depends on XEN
diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/trans_pgd.h
new file mode 100644
index 000000000000..23153c13d1ce
--- /dev/null
+++ b/arch/arm64/include/asm/trans_pgd.h
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Copyright (c) 2020, Microsoft Corporation.
+ * Pavel Tatashin <pasha.tatashin@soleen.com>
+ */
+
+#ifndef _ASM_TRANS_TABLE_H
+#define _ASM_TRANS_TABLE_H
+
+#include <linux/bits.h>
+#include <linux/types.h>
+#include <asm/pgtable-types.h>
+
+int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
+ unsigned long end);
+
+int trans_pgd_map_page(pgd_t *trans_pgd, void *page, unsigned long dst_addr,
+ pgprot_t pgprot);
+
+#endif /* _ASM_TRANS_TABLE_H */
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 0a54d81c90f9..4a38662f0d90 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -16,7 +16,6 @@
#define pr_fmt(x) "hibernate: " x
#include <linux/cpu.h>
#include <linux/kvm_host.h>
-#include <linux/mm.h>
#include <linux/pm.h>
#include <linux/sched.h>
#include <linux/suspend.h>
@@ -31,13 +30,12 @@
#include <asm/memory.h>
#include <asm/mmu_context.h>
#include <asm/mte.h>
-#include <asm/pgalloc.h>
-#include <asm/pgtable-hwdef.h>
#include <asm/sections.h>
#include <asm/smp.h>
#include <asm/smp_plat.h>
#include <asm/suspend.h>
#include <asm/sysreg.h>
+#include <asm/trans_pgd.h>
#include <asm/virt.h>
/*
@@ -178,54 +176,6 @@ int arch_hibernation_header_restore(void *addr)
}
EXPORT_SYMBOL(arch_hibernation_header_restore);
-static int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
- unsigned long dst_addr,
- pgprot_t pgprot)
-{
- pgd_t *pgdp;
- p4d_t *p4dp;
- pud_t *pudp;
- pmd_t *pmdp;
- pte_t *ptep;
-
- pgdp = pgd_offset_pgd(trans_pgd, dst_addr);
- if (pgd_none(READ_ONCE(*pgdp))) {
- p4dp = (void *)get_safe_page(GFP_ATOMIC);
- if (!pgdp)
- return -ENOMEM;
- pgd_populate(&init_mm, pgdp, p4dp);
- }
-
- p4dp = p4d_offset(pgdp, dst_addr);
- if (p4d_none(READ_ONCE(*p4dp))) {
- pudp = (void *)get_safe_page(GFP_ATOMIC);
- if (!pudp)
- return -ENOMEM;
- p4d_populate(&init_mm, p4dp, pudp);
- }
-
- pudp = pud_offset(p4dp, dst_addr);
- if (pud_none(READ_ONCE(*pudp))) {
- pmdp = (void *)get_safe_page(GFP_ATOMIC);
- if (!pmdp)
- return -ENOMEM;
- pud_populate(&init_mm, pudp, pmdp);
- }
-
- pmdp = pmd_offset(pudp, dst_addr);
- if (pmd_none(READ_ONCE(*pmdp))) {
- ptep = (void *)get_safe_page(GFP_ATOMIC);
- if (!ptep)
- return -ENOMEM;
- pmd_populate_kernel(&init_mm, pmdp, ptep);
- }
-
- ptep = pte_offset_kernel(pmdp, dst_addr);
- set_pte(ptep, pfn_pte(virt_to_pfn(page), PAGE_KERNEL_EXEC));
-
- return 0;
-}
-
/*
* Copies length bytes, starting at src_start into an new page,
* perform cache maintenance, then maps it at the specified address low
@@ -462,182 +412,6 @@ int swsusp_arch_suspend(void)
return ret;
}
-static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr)
-{
- pte_t pte = READ_ONCE(*src_ptep);
-
- if (pte_valid(pte)) {
- /*
- * Resume will overwrite areas that may be marked
- * read only (code, rodata). Clear the RDONLY bit from
- * the temporary mappings we use during restore.
- */
- set_pte(dst_ptep, pte_mkwrite(pte));
- } else if (debug_pagealloc_enabled() && !pte_none(pte)) {
- /*
- * debug_pagealloc will removed the PTE_VALID bit if
- * the page isn't in use by the resume kernel. It may have
- * been in use by the original kernel, in which case we need
- * to put it back in our copy to do the restore.
- *
- * Before marking this entry valid, check the pfn should
- * be mapped.
- */
- BUG_ON(!pfn_valid(pte_pfn(pte)));
-
- set_pte(dst_ptep, pte_mkpresent(pte_mkwrite(pte)));
- }
-}
-
-static int copy_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long start,
- unsigned long end)
-{
- pte_t *src_ptep;
- pte_t *dst_ptep;
- unsigned long addr = start;
-
- dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC);
- if (!dst_ptep)
- return -ENOMEM;
- pmd_populate_kernel(&init_mm, dst_pmdp, dst_ptep);
- dst_ptep = pte_offset_kernel(dst_pmdp, start);
-
- src_ptep = pte_offset_kernel(src_pmdp, start);
- do {
- _copy_pte(dst_ptep, src_ptep, addr);
- } while (dst_ptep++, src_ptep++, addr += PAGE_SIZE, addr != end);
-
- return 0;
-}
-
-static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
- unsigned long end)
-{
- pmd_t *src_pmdp;
- pmd_t *dst_pmdp;
- unsigned long next;
- unsigned long addr = start;
-
- if (pud_none(READ_ONCE(*dst_pudp))) {
- dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC);
- if (!dst_pmdp)
- return -ENOMEM;
- pud_populate(&init_mm, dst_pudp, dst_pmdp);
- }
- dst_pmdp = pmd_offset(dst_pudp, start);
-
- src_pmdp = pmd_offset(src_pudp, start);
- do {
- pmd_t pmd = READ_ONCE(*src_pmdp);
-
- next = pmd_addr_end(addr, end);
- if (pmd_none(pmd))
- continue;
- if (pmd_table(pmd)) {
- if (copy_pte(dst_pmdp, src_pmdp, addr, next))
- return -ENOMEM;
- } else {
- set_pmd(dst_pmdp,
- __pmd(pmd_val(pmd) & ~PMD_SECT_RDONLY));
- }
- } while (dst_pmdp++, src_pmdp++, addr = next, addr != end);
-
- return 0;
-}
-
-static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
- unsigned long end)
-{
- pud_t *dst_pudp;
- pud_t *src_pudp;
- unsigned long next;
- unsigned long addr = start;
-
- if (p4d_none(READ_ONCE(*dst_p4dp))) {
- dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC);
- if (!dst_pudp)
- return -ENOMEM;
- p4d_populate(&init_mm, dst_p4dp, dst_pudp);
- }
- dst_pudp = pud_offset(dst_p4dp, start);
-
- src_pudp = pud_offset(src_p4dp, start);
- do {
- pud_t pud = READ_ONCE(*src_pudp);
-
- next = pud_addr_end(addr, end);
- if (pud_none(pud))
- continue;
- if (pud_table(pud)) {
- if (copy_pmd(dst_pudp, src_pudp, addr, next))
- return -ENOMEM;
- } else {
- set_pud(dst_pudp,
- __pud(pud_val(pud) & ~PUD_SECT_RDONLY));
- }
- } while (dst_pudp++, src_pudp++, addr = next, addr != end);
-
- return 0;
-}
-
-static int copy_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long start,
- unsigned long end)
-{
- p4d_t *dst_p4dp;
- p4d_t *src_p4dp;
- unsigned long next;
- unsigned long addr = start;
-
- dst_p4dp = p4d_offset(dst_pgdp, start);
- src_p4dp = p4d_offset(src_pgdp, start);
- do {
- next = p4d_addr_end(addr, end);
- if (p4d_none(READ_ONCE(*src_p4dp)))
- continue;
- if (copy_pud(dst_p4dp, src_p4dp, addr, next))
- return -ENOMEM;
- } while (dst_p4dp++, src_p4dp++, addr = next, addr != end);
-
- return 0;
-}
-
-static int copy_page_tables(pgd_t *dst_pgdp, unsigned long start,
- unsigned long end)
-{
- unsigned long next;
- unsigned long addr = start;
- pgd_t *src_pgdp = pgd_offset_k(start);
-
- dst_pgdp = pgd_offset_pgd(dst_pgdp, start);
- do {
- next = pgd_addr_end(addr, end);
- if (pgd_none(READ_ONCE(*src_pgdp)))
- continue;
- if (copy_p4d(dst_pgdp, src_pgdp, addr, next))
- return -ENOMEM;
- } while (dst_pgdp++, src_pgdp++, addr = next, addr != end);
-
- return 0;
-}
-
-static int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
- unsigned long end)
-{
- int rc;
- pgd_t *trans_pgd = (pgd_t *)get_safe_page(GFP_ATOMIC);
-
- if (!trans_pgd) {
- pr_err("Failed to allocate memory for temporary page tables.\n");
- return -ENOMEM;
- }
-
- rc = copy_page_tables(trans_pgd, start, end);
- if (!rc)
- *dst_pgdp = trans_pgd;
-
- return rc;
-}
-
/*
* Setup then Resume from the hibernate image using swsusp_arch_suspend_exit().
*
diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile
index 5ead3c3de3b6..77222d92667a 100644
--- a/arch/arm64/mm/Makefile
+++ b/arch/arm64/mm/Makefile
@@ -6,6 +6,7 @@ obj-y := dma-mapping.o extable.o fault.o init.o \
obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o
obj-$(CONFIG_PTDUMP_CORE) += ptdump.o
obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o
+obj-$(CONFIG_TRANS_TABLE) += trans_pgd.o
obj-$(CONFIG_NUMA) += numa.o
obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o
obj-$(CONFIG_ARM64_MTE) += mteswap.o
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
new file mode 100644
index 000000000000..e048d1f5c912
--- /dev/null
+++ b/arch/arm64/mm/trans_pgd.c
@@ -0,0 +1,250 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * Transitional page tables for kexec and hibernate
+ *
+ * This file derived from: arch/arm64/kernel/hibernate.c
+ *
+ * Copyright (c) 2020, Microsoft Corporation.
+ * Pavel Tatashin <pasha.tatashin@soleen.com>
+ *
+ */
+
+/*
+ * Transitional tables are used during system transferring from one world to
+ * another: such as during hibernate restore, and kexec reboots. During these
+ * phases one cannot rely on page table not being overwritten. This is because
+ * hibernate and kexec can overwrite the current page tables during transition.
+ */
+
+#include <asm/trans_pgd.h>
+#include <asm/pgalloc.h>
+#include <asm/pgtable.h>
+#include <linux/suspend.h>
+#include <linux/bug.h>
+#include <linux/mm.h>
+#include <linux/mmzone.h>
+
+static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr)
+{
+ pte_t pte = READ_ONCE(*src_ptep);
+
+ if (pte_valid(pte)) {
+ /*
+ * Resume will overwrite areas that may be marked
+ * read only (code, rodata). Clear the RDONLY bit from
+ * the temporary mappings we use during restore.
+ */
+ set_pte(dst_ptep, pte_mkwrite(pte));
+ } else if (debug_pagealloc_enabled() && !pte_none(pte)) {
+ /*
+ * debug_pagealloc will removed the PTE_VALID bit if
+ * the page isn't in use by the resume kernel. It may have
+ * been in use by the original kernel, in which case we need
+ * to put it back in our copy to do the restore.
+ *
+ * Before marking this entry valid, check the pfn should
+ * be mapped.
+ */
+ BUG_ON(!pfn_valid(pte_pfn(pte)));
+
+ set_pte(dst_ptep, pte_mkpresent(pte_mkwrite(pte)));
+ }
+}
+
+static int copy_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long start,
+ unsigned long end)
+{
+ pte_t *src_ptep;
+ pte_t *dst_ptep;
+ unsigned long addr = start;
+
+ dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC);
+ if (!dst_ptep)
+ return -ENOMEM;
+ pmd_populate_kernel(&init_mm, dst_pmdp, dst_ptep);
+ dst_ptep = pte_offset_kernel(dst_pmdp, start);
+
+ src_ptep = pte_offset_kernel(src_pmdp, start);
+ do {
+ _copy_pte(dst_ptep, src_ptep, addr);
+ } while (dst_ptep++, src_ptep++, addr += PAGE_SIZE, addr != end);
+
+ return 0;
+}
+
+static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
+ unsigned long end)
+{
+ pmd_t *src_pmdp;
+ pmd_t *dst_pmdp;
+ unsigned long next;
+ unsigned long addr = start;
+
+ if (pud_none(READ_ONCE(*dst_pudp))) {
+ dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC);
+ if (!dst_pmdp)
+ return -ENOMEM;
+ pud_populate(&init_mm, dst_pudp, dst_pmdp);
+ }
+ dst_pmdp = pmd_offset(dst_pudp, start);
+
+ src_pmdp = pmd_offset(src_pudp, start);
+ do {
+ pmd_t pmd = READ_ONCE(*src_pmdp);
+
+ next = pmd_addr_end(addr, end);
+ if (pmd_none(pmd))
+ continue;
+ if (pmd_table(pmd)) {
+ if (copy_pte(dst_pmdp, src_pmdp, addr, next))
+ return -ENOMEM;
+ } else {
+ set_pmd(dst_pmdp,
+ __pmd(pmd_val(pmd) & ~PMD_SECT_RDONLY));
+ }
+ } while (dst_pmdp++, src_pmdp++, addr = next, addr != end);
+
+ return 0;
+}
+
+static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
+ unsigned long end)
+{
+ pud_t *dst_pudp;
+ pud_t *src_pudp;
+ unsigned long next;
+ unsigned long addr = start;
+
+ if (p4d_none(READ_ONCE(*dst_p4dp))) {
+ dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC);
+ if (!dst_pudp)
+ return -ENOMEM;
+ p4d_populate(&init_mm, dst_p4dp, dst_pudp);
+ }
+ dst_pudp = pud_offset(dst_p4dp, start);
+
+ src_pudp = pud_offset(src_p4dp, start);
+ do {
+ pud_t pud = READ_ONCE(*src_pudp);
+
+ next = pud_addr_end(addr, end);
+ if (pud_none(pud))
+ continue;
+ if (pud_table(pud)) {
+ if (copy_pmd(dst_pudp, src_pudp, addr, next))
+ return -ENOMEM;
+ } else {
+ set_pud(dst_pudp,
+ __pud(pud_val(pud) & ~PUD_SECT_RDONLY));
+ }
+ } while (dst_pudp++, src_pudp++, addr = next, addr != end);
+
+ return 0;
+}
+
+static int copy_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long start,
+ unsigned long end)
+{
+ p4d_t *dst_p4dp;
+ p4d_t *src_p4dp;
+ unsigned long next;
+ unsigned long addr = start;
+
+ dst_p4dp = p4d_offset(dst_pgdp, start);
+ src_p4dp = p4d_offset(src_pgdp, start);
+ do {
+ next = p4d_addr_end(addr, end);
+ if (p4d_none(READ_ONCE(*src_p4dp)))
+ continue;
+ if (copy_pud(dst_p4dp, src_p4dp, addr, next))
+ return -ENOMEM;
+ } while (dst_p4dp++, src_p4dp++, addr = next, addr != end);
+
+ return 0;
+}
+
+static int copy_page_tables(pgd_t *dst_pgdp, unsigned long start,
+ unsigned long end)
+{
+ unsigned long next;
+ unsigned long addr = start;
+ pgd_t *src_pgdp = pgd_offset_k(start);
+
+ dst_pgdp = pgd_offset_pgd(dst_pgdp, start);
+ do {
+ next = pgd_addr_end(addr, end);
+ if (pgd_none(READ_ONCE(*src_pgdp)))
+ continue;
+ if (copy_p4d(dst_pgdp, src_pgdp, addr, next))
+ return -ENOMEM;
+ } while (dst_pgdp++, src_pgdp++, addr = next, addr != end);
+
+ return 0;
+}
+
+int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
+ unsigned long end)
+{
+ int rc;
+ pgd_t *trans_pgd = (pgd_t *)get_safe_page(GFP_ATOMIC);
+
+ if (!trans_pgd) {
+ pr_err("Failed to allocate memory for temporary page tables.\n");
+ return -ENOMEM;
+ }
+
+ rc = copy_page_tables(trans_pgd, start, end);
+ if (!rc)
+ *dst_pgdp = trans_pgd;
+
+ return rc;
+}
+
+int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
+ unsigned long dst_addr,
+ pgprot_t pgprot)
+{
+ pgd_t *pgdp;
+ p4d_t *p4dp;
+ pud_t *pudp;
+ pmd_t *pmdp;
+ pte_t *ptep;
+
+ pgdp = pgd_offset_pgd(trans_pgd, dst_addr);
+ if (pgd_none(READ_ONCE(*pgdp))) {
+ p4dp = (void *)get_safe_page(GFP_ATOMIC);
+ if (!pgdp)
+ return -ENOMEM;
+ pgd_populate(&init_mm, pgdp, p4dp);
+ }
+
+ p4dp = p4d_offset(pgdp, dst_addr);
+ if (p4d_none(READ_ONCE(*p4dp))) {
+ pudp = (void *)get_safe_page(GFP_ATOMIC);
+ if (!pudp)
+ return -ENOMEM;
+ p4d_populate(&init_mm, p4dp, pudp);
+ }
+
+ pudp = pud_offset(p4dp, dst_addr);
+ if (pud_none(READ_ONCE(*pudp))) {
+ pmdp = (void *)get_safe_page(GFP_ATOMIC);
+ if (!pmdp)
+ return -ENOMEM;
+ pud_populate(&init_mm, pudp, pmdp);
+ }
+
+ pmdp = pmd_offset(pudp, dst_addr);
+ if (pmd_none(READ_ONCE(*pmdp))) {
+ ptep = (void *)get_safe_page(GFP_ATOMIC);
+ if (!ptep)
+ return -ENOMEM;
+ pmd_populate_kernel(&init_mm, pmdp, ptep);
+ }
+
+ ptep = pte_offset_kernel(pmdp, dst_addr);
+ set_pte(ptep, pfn_pte(virt_to_pfn(page), PAGE_KERNEL_EXEC));
+
+ return 0;
+}
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 03/18] arm64: hibernate: move page handling function to new trans_pgd.c
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Now, that we abstracted the required functions move them to a new home.
Later, we will generalize these function in order to be useful outside
of hibernation.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: James Morse <james.morse@arm.com>
---
arch/arm64/Kconfig | 4 +
arch/arm64/include/asm/trans_pgd.h | 21 +++
arch/arm64/kernel/hibernate.c | 228 +-------------------------
arch/arm64/mm/Makefile | 1 +
arch/arm64/mm/trans_pgd.c | 250 +++++++++++++++++++++++++++++
5 files changed, 277 insertions(+), 227 deletions(-)
create mode 100644 arch/arm64/include/asm/trans_pgd.h
create mode 100644 arch/arm64/mm/trans_pgd.c
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index f39568b28ec1..fc0ed9d6e011 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1132,6 +1132,10 @@ config CRASH_DUMP
For more details see Documentation/admin-guide/kdump/kdump.rst
+config TRANS_TABLE
+ def_bool y
+ depends on HIBERNATION
+
config XEN_DOM0
def_bool y
depends on XEN
diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/trans_pgd.h
new file mode 100644
index 000000000000..23153c13d1ce
--- /dev/null
+++ b/arch/arm64/include/asm/trans_pgd.h
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Copyright (c) 2020, Microsoft Corporation.
+ * Pavel Tatashin <pasha.tatashin@soleen.com>
+ */
+
+#ifndef _ASM_TRANS_TABLE_H
+#define _ASM_TRANS_TABLE_H
+
+#include <linux/bits.h>
+#include <linux/types.h>
+#include <asm/pgtable-types.h>
+
+int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
+ unsigned long end);
+
+int trans_pgd_map_page(pgd_t *trans_pgd, void *page, unsigned long dst_addr,
+ pgprot_t pgprot);
+
+#endif /* _ASM_TRANS_TABLE_H */
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 0a54d81c90f9..4a38662f0d90 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -16,7 +16,6 @@
#define pr_fmt(x) "hibernate: " x
#include <linux/cpu.h>
#include <linux/kvm_host.h>
-#include <linux/mm.h>
#include <linux/pm.h>
#include <linux/sched.h>
#include <linux/suspend.h>
@@ -31,13 +30,12 @@
#include <asm/memory.h>
#include <asm/mmu_context.h>
#include <asm/mte.h>
-#include <asm/pgalloc.h>
-#include <asm/pgtable-hwdef.h>
#include <asm/sections.h>
#include <asm/smp.h>
#include <asm/smp_plat.h>
#include <asm/suspend.h>
#include <asm/sysreg.h>
+#include <asm/trans_pgd.h>
#include <asm/virt.h>
/*
@@ -178,54 +176,6 @@ int arch_hibernation_header_restore(void *addr)
}
EXPORT_SYMBOL(arch_hibernation_header_restore);
-static int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
- unsigned long dst_addr,
- pgprot_t pgprot)
-{
- pgd_t *pgdp;
- p4d_t *p4dp;
- pud_t *pudp;
- pmd_t *pmdp;
- pte_t *ptep;
-
- pgdp = pgd_offset_pgd(trans_pgd, dst_addr);
- if (pgd_none(READ_ONCE(*pgdp))) {
- p4dp = (void *)get_safe_page(GFP_ATOMIC);
- if (!pgdp)
- return -ENOMEM;
- pgd_populate(&init_mm, pgdp, p4dp);
- }
-
- p4dp = p4d_offset(pgdp, dst_addr);
- if (p4d_none(READ_ONCE(*p4dp))) {
- pudp = (void *)get_safe_page(GFP_ATOMIC);
- if (!pudp)
- return -ENOMEM;
- p4d_populate(&init_mm, p4dp, pudp);
- }
-
- pudp = pud_offset(p4dp, dst_addr);
- if (pud_none(READ_ONCE(*pudp))) {
- pmdp = (void *)get_safe_page(GFP_ATOMIC);
- if (!pmdp)
- return -ENOMEM;
- pud_populate(&init_mm, pudp, pmdp);
- }
-
- pmdp = pmd_offset(pudp, dst_addr);
- if (pmd_none(READ_ONCE(*pmdp))) {
- ptep = (void *)get_safe_page(GFP_ATOMIC);
- if (!ptep)
- return -ENOMEM;
- pmd_populate_kernel(&init_mm, pmdp, ptep);
- }
-
- ptep = pte_offset_kernel(pmdp, dst_addr);
- set_pte(ptep, pfn_pte(virt_to_pfn(page), PAGE_KERNEL_EXEC));
-
- return 0;
-}
-
/*
* Copies length bytes, starting at src_start into an new page,
* perform cache maintenance, then maps it at the specified address low
@@ -462,182 +412,6 @@ int swsusp_arch_suspend(void)
return ret;
}
-static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr)
-{
- pte_t pte = READ_ONCE(*src_ptep);
-
- if (pte_valid(pte)) {
- /*
- * Resume will overwrite areas that may be marked
- * read only (code, rodata). Clear the RDONLY bit from
- * the temporary mappings we use during restore.
- */
- set_pte(dst_ptep, pte_mkwrite(pte));
- } else if (debug_pagealloc_enabled() && !pte_none(pte)) {
- /*
- * debug_pagealloc will removed the PTE_VALID bit if
- * the page isn't in use by the resume kernel. It may have
- * been in use by the original kernel, in which case we need
- * to put it back in our copy to do the restore.
- *
- * Before marking this entry valid, check the pfn should
- * be mapped.
- */
- BUG_ON(!pfn_valid(pte_pfn(pte)));
-
- set_pte(dst_ptep, pte_mkpresent(pte_mkwrite(pte)));
- }
-}
-
-static int copy_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long start,
- unsigned long end)
-{
- pte_t *src_ptep;
- pte_t *dst_ptep;
- unsigned long addr = start;
-
- dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC);
- if (!dst_ptep)
- return -ENOMEM;
- pmd_populate_kernel(&init_mm, dst_pmdp, dst_ptep);
- dst_ptep = pte_offset_kernel(dst_pmdp, start);
-
- src_ptep = pte_offset_kernel(src_pmdp, start);
- do {
- _copy_pte(dst_ptep, src_ptep, addr);
- } while (dst_ptep++, src_ptep++, addr += PAGE_SIZE, addr != end);
-
- return 0;
-}
-
-static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
- unsigned long end)
-{
- pmd_t *src_pmdp;
- pmd_t *dst_pmdp;
- unsigned long next;
- unsigned long addr = start;
-
- if (pud_none(READ_ONCE(*dst_pudp))) {
- dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC);
- if (!dst_pmdp)
- return -ENOMEM;
- pud_populate(&init_mm, dst_pudp, dst_pmdp);
- }
- dst_pmdp = pmd_offset(dst_pudp, start);
-
- src_pmdp = pmd_offset(src_pudp, start);
- do {
- pmd_t pmd = READ_ONCE(*src_pmdp);
-
- next = pmd_addr_end(addr, end);
- if (pmd_none(pmd))
- continue;
- if (pmd_table(pmd)) {
- if (copy_pte(dst_pmdp, src_pmdp, addr, next))
- return -ENOMEM;
- } else {
- set_pmd(dst_pmdp,
- __pmd(pmd_val(pmd) & ~PMD_SECT_RDONLY));
- }
- } while (dst_pmdp++, src_pmdp++, addr = next, addr != end);
-
- return 0;
-}
-
-static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
- unsigned long end)
-{
- pud_t *dst_pudp;
- pud_t *src_pudp;
- unsigned long next;
- unsigned long addr = start;
-
- if (p4d_none(READ_ONCE(*dst_p4dp))) {
- dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC);
- if (!dst_pudp)
- return -ENOMEM;
- p4d_populate(&init_mm, dst_p4dp, dst_pudp);
- }
- dst_pudp = pud_offset(dst_p4dp, start);
-
- src_pudp = pud_offset(src_p4dp, start);
- do {
- pud_t pud = READ_ONCE(*src_pudp);
-
- next = pud_addr_end(addr, end);
- if (pud_none(pud))
- continue;
- if (pud_table(pud)) {
- if (copy_pmd(dst_pudp, src_pudp, addr, next))
- return -ENOMEM;
- } else {
- set_pud(dst_pudp,
- __pud(pud_val(pud) & ~PUD_SECT_RDONLY));
- }
- } while (dst_pudp++, src_pudp++, addr = next, addr != end);
-
- return 0;
-}
-
-static int copy_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long start,
- unsigned long end)
-{
- p4d_t *dst_p4dp;
- p4d_t *src_p4dp;
- unsigned long next;
- unsigned long addr = start;
-
- dst_p4dp = p4d_offset(dst_pgdp, start);
- src_p4dp = p4d_offset(src_pgdp, start);
- do {
- next = p4d_addr_end(addr, end);
- if (p4d_none(READ_ONCE(*src_p4dp)))
- continue;
- if (copy_pud(dst_p4dp, src_p4dp, addr, next))
- return -ENOMEM;
- } while (dst_p4dp++, src_p4dp++, addr = next, addr != end);
-
- return 0;
-}
-
-static int copy_page_tables(pgd_t *dst_pgdp, unsigned long start,
- unsigned long end)
-{
- unsigned long next;
- unsigned long addr = start;
- pgd_t *src_pgdp = pgd_offset_k(start);
-
- dst_pgdp = pgd_offset_pgd(dst_pgdp, start);
- do {
- next = pgd_addr_end(addr, end);
- if (pgd_none(READ_ONCE(*src_pgdp)))
- continue;
- if (copy_p4d(dst_pgdp, src_pgdp, addr, next))
- return -ENOMEM;
- } while (dst_pgdp++, src_pgdp++, addr = next, addr != end);
-
- return 0;
-}
-
-static int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
- unsigned long end)
-{
- int rc;
- pgd_t *trans_pgd = (pgd_t *)get_safe_page(GFP_ATOMIC);
-
- if (!trans_pgd) {
- pr_err("Failed to allocate memory for temporary page tables.\n");
- return -ENOMEM;
- }
-
- rc = copy_page_tables(trans_pgd, start, end);
- if (!rc)
- *dst_pgdp = trans_pgd;
-
- return rc;
-}
-
/*
* Setup then Resume from the hibernate image using swsusp_arch_suspend_exit().
*
diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile
index 5ead3c3de3b6..77222d92667a 100644
--- a/arch/arm64/mm/Makefile
+++ b/arch/arm64/mm/Makefile
@@ -6,6 +6,7 @@ obj-y := dma-mapping.o extable.o fault.o init.o \
obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o
obj-$(CONFIG_PTDUMP_CORE) += ptdump.o
obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o
+obj-$(CONFIG_TRANS_TABLE) += trans_pgd.o
obj-$(CONFIG_NUMA) += numa.o
obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o
obj-$(CONFIG_ARM64_MTE) += mteswap.o
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
new file mode 100644
index 000000000000..e048d1f5c912
--- /dev/null
+++ b/arch/arm64/mm/trans_pgd.c
@@ -0,0 +1,250 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/*
+ * Transitional page tables for kexec and hibernate
+ *
+ * This file derived from: arch/arm64/kernel/hibernate.c
+ *
+ * Copyright (c) 2020, Microsoft Corporation.
+ * Pavel Tatashin <pasha.tatashin@soleen.com>
+ *
+ */
+
+/*
+ * Transitional tables are used during system transferring from one world to
+ * another: such as during hibernate restore, and kexec reboots. During these
+ * phases one cannot rely on page table not being overwritten. This is because
+ * hibernate and kexec can overwrite the current page tables during transition.
+ */
+
+#include <asm/trans_pgd.h>
+#include <asm/pgalloc.h>
+#include <asm/pgtable.h>
+#include <linux/suspend.h>
+#include <linux/bug.h>
+#include <linux/mm.h>
+#include <linux/mmzone.h>
+
+static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr)
+{
+ pte_t pte = READ_ONCE(*src_ptep);
+
+ if (pte_valid(pte)) {
+ /*
+ * Resume will overwrite areas that may be marked
+ * read only (code, rodata). Clear the RDONLY bit from
+ * the temporary mappings we use during restore.
+ */
+ set_pte(dst_ptep, pte_mkwrite(pte));
+ } else if (debug_pagealloc_enabled() && !pte_none(pte)) {
+ /*
+ * debug_pagealloc will removed the PTE_VALID bit if
+ * the page isn't in use by the resume kernel. It may have
+ * been in use by the original kernel, in which case we need
+ * to put it back in our copy to do the restore.
+ *
+ * Before marking this entry valid, check the pfn should
+ * be mapped.
+ */
+ BUG_ON(!pfn_valid(pte_pfn(pte)));
+
+ set_pte(dst_ptep, pte_mkpresent(pte_mkwrite(pte)));
+ }
+}
+
+static int copy_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long start,
+ unsigned long end)
+{
+ pte_t *src_ptep;
+ pte_t *dst_ptep;
+ unsigned long addr = start;
+
+ dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC);
+ if (!dst_ptep)
+ return -ENOMEM;
+ pmd_populate_kernel(&init_mm, dst_pmdp, dst_ptep);
+ dst_ptep = pte_offset_kernel(dst_pmdp, start);
+
+ src_ptep = pte_offset_kernel(src_pmdp, start);
+ do {
+ _copy_pte(dst_ptep, src_ptep, addr);
+ } while (dst_ptep++, src_ptep++, addr += PAGE_SIZE, addr != end);
+
+ return 0;
+}
+
+static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
+ unsigned long end)
+{
+ pmd_t *src_pmdp;
+ pmd_t *dst_pmdp;
+ unsigned long next;
+ unsigned long addr = start;
+
+ if (pud_none(READ_ONCE(*dst_pudp))) {
+ dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC);
+ if (!dst_pmdp)
+ return -ENOMEM;
+ pud_populate(&init_mm, dst_pudp, dst_pmdp);
+ }
+ dst_pmdp = pmd_offset(dst_pudp, start);
+
+ src_pmdp = pmd_offset(src_pudp, start);
+ do {
+ pmd_t pmd = READ_ONCE(*src_pmdp);
+
+ next = pmd_addr_end(addr, end);
+ if (pmd_none(pmd))
+ continue;
+ if (pmd_table(pmd)) {
+ if (copy_pte(dst_pmdp, src_pmdp, addr, next))
+ return -ENOMEM;
+ } else {
+ set_pmd(dst_pmdp,
+ __pmd(pmd_val(pmd) & ~PMD_SECT_RDONLY));
+ }
+ } while (dst_pmdp++, src_pmdp++, addr = next, addr != end);
+
+ return 0;
+}
+
+static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
+ unsigned long end)
+{
+ pud_t *dst_pudp;
+ pud_t *src_pudp;
+ unsigned long next;
+ unsigned long addr = start;
+
+ if (p4d_none(READ_ONCE(*dst_p4dp))) {
+ dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC);
+ if (!dst_pudp)
+ return -ENOMEM;
+ p4d_populate(&init_mm, dst_p4dp, dst_pudp);
+ }
+ dst_pudp = pud_offset(dst_p4dp, start);
+
+ src_pudp = pud_offset(src_p4dp, start);
+ do {
+ pud_t pud = READ_ONCE(*src_pudp);
+
+ next = pud_addr_end(addr, end);
+ if (pud_none(pud))
+ continue;
+ if (pud_table(pud)) {
+ if (copy_pmd(dst_pudp, src_pudp, addr, next))
+ return -ENOMEM;
+ } else {
+ set_pud(dst_pudp,
+ __pud(pud_val(pud) & ~PUD_SECT_RDONLY));
+ }
+ } while (dst_pudp++, src_pudp++, addr = next, addr != end);
+
+ return 0;
+}
+
+static int copy_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long start,
+ unsigned long end)
+{
+ p4d_t *dst_p4dp;
+ p4d_t *src_p4dp;
+ unsigned long next;
+ unsigned long addr = start;
+
+ dst_p4dp = p4d_offset(dst_pgdp, start);
+ src_p4dp = p4d_offset(src_pgdp, start);
+ do {
+ next = p4d_addr_end(addr, end);
+ if (p4d_none(READ_ONCE(*src_p4dp)))
+ continue;
+ if (copy_pud(dst_p4dp, src_p4dp, addr, next))
+ return -ENOMEM;
+ } while (dst_p4dp++, src_p4dp++, addr = next, addr != end);
+
+ return 0;
+}
+
+static int copy_page_tables(pgd_t *dst_pgdp, unsigned long start,
+ unsigned long end)
+{
+ unsigned long next;
+ unsigned long addr = start;
+ pgd_t *src_pgdp = pgd_offset_k(start);
+
+ dst_pgdp = pgd_offset_pgd(dst_pgdp, start);
+ do {
+ next = pgd_addr_end(addr, end);
+ if (pgd_none(READ_ONCE(*src_pgdp)))
+ continue;
+ if (copy_p4d(dst_pgdp, src_pgdp, addr, next))
+ return -ENOMEM;
+ } while (dst_pgdp++, src_pgdp++, addr = next, addr != end);
+
+ return 0;
+}
+
+int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
+ unsigned long end)
+{
+ int rc;
+ pgd_t *trans_pgd = (pgd_t *)get_safe_page(GFP_ATOMIC);
+
+ if (!trans_pgd) {
+ pr_err("Failed to allocate memory for temporary page tables.\n");
+ return -ENOMEM;
+ }
+
+ rc = copy_page_tables(trans_pgd, start, end);
+ if (!rc)
+ *dst_pgdp = trans_pgd;
+
+ return rc;
+}
+
+int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
+ unsigned long dst_addr,
+ pgprot_t pgprot)
+{
+ pgd_t *pgdp;
+ p4d_t *p4dp;
+ pud_t *pudp;
+ pmd_t *pmdp;
+ pte_t *ptep;
+
+ pgdp = pgd_offset_pgd(trans_pgd, dst_addr);
+ if (pgd_none(READ_ONCE(*pgdp))) {
+ p4dp = (void *)get_safe_page(GFP_ATOMIC);
+ if (!pgdp)
+ return -ENOMEM;
+ pgd_populate(&init_mm, pgdp, p4dp);
+ }
+
+ p4dp = p4d_offset(pgdp, dst_addr);
+ if (p4d_none(READ_ONCE(*p4dp))) {
+ pudp = (void *)get_safe_page(GFP_ATOMIC);
+ if (!pudp)
+ return -ENOMEM;
+ p4d_populate(&init_mm, p4dp, pudp);
+ }
+
+ pudp = pud_offset(p4dp, dst_addr);
+ if (pud_none(READ_ONCE(*pudp))) {
+ pmdp = (void *)get_safe_page(GFP_ATOMIC);
+ if (!pmdp)
+ return -ENOMEM;
+ pud_populate(&init_mm, pudp, pmdp);
+ }
+
+ pmdp = pmd_offset(pudp, dst_addr);
+ if (pmd_none(READ_ONCE(*pmdp))) {
+ ptep = (void *)get_safe_page(GFP_ATOMIC);
+ if (!ptep)
+ return -ENOMEM;
+ pmd_populate_kernel(&init_mm, pmdp, ptep);
+ }
+
+ ptep = pte_offset_kernel(pmdp, dst_addr);
+ set_pte(ptep, pfn_pte(virt_to_pfn(page), PAGE_KERNEL_EXEC));
+
+ return 0;
+}
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 04/18] arm64: trans_pgd: make trans_pgd_map_page generic
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-25 19:19 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
kexec is going to use a different allocator, so make
trans_pgd_map_page to accept allocator as an argument, and also
kexec is going to use a different map protection, so also pass
it via argument.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: Matthias Brugger <mbrugger@suse.com>
---
arch/arm64/include/asm/trans_pgd.h | 19 +++++++++++++++++--
arch/arm64/kernel/hibernate.c | 12 +++++++++++-
arch/arm64/mm/trans_pgd.c | 30 ++++++++++++++++++++++--------
3 files changed, 50 insertions(+), 11 deletions(-)
diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/trans_pgd.h
index 23153c13d1ce..b46409b25234 100644
--- a/arch/arm64/include/asm/trans_pgd.h
+++ b/arch/arm64/include/asm/trans_pgd.h
@@ -12,10 +12,25 @@
#include <linux/types.h>
#include <asm/pgtable-types.h>
+/*
+ * trans_alloc_page
+ * - Allocator that should return exactly one zeroed page, if this
+ * allocator fails, trans_pgd_create_copy() and trans_pgd_map_page()
+ * return -ENOMEM error.
+ *
+ * trans_alloc_arg
+ * - Passed to trans_alloc_page as an argument
+ */
+
+struct trans_pgd_info {
+ void * (*trans_alloc_page)(void *arg);
+ void *trans_alloc_arg;
+};
+
int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
unsigned long end);
-int trans_pgd_map_page(pgd_t *trans_pgd, void *page, unsigned long dst_addr,
- pgprot_t pgprot);
+int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
+ void *page, unsigned long dst_addr, pgprot_t pgprot);
#endif /* _ASM_TRANS_TABLE_H */
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 4a38662f0d90..c173f280bfea 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -176,6 +176,11 @@ int arch_hibernation_header_restore(void *addr)
}
EXPORT_SYMBOL(arch_hibernation_header_restore);
+static void *hibernate_page_alloc(void *arg)
+{
+ return (void *)get_safe_page((gfp_t)(unsigned long)arg);
+}
+
/*
* Copies length bytes, starting at src_start into an new page,
* perform cache maintenance, then maps it at the specified address low
@@ -192,6 +197,11 @@ static int create_safe_exec_page(void *src_start, size_t length,
unsigned long dst_addr,
phys_addr_t *phys_dst_addr)
{
+ struct trans_pgd_info trans_info = {
+ .trans_alloc_page = hibernate_page_alloc,
+ .trans_alloc_arg = (void *)GFP_ATOMIC,
+ };
+
void *page = (void *)get_safe_page(GFP_ATOMIC);
pgd_t *trans_pgd;
int rc;
@@ -206,7 +216,7 @@ static int create_safe_exec_page(void *src_start, size_t length,
if (!trans_pgd)
return -ENOMEM;
- rc = trans_pgd_map_page(trans_pgd, page, dst_addr,
+ rc = trans_pgd_map_page(&trans_info, trans_pgd, page, dst_addr,
PAGE_KERNEL_EXEC);
if (rc)
return rc;
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
index e048d1f5c912..f28eceba2242 100644
--- a/arch/arm64/mm/trans_pgd.c
+++ b/arch/arm64/mm/trans_pgd.c
@@ -25,6 +25,11 @@
#include <linux/mm.h>
#include <linux/mmzone.h>
+static void *trans_alloc(struct trans_pgd_info *info)
+{
+ return info->trans_alloc_page(info->trans_alloc_arg);
+}
+
static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr)
{
pte_t pte = READ_ONCE(*src_ptep);
@@ -201,9 +206,18 @@ int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
return rc;
}
-int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
- unsigned long dst_addr,
- pgprot_t pgprot)
+/*
+ * Add map entry to trans_pgd for a base-size page at PTE level.
+ * info: contains allocator and its argument
+ * trans_pgd: page table in which new map is added.
+ * page: page to be mapped.
+ * dst_addr: new VA address for the page
+ * pgprot: protection for the page.
+ *
+ * Returns 0 on success, and -ENOMEM on failure.
+ */
+int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
+ void *page, unsigned long dst_addr, pgprot_t pgprot)
{
pgd_t *pgdp;
p4d_t *p4dp;
@@ -213,7 +227,7 @@ int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
pgdp = pgd_offset_pgd(trans_pgd, dst_addr);
if (pgd_none(READ_ONCE(*pgdp))) {
- p4dp = (void *)get_safe_page(GFP_ATOMIC);
+ p4dp = trans_alloc(info);
if (!pgdp)
return -ENOMEM;
pgd_populate(&init_mm, pgdp, p4dp);
@@ -221,7 +235,7 @@ int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
p4dp = p4d_offset(pgdp, dst_addr);
if (p4d_none(READ_ONCE(*p4dp))) {
- pudp = (void *)get_safe_page(GFP_ATOMIC);
+ pudp = trans_alloc(info);
if (!pudp)
return -ENOMEM;
p4d_populate(&init_mm, p4dp, pudp);
@@ -229,7 +243,7 @@ int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
pudp = pud_offset(p4dp, dst_addr);
if (pud_none(READ_ONCE(*pudp))) {
- pmdp = (void *)get_safe_page(GFP_ATOMIC);
+ pmdp = trans_alloc(info);
if (!pmdp)
return -ENOMEM;
pud_populate(&init_mm, pudp, pmdp);
@@ -237,14 +251,14 @@ int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
pmdp = pmd_offset(pudp, dst_addr);
if (pmd_none(READ_ONCE(*pmdp))) {
- ptep = (void *)get_safe_page(GFP_ATOMIC);
+ ptep = trans_alloc(info);
if (!ptep)
return -ENOMEM;
pmd_populate_kernel(&init_mm, pmdp, ptep);
}
ptep = pte_offset_kernel(pmdp, dst_addr);
- set_pte(ptep, pfn_pte(virt_to_pfn(page), PAGE_KERNEL_EXEC));
+ set_pte(ptep, pfn_pte(virt_to_pfn(page), pgprot));
return 0;
}
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 04/18] arm64: trans_pgd: make trans_pgd_map_page generic
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
kexec is going to use a different allocator, so make
trans_pgd_map_page to accept allocator as an argument, and also
kexec is going to use a different map protection, so also pass
it via argument.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: Matthias Brugger <mbrugger@suse.com>
---
arch/arm64/include/asm/trans_pgd.h | 19 +++++++++++++++++--
arch/arm64/kernel/hibernate.c | 12 +++++++++++-
arch/arm64/mm/trans_pgd.c | 30 ++++++++++++++++++++++--------
3 files changed, 50 insertions(+), 11 deletions(-)
diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/trans_pgd.h
index 23153c13d1ce..b46409b25234 100644
--- a/arch/arm64/include/asm/trans_pgd.h
+++ b/arch/arm64/include/asm/trans_pgd.h
@@ -12,10 +12,25 @@
#include <linux/types.h>
#include <asm/pgtable-types.h>
+/*
+ * trans_alloc_page
+ * - Allocator that should return exactly one zeroed page, if this
+ * allocator fails, trans_pgd_create_copy() and trans_pgd_map_page()
+ * return -ENOMEM error.
+ *
+ * trans_alloc_arg
+ * - Passed to trans_alloc_page as an argument
+ */
+
+struct trans_pgd_info {
+ void * (*trans_alloc_page)(void *arg);
+ void *trans_alloc_arg;
+};
+
int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
unsigned long end);
-int trans_pgd_map_page(pgd_t *trans_pgd, void *page, unsigned long dst_addr,
- pgprot_t pgprot);
+int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
+ void *page, unsigned long dst_addr, pgprot_t pgprot);
#endif /* _ASM_TRANS_TABLE_H */
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 4a38662f0d90..c173f280bfea 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -176,6 +176,11 @@ int arch_hibernation_header_restore(void *addr)
}
EXPORT_SYMBOL(arch_hibernation_header_restore);
+static void *hibernate_page_alloc(void *arg)
+{
+ return (void *)get_safe_page((gfp_t)(unsigned long)arg);
+}
+
/*
* Copies length bytes, starting at src_start into an new page,
* perform cache maintenance, then maps it at the specified address low
@@ -192,6 +197,11 @@ static int create_safe_exec_page(void *src_start, size_t length,
unsigned long dst_addr,
phys_addr_t *phys_dst_addr)
{
+ struct trans_pgd_info trans_info = {
+ .trans_alloc_page = hibernate_page_alloc,
+ .trans_alloc_arg = (void *)GFP_ATOMIC,
+ };
+
void *page = (void *)get_safe_page(GFP_ATOMIC);
pgd_t *trans_pgd;
int rc;
@@ -206,7 +216,7 @@ static int create_safe_exec_page(void *src_start, size_t length,
if (!trans_pgd)
return -ENOMEM;
- rc = trans_pgd_map_page(trans_pgd, page, dst_addr,
+ rc = trans_pgd_map_page(&trans_info, trans_pgd, page, dst_addr,
PAGE_KERNEL_EXEC);
if (rc)
return rc;
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
index e048d1f5c912..f28eceba2242 100644
--- a/arch/arm64/mm/trans_pgd.c
+++ b/arch/arm64/mm/trans_pgd.c
@@ -25,6 +25,11 @@
#include <linux/mm.h>
#include <linux/mmzone.h>
+static void *trans_alloc(struct trans_pgd_info *info)
+{
+ return info->trans_alloc_page(info->trans_alloc_arg);
+}
+
static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr)
{
pte_t pte = READ_ONCE(*src_ptep);
@@ -201,9 +206,18 @@ int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
return rc;
}
-int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
- unsigned long dst_addr,
- pgprot_t pgprot)
+/*
+ * Add map entry to trans_pgd for a base-size page at PTE level.
+ * info: contains allocator and its argument
+ * trans_pgd: page table in which new map is added.
+ * page: page to be mapped.
+ * dst_addr: new VA address for the page
+ * pgprot: protection for the page.
+ *
+ * Returns 0 on success, and -ENOMEM on failure.
+ */
+int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
+ void *page, unsigned long dst_addr, pgprot_t pgprot)
{
pgd_t *pgdp;
p4d_t *p4dp;
@@ -213,7 +227,7 @@ int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
pgdp = pgd_offset_pgd(trans_pgd, dst_addr);
if (pgd_none(READ_ONCE(*pgdp))) {
- p4dp = (void *)get_safe_page(GFP_ATOMIC);
+ p4dp = trans_alloc(info);
if (!pgdp)
return -ENOMEM;
pgd_populate(&init_mm, pgdp, p4dp);
@@ -221,7 +235,7 @@ int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
p4dp = p4d_offset(pgdp, dst_addr);
if (p4d_none(READ_ONCE(*p4dp))) {
- pudp = (void *)get_safe_page(GFP_ATOMIC);
+ pudp = trans_alloc(info);
if (!pudp)
return -ENOMEM;
p4d_populate(&init_mm, p4dp, pudp);
@@ -229,7 +243,7 @@ int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
pudp = pud_offset(p4dp, dst_addr);
if (pud_none(READ_ONCE(*pudp))) {
- pmdp = (void *)get_safe_page(GFP_ATOMIC);
+ pmdp = trans_alloc(info);
if (!pmdp)
return -ENOMEM;
pud_populate(&init_mm, pudp, pmdp);
@@ -237,14 +251,14 @@ int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
pmdp = pmd_offset(pudp, dst_addr);
if (pmd_none(READ_ONCE(*pmdp))) {
- ptep = (void *)get_safe_page(GFP_ATOMIC);
+ ptep = trans_alloc(info);
if (!ptep)
return -ENOMEM;
pmd_populate_kernel(&init_mm, pmdp, ptep);
}
ptep = pte_offset_kernel(pmdp, dst_addr);
- set_pte(ptep, pfn_pte(virt_to_pfn(page), PAGE_KERNEL_EXEC));
+ set_pte(ptep, pfn_pte(virt_to_pfn(page), pgprot));
return 0;
}
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 04/18] arm64: trans_pgd: make trans_pgd_map_page generic
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
kexec is going to use a different allocator, so make
trans_pgd_map_page to accept allocator as an argument, and also
kexec is going to use a different map protection, so also pass
it via argument.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: Matthias Brugger <mbrugger@suse.com>
---
arch/arm64/include/asm/trans_pgd.h | 19 +++++++++++++++++--
arch/arm64/kernel/hibernate.c | 12 +++++++++++-
arch/arm64/mm/trans_pgd.c | 30 ++++++++++++++++++++++--------
3 files changed, 50 insertions(+), 11 deletions(-)
diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/trans_pgd.h
index 23153c13d1ce..b46409b25234 100644
--- a/arch/arm64/include/asm/trans_pgd.h
+++ b/arch/arm64/include/asm/trans_pgd.h
@@ -12,10 +12,25 @@
#include <linux/types.h>
#include <asm/pgtable-types.h>
+/*
+ * trans_alloc_page
+ * - Allocator that should return exactly one zeroed page, if this
+ * allocator fails, trans_pgd_create_copy() and trans_pgd_map_page()
+ * return -ENOMEM error.
+ *
+ * trans_alloc_arg
+ * - Passed to trans_alloc_page as an argument
+ */
+
+struct trans_pgd_info {
+ void * (*trans_alloc_page)(void *arg);
+ void *trans_alloc_arg;
+};
+
int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
unsigned long end);
-int trans_pgd_map_page(pgd_t *trans_pgd, void *page, unsigned long dst_addr,
- pgprot_t pgprot);
+int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
+ void *page, unsigned long dst_addr, pgprot_t pgprot);
#endif /* _ASM_TRANS_TABLE_H */
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 4a38662f0d90..c173f280bfea 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -176,6 +176,11 @@ int arch_hibernation_header_restore(void *addr)
}
EXPORT_SYMBOL(arch_hibernation_header_restore);
+static void *hibernate_page_alloc(void *arg)
+{
+ return (void *)get_safe_page((gfp_t)(unsigned long)arg);
+}
+
/*
* Copies length bytes, starting at src_start into an new page,
* perform cache maintenance, then maps it at the specified address low
@@ -192,6 +197,11 @@ static int create_safe_exec_page(void *src_start, size_t length,
unsigned long dst_addr,
phys_addr_t *phys_dst_addr)
{
+ struct trans_pgd_info trans_info = {
+ .trans_alloc_page = hibernate_page_alloc,
+ .trans_alloc_arg = (void *)GFP_ATOMIC,
+ };
+
void *page = (void *)get_safe_page(GFP_ATOMIC);
pgd_t *trans_pgd;
int rc;
@@ -206,7 +216,7 @@ static int create_safe_exec_page(void *src_start, size_t length,
if (!trans_pgd)
return -ENOMEM;
- rc = trans_pgd_map_page(trans_pgd, page, dst_addr,
+ rc = trans_pgd_map_page(&trans_info, trans_pgd, page, dst_addr,
PAGE_KERNEL_EXEC);
if (rc)
return rc;
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
index e048d1f5c912..f28eceba2242 100644
--- a/arch/arm64/mm/trans_pgd.c
+++ b/arch/arm64/mm/trans_pgd.c
@@ -25,6 +25,11 @@
#include <linux/mm.h>
#include <linux/mmzone.h>
+static void *trans_alloc(struct trans_pgd_info *info)
+{
+ return info->trans_alloc_page(info->trans_alloc_arg);
+}
+
static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr)
{
pte_t pte = READ_ONCE(*src_ptep);
@@ -201,9 +206,18 @@ int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
return rc;
}
-int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
- unsigned long dst_addr,
- pgprot_t pgprot)
+/*
+ * Add map entry to trans_pgd for a base-size page at PTE level.
+ * info: contains allocator and its argument
+ * trans_pgd: page table in which new map is added.
+ * page: page to be mapped.
+ * dst_addr: new VA address for the page
+ * pgprot: protection for the page.
+ *
+ * Returns 0 on success, and -ENOMEM on failure.
+ */
+int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
+ void *page, unsigned long dst_addr, pgprot_t pgprot)
{
pgd_t *pgdp;
p4d_t *p4dp;
@@ -213,7 +227,7 @@ int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
pgdp = pgd_offset_pgd(trans_pgd, dst_addr);
if (pgd_none(READ_ONCE(*pgdp))) {
- p4dp = (void *)get_safe_page(GFP_ATOMIC);
+ p4dp = trans_alloc(info);
if (!pgdp)
return -ENOMEM;
pgd_populate(&init_mm, pgdp, p4dp);
@@ -221,7 +235,7 @@ int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
p4dp = p4d_offset(pgdp, dst_addr);
if (p4d_none(READ_ONCE(*p4dp))) {
- pudp = (void *)get_safe_page(GFP_ATOMIC);
+ pudp = trans_alloc(info);
if (!pudp)
return -ENOMEM;
p4d_populate(&init_mm, p4dp, pudp);
@@ -229,7 +243,7 @@ int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
pudp = pud_offset(p4dp, dst_addr);
if (pud_none(READ_ONCE(*pudp))) {
- pmdp = (void *)get_safe_page(GFP_ATOMIC);
+ pmdp = trans_alloc(info);
if (!pmdp)
return -ENOMEM;
pud_populate(&init_mm, pudp, pmdp);
@@ -237,14 +251,14 @@ int trans_pgd_map_page(pgd_t *trans_pgd, void *page,
pmdp = pmd_offset(pudp, dst_addr);
if (pmd_none(READ_ONCE(*pmdp))) {
- ptep = (void *)get_safe_page(GFP_ATOMIC);
+ ptep = trans_alloc(info);
if (!ptep)
return -ENOMEM;
pmd_populate_kernel(&init_mm, pmdp, ptep);
}
ptep = pte_offset_kernel(pmdp, dst_addr);
- set_pte(ptep, pfn_pte(virt_to_pfn(page), PAGE_KERNEL_EXEC));
+ set_pte(ptep, pfn_pte(virt_to_pfn(page), pgprot));
return 0;
}
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 05/18] arm64: trans_pgd: pass allocator trans_pgd_create_copy
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-25 19:19 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Make trans_pgd_create_copy and its subroutines to use allocator that is
passed as an argument
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: James Morse <james.morse@arm.com>
---
arch/arm64/include/asm/trans_pgd.h | 4 +--
arch/arm64/kernel/hibernate.c | 7 ++++-
arch/arm64/mm/trans_pgd.c | 49 ++++++++++++++++++------------
3 files changed, 38 insertions(+), 22 deletions(-)
diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/trans_pgd.h
index b46409b25234..7fbf6a3ccff7 100644
--- a/arch/arm64/include/asm/trans_pgd.h
+++ b/arch/arm64/include/asm/trans_pgd.h
@@ -27,8 +27,8 @@ struct trans_pgd_info {
void *trans_alloc_arg;
};
-int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
- unsigned long end);
+int trans_pgd_create_copy(struct trans_pgd_info *info, pgd_t **trans_pgd,
+ unsigned long start, unsigned long end);
int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
void *page, unsigned long dst_addr, pgprot_t pgprot);
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index c173f280bfea..94fc275cdd21 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -437,13 +437,18 @@ int swsusp_arch_resume(void)
phys_addr_t phys_hibernate_exit;
void __noreturn (*hibernate_exit)(phys_addr_t, phys_addr_t, void *,
void *, phys_addr_t, phys_addr_t);
+ struct trans_pgd_info trans_info = {
+ .trans_alloc_page = hibernate_page_alloc,
+ .trans_alloc_arg = (void *)GFP_ATOMIC,
+ };
/*
* Restoring the memory image will overwrite the ttbr1 page tables.
* Create a second copy of just the linear map, and use this when
* restoring.
*/
- rc = trans_pgd_create_copy(&tmp_pg_dir, PAGE_OFFSET, PAGE_END);
+ rc = trans_pgd_create_copy(&trans_info, &tmp_pg_dir, PAGE_OFFSET,
+ PAGE_END);
if (rc)
return rc;
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
index f28eceba2242..47b6b7029907 100644
--- a/arch/arm64/mm/trans_pgd.c
+++ b/arch/arm64/mm/trans_pgd.c
@@ -57,14 +57,14 @@ static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr)
}
}
-static int copy_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long start,
- unsigned long end)
+static int copy_pte(struct trans_pgd_info *info, pmd_t *dst_pmdp,
+ pmd_t *src_pmdp, unsigned long start, unsigned long end)
{
pte_t *src_ptep;
pte_t *dst_ptep;
unsigned long addr = start;
- dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC);
+ dst_ptep = trans_alloc(info);
if (!dst_ptep)
return -ENOMEM;
pmd_populate_kernel(&init_mm, dst_pmdp, dst_ptep);
@@ -78,8 +78,8 @@ static int copy_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long start,
return 0;
}
-static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
- unsigned long end)
+static int copy_pmd(struct trans_pgd_info *info, pud_t *dst_pudp,
+ pud_t *src_pudp, unsigned long start, unsigned long end)
{
pmd_t *src_pmdp;
pmd_t *dst_pmdp;
@@ -87,7 +87,7 @@ static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
unsigned long addr = start;
if (pud_none(READ_ONCE(*dst_pudp))) {
- dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC);
+ dst_pmdp = trans_alloc(info);
if (!dst_pmdp)
return -ENOMEM;
pud_populate(&init_mm, dst_pudp, dst_pmdp);
@@ -102,7 +102,7 @@ static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
if (pmd_none(pmd))
continue;
if (pmd_table(pmd)) {
- if (copy_pte(dst_pmdp, src_pmdp, addr, next))
+ if (copy_pte(info, dst_pmdp, src_pmdp, addr, next))
return -ENOMEM;
} else {
set_pmd(dst_pmdp,
@@ -113,7 +113,8 @@ static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
return 0;
}
-static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
+static int copy_pud(struct trans_pgd_info *info, p4d_t *dst_p4dp,
+ p4d_t *src_p4dp, unsigned long start,
unsigned long end)
{
pud_t *dst_pudp;
@@ -122,7 +123,7 @@ static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
unsigned long addr = start;
if (p4d_none(READ_ONCE(*dst_p4dp))) {
- dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC);
+ dst_pudp = trans_alloc(info);
if (!dst_pudp)
return -ENOMEM;
p4d_populate(&init_mm, dst_p4dp, dst_pudp);
@@ -137,7 +138,7 @@ static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
if (pud_none(pud))
continue;
if (pud_table(pud)) {
- if (copy_pmd(dst_pudp, src_pudp, addr, next))
+ if (copy_pmd(info, dst_pudp, src_pudp, addr, next))
return -ENOMEM;
} else {
set_pud(dst_pudp,
@@ -148,7 +149,8 @@ static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
return 0;
}
-static int copy_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long start,
+static int copy_p4d(struct trans_pgd_info *info, pgd_t *dst_pgdp,
+ pgd_t *src_pgdp, unsigned long start,
unsigned long end)
{
p4d_t *dst_p4dp;
@@ -162,15 +164,15 @@ static int copy_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long start,
next = p4d_addr_end(addr, end);
if (p4d_none(READ_ONCE(*src_p4dp)))
continue;
- if (copy_pud(dst_p4dp, src_p4dp, addr, next))
+ if (copy_pud(info, dst_p4dp, src_p4dp, addr, next))
return -ENOMEM;
} while (dst_p4dp++, src_p4dp++, addr = next, addr != end);
return 0;
}
-static int copy_page_tables(pgd_t *dst_pgdp, unsigned long start,
- unsigned long end)
+static int copy_page_tables(struct trans_pgd_info *info, pgd_t *dst_pgdp,
+ unsigned long start, unsigned long end)
{
unsigned long next;
unsigned long addr = start;
@@ -181,25 +183,34 @@ static int copy_page_tables(pgd_t *dst_pgdp, unsigned long start,
next = pgd_addr_end(addr, end);
if (pgd_none(READ_ONCE(*src_pgdp)))
continue;
- if (copy_p4d(dst_pgdp, src_pgdp, addr, next))
+ if (copy_p4d(info, dst_pgdp, src_pgdp, addr, next))
return -ENOMEM;
} while (dst_pgdp++, src_pgdp++, addr = next, addr != end);
return 0;
}
-int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
- unsigned long end)
+/*
+ * Create trans_pgd and copy linear map.
+ * info: contains allocator and its argument
+ * dst_pgdp: new page table that is created, and to which map is copied.
+ * start: Start of the interval (inclusive).
+ * end: End of the interval (exclusive).
+ *
+ * Returns 0 on success, and -ENOMEM on failure.
+ */
+int trans_pgd_create_copy(struct trans_pgd_info *info, pgd_t **dst_pgdp,
+ unsigned long start, unsigned long end)
{
int rc;
- pgd_t *trans_pgd = (pgd_t *)get_safe_page(GFP_ATOMIC);
+ pgd_t *trans_pgd = trans_alloc(info);
if (!trans_pgd) {
pr_err("Failed to allocate memory for temporary page tables.\n");
return -ENOMEM;
}
- rc = copy_page_tables(trans_pgd, start, end);
+ rc = copy_page_tables(info, trans_pgd, start, end);
if (!rc)
*dst_pgdp = trans_pgd;
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 05/18] arm64: trans_pgd: pass allocator trans_pgd_create_copy
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Make trans_pgd_create_copy and its subroutines to use allocator that is
passed as an argument
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: James Morse <james.morse@arm.com>
---
arch/arm64/include/asm/trans_pgd.h | 4 +--
arch/arm64/kernel/hibernate.c | 7 ++++-
arch/arm64/mm/trans_pgd.c | 49 ++++++++++++++++++------------
3 files changed, 38 insertions(+), 22 deletions(-)
diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/trans_pgd.h
index b46409b25234..7fbf6a3ccff7 100644
--- a/arch/arm64/include/asm/trans_pgd.h
+++ b/arch/arm64/include/asm/trans_pgd.h
@@ -27,8 +27,8 @@ struct trans_pgd_info {
void *trans_alloc_arg;
};
-int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
- unsigned long end);
+int trans_pgd_create_copy(struct trans_pgd_info *info, pgd_t **trans_pgd,
+ unsigned long start, unsigned long end);
int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
void *page, unsigned long dst_addr, pgprot_t pgprot);
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index c173f280bfea..94fc275cdd21 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -437,13 +437,18 @@ int swsusp_arch_resume(void)
phys_addr_t phys_hibernate_exit;
void __noreturn (*hibernate_exit)(phys_addr_t, phys_addr_t, void *,
void *, phys_addr_t, phys_addr_t);
+ struct trans_pgd_info trans_info = {
+ .trans_alloc_page = hibernate_page_alloc,
+ .trans_alloc_arg = (void *)GFP_ATOMIC,
+ };
/*
* Restoring the memory image will overwrite the ttbr1 page tables.
* Create a second copy of just the linear map, and use this when
* restoring.
*/
- rc = trans_pgd_create_copy(&tmp_pg_dir, PAGE_OFFSET, PAGE_END);
+ rc = trans_pgd_create_copy(&trans_info, &tmp_pg_dir, PAGE_OFFSET,
+ PAGE_END);
if (rc)
return rc;
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
index f28eceba2242..47b6b7029907 100644
--- a/arch/arm64/mm/trans_pgd.c
+++ b/arch/arm64/mm/trans_pgd.c
@@ -57,14 +57,14 @@ static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr)
}
}
-static int copy_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long start,
- unsigned long end)
+static int copy_pte(struct trans_pgd_info *info, pmd_t *dst_pmdp,
+ pmd_t *src_pmdp, unsigned long start, unsigned long end)
{
pte_t *src_ptep;
pte_t *dst_ptep;
unsigned long addr = start;
- dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC);
+ dst_ptep = trans_alloc(info);
if (!dst_ptep)
return -ENOMEM;
pmd_populate_kernel(&init_mm, dst_pmdp, dst_ptep);
@@ -78,8 +78,8 @@ static int copy_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long start,
return 0;
}
-static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
- unsigned long end)
+static int copy_pmd(struct trans_pgd_info *info, pud_t *dst_pudp,
+ pud_t *src_pudp, unsigned long start, unsigned long end)
{
pmd_t *src_pmdp;
pmd_t *dst_pmdp;
@@ -87,7 +87,7 @@ static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
unsigned long addr = start;
if (pud_none(READ_ONCE(*dst_pudp))) {
- dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC);
+ dst_pmdp = trans_alloc(info);
if (!dst_pmdp)
return -ENOMEM;
pud_populate(&init_mm, dst_pudp, dst_pmdp);
@@ -102,7 +102,7 @@ static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
if (pmd_none(pmd))
continue;
if (pmd_table(pmd)) {
- if (copy_pte(dst_pmdp, src_pmdp, addr, next))
+ if (copy_pte(info, dst_pmdp, src_pmdp, addr, next))
return -ENOMEM;
} else {
set_pmd(dst_pmdp,
@@ -113,7 +113,8 @@ static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
return 0;
}
-static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
+static int copy_pud(struct trans_pgd_info *info, p4d_t *dst_p4dp,
+ p4d_t *src_p4dp, unsigned long start,
unsigned long end)
{
pud_t *dst_pudp;
@@ -122,7 +123,7 @@ static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
unsigned long addr = start;
if (p4d_none(READ_ONCE(*dst_p4dp))) {
- dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC);
+ dst_pudp = trans_alloc(info);
if (!dst_pudp)
return -ENOMEM;
p4d_populate(&init_mm, dst_p4dp, dst_pudp);
@@ -137,7 +138,7 @@ static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
if (pud_none(pud))
continue;
if (pud_table(pud)) {
- if (copy_pmd(dst_pudp, src_pudp, addr, next))
+ if (copy_pmd(info, dst_pudp, src_pudp, addr, next))
return -ENOMEM;
} else {
set_pud(dst_pudp,
@@ -148,7 +149,8 @@ static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
return 0;
}
-static int copy_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long start,
+static int copy_p4d(struct trans_pgd_info *info, pgd_t *dst_pgdp,
+ pgd_t *src_pgdp, unsigned long start,
unsigned long end)
{
p4d_t *dst_p4dp;
@@ -162,15 +164,15 @@ static int copy_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long start,
next = p4d_addr_end(addr, end);
if (p4d_none(READ_ONCE(*src_p4dp)))
continue;
- if (copy_pud(dst_p4dp, src_p4dp, addr, next))
+ if (copy_pud(info, dst_p4dp, src_p4dp, addr, next))
return -ENOMEM;
} while (dst_p4dp++, src_p4dp++, addr = next, addr != end);
return 0;
}
-static int copy_page_tables(pgd_t *dst_pgdp, unsigned long start,
- unsigned long end)
+static int copy_page_tables(struct trans_pgd_info *info, pgd_t *dst_pgdp,
+ unsigned long start, unsigned long end)
{
unsigned long next;
unsigned long addr = start;
@@ -181,25 +183,34 @@ static int copy_page_tables(pgd_t *dst_pgdp, unsigned long start,
next = pgd_addr_end(addr, end);
if (pgd_none(READ_ONCE(*src_pgdp)))
continue;
- if (copy_p4d(dst_pgdp, src_pgdp, addr, next))
+ if (copy_p4d(info, dst_pgdp, src_pgdp, addr, next))
return -ENOMEM;
} while (dst_pgdp++, src_pgdp++, addr = next, addr != end);
return 0;
}
-int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
- unsigned long end)
+/*
+ * Create trans_pgd and copy linear map.
+ * info: contains allocator and its argument
+ * dst_pgdp: new page table that is created, and to which map is copied.
+ * start: Start of the interval (inclusive).
+ * end: End of the interval (exclusive).
+ *
+ * Returns 0 on success, and -ENOMEM on failure.
+ */
+int trans_pgd_create_copy(struct trans_pgd_info *info, pgd_t **dst_pgdp,
+ unsigned long start, unsigned long end)
{
int rc;
- pgd_t *trans_pgd = (pgd_t *)get_safe_page(GFP_ATOMIC);
+ pgd_t *trans_pgd = trans_alloc(info);
if (!trans_pgd) {
pr_err("Failed to allocate memory for temporary page tables.\n");
return -ENOMEM;
}
- rc = copy_page_tables(trans_pgd, start, end);
+ rc = copy_page_tables(info, trans_pgd, start, end);
if (!rc)
*dst_pgdp = trans_pgd;
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 05/18] arm64: trans_pgd: pass allocator trans_pgd_create_copy
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Make trans_pgd_create_copy and its subroutines to use allocator that is
passed as an argument
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: James Morse <james.morse@arm.com>
---
arch/arm64/include/asm/trans_pgd.h | 4 +--
arch/arm64/kernel/hibernate.c | 7 ++++-
arch/arm64/mm/trans_pgd.c | 49 ++++++++++++++++++------------
3 files changed, 38 insertions(+), 22 deletions(-)
diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/trans_pgd.h
index b46409b25234..7fbf6a3ccff7 100644
--- a/arch/arm64/include/asm/trans_pgd.h
+++ b/arch/arm64/include/asm/trans_pgd.h
@@ -27,8 +27,8 @@ struct trans_pgd_info {
void *trans_alloc_arg;
};
-int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
- unsigned long end);
+int trans_pgd_create_copy(struct trans_pgd_info *info, pgd_t **trans_pgd,
+ unsigned long start, unsigned long end);
int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
void *page, unsigned long dst_addr, pgprot_t pgprot);
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index c173f280bfea..94fc275cdd21 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -437,13 +437,18 @@ int swsusp_arch_resume(void)
phys_addr_t phys_hibernate_exit;
void __noreturn (*hibernate_exit)(phys_addr_t, phys_addr_t, void *,
void *, phys_addr_t, phys_addr_t);
+ struct trans_pgd_info trans_info = {
+ .trans_alloc_page = hibernate_page_alloc,
+ .trans_alloc_arg = (void *)GFP_ATOMIC,
+ };
/*
* Restoring the memory image will overwrite the ttbr1 page tables.
* Create a second copy of just the linear map, and use this when
* restoring.
*/
- rc = trans_pgd_create_copy(&tmp_pg_dir, PAGE_OFFSET, PAGE_END);
+ rc = trans_pgd_create_copy(&trans_info, &tmp_pg_dir, PAGE_OFFSET,
+ PAGE_END);
if (rc)
return rc;
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
index f28eceba2242..47b6b7029907 100644
--- a/arch/arm64/mm/trans_pgd.c
+++ b/arch/arm64/mm/trans_pgd.c
@@ -57,14 +57,14 @@ static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr)
}
}
-static int copy_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long start,
- unsigned long end)
+static int copy_pte(struct trans_pgd_info *info, pmd_t *dst_pmdp,
+ pmd_t *src_pmdp, unsigned long start, unsigned long end)
{
pte_t *src_ptep;
pte_t *dst_ptep;
unsigned long addr = start;
- dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC);
+ dst_ptep = trans_alloc(info);
if (!dst_ptep)
return -ENOMEM;
pmd_populate_kernel(&init_mm, dst_pmdp, dst_ptep);
@@ -78,8 +78,8 @@ static int copy_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long start,
return 0;
}
-static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
- unsigned long end)
+static int copy_pmd(struct trans_pgd_info *info, pud_t *dst_pudp,
+ pud_t *src_pudp, unsigned long start, unsigned long end)
{
pmd_t *src_pmdp;
pmd_t *dst_pmdp;
@@ -87,7 +87,7 @@ static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
unsigned long addr = start;
if (pud_none(READ_ONCE(*dst_pudp))) {
- dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC);
+ dst_pmdp = trans_alloc(info);
if (!dst_pmdp)
return -ENOMEM;
pud_populate(&init_mm, dst_pudp, dst_pmdp);
@@ -102,7 +102,7 @@ static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
if (pmd_none(pmd))
continue;
if (pmd_table(pmd)) {
- if (copy_pte(dst_pmdp, src_pmdp, addr, next))
+ if (copy_pte(info, dst_pmdp, src_pmdp, addr, next))
return -ENOMEM;
} else {
set_pmd(dst_pmdp,
@@ -113,7 +113,8 @@ static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start,
return 0;
}
-static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
+static int copy_pud(struct trans_pgd_info *info, p4d_t *dst_p4dp,
+ p4d_t *src_p4dp, unsigned long start,
unsigned long end)
{
pud_t *dst_pudp;
@@ -122,7 +123,7 @@ static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
unsigned long addr = start;
if (p4d_none(READ_ONCE(*dst_p4dp))) {
- dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC);
+ dst_pudp = trans_alloc(info);
if (!dst_pudp)
return -ENOMEM;
p4d_populate(&init_mm, dst_p4dp, dst_pudp);
@@ -137,7 +138,7 @@ static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
if (pud_none(pud))
continue;
if (pud_table(pud)) {
- if (copy_pmd(dst_pudp, src_pudp, addr, next))
+ if (copy_pmd(info, dst_pudp, src_pudp, addr, next))
return -ENOMEM;
} else {
set_pud(dst_pudp,
@@ -148,7 +149,8 @@ static int copy_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long start,
return 0;
}
-static int copy_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long start,
+static int copy_p4d(struct trans_pgd_info *info, pgd_t *dst_pgdp,
+ pgd_t *src_pgdp, unsigned long start,
unsigned long end)
{
p4d_t *dst_p4dp;
@@ -162,15 +164,15 @@ static int copy_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long start,
next = p4d_addr_end(addr, end);
if (p4d_none(READ_ONCE(*src_p4dp)))
continue;
- if (copy_pud(dst_p4dp, src_p4dp, addr, next))
+ if (copy_pud(info, dst_p4dp, src_p4dp, addr, next))
return -ENOMEM;
} while (dst_p4dp++, src_p4dp++, addr = next, addr != end);
return 0;
}
-static int copy_page_tables(pgd_t *dst_pgdp, unsigned long start,
- unsigned long end)
+static int copy_page_tables(struct trans_pgd_info *info, pgd_t *dst_pgdp,
+ unsigned long start, unsigned long end)
{
unsigned long next;
unsigned long addr = start;
@@ -181,25 +183,34 @@ static int copy_page_tables(pgd_t *dst_pgdp, unsigned long start,
next = pgd_addr_end(addr, end);
if (pgd_none(READ_ONCE(*src_pgdp)))
continue;
- if (copy_p4d(dst_pgdp, src_pgdp, addr, next))
+ if (copy_p4d(info, dst_pgdp, src_pgdp, addr, next))
return -ENOMEM;
} while (dst_pgdp++, src_pgdp++, addr = next, addr != end);
return 0;
}
-int trans_pgd_create_copy(pgd_t **dst_pgdp, unsigned long start,
- unsigned long end)
+/*
+ * Create trans_pgd and copy linear map.
+ * info: contains allocator and its argument
+ * dst_pgdp: new page table that is created, and to which map is copied.
+ * start: Start of the interval (inclusive).
+ * end: End of the interval (exclusive).
+ *
+ * Returns 0 on success, and -ENOMEM on failure.
+ */
+int trans_pgd_create_copy(struct trans_pgd_info *info, pgd_t **dst_pgdp,
+ unsigned long start, unsigned long end)
{
int rc;
- pgd_t *trans_pgd = (pgd_t *)get_safe_page(GFP_ATOMIC);
+ pgd_t *trans_pgd = trans_alloc(info);
if (!trans_pgd) {
pr_err("Failed to allocate memory for temporary page tables.\n");
return -ENOMEM;
}
- rc = copy_page_tables(trans_pgd, start, end);
+ rc = copy_page_tables(info, trans_pgd, start, end);
if (!rc)
*dst_pgdp = trans_pgd;
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 06/18] arm64: trans_pgd: pass NULL instead of init_mm to *_populate functions
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-25 19:19 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
trans_pgd_* should be independent from mm context because the tables that
are created by this code are used when there are no mm context around, as
it is between kernels. Simply replace mm_init's with NULL.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Acked-by: James Morse <james.morse@arm.com>
---
arch/arm64/mm/trans_pgd.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
index 47b6b7029907..ded8e2ba0308 100644
--- a/arch/arm64/mm/trans_pgd.c
+++ b/arch/arm64/mm/trans_pgd.c
@@ -67,7 +67,7 @@ static int copy_pte(struct trans_pgd_info *info, pmd_t *dst_pmdp,
dst_ptep = trans_alloc(info);
if (!dst_ptep)
return -ENOMEM;
- pmd_populate_kernel(&init_mm, dst_pmdp, dst_ptep);
+ pmd_populate_kernel(NULL, dst_pmdp, dst_ptep);
dst_ptep = pte_offset_kernel(dst_pmdp, start);
src_ptep = pte_offset_kernel(src_pmdp, start);
@@ -90,7 +90,7 @@ static int copy_pmd(struct trans_pgd_info *info, pud_t *dst_pudp,
dst_pmdp = trans_alloc(info);
if (!dst_pmdp)
return -ENOMEM;
- pud_populate(&init_mm, dst_pudp, dst_pmdp);
+ pud_populate(NULL, dst_pudp, dst_pmdp);
}
dst_pmdp = pmd_offset(dst_pudp, start);
@@ -126,7 +126,7 @@ static int copy_pud(struct trans_pgd_info *info, p4d_t *dst_p4dp,
dst_pudp = trans_alloc(info);
if (!dst_pudp)
return -ENOMEM;
- p4d_populate(&init_mm, dst_p4dp, dst_pudp);
+ p4d_populate(NULL, dst_p4dp, dst_pudp);
}
dst_pudp = pud_offset(dst_p4dp, start);
@@ -241,7 +241,7 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
p4dp = trans_alloc(info);
if (!pgdp)
return -ENOMEM;
- pgd_populate(&init_mm, pgdp, p4dp);
+ pgd_populate(NULL, pgdp, p4dp);
}
p4dp = p4d_offset(pgdp, dst_addr);
@@ -249,7 +249,7 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
pudp = trans_alloc(info);
if (!pudp)
return -ENOMEM;
- p4d_populate(&init_mm, p4dp, pudp);
+ p4d_populate(NULL, p4dp, pudp);
}
pudp = pud_offset(p4dp, dst_addr);
@@ -257,7 +257,7 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
pmdp = trans_alloc(info);
if (!pmdp)
return -ENOMEM;
- pud_populate(&init_mm, pudp, pmdp);
+ pud_populate(NULL, pudp, pmdp);
}
pmdp = pmd_offset(pudp, dst_addr);
@@ -265,7 +265,7 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
ptep = trans_alloc(info);
if (!ptep)
return -ENOMEM;
- pmd_populate_kernel(&init_mm, pmdp, ptep);
+ pmd_populate_kernel(NULL, pmdp, ptep);
}
ptep = pte_offset_kernel(pmdp, dst_addr);
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 06/18] arm64: trans_pgd: pass NULL instead of init_mm to *_populate functions
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
trans_pgd_* should be independent from mm context because the tables that
are created by this code are used when there are no mm context around, as
it is between kernels. Simply replace mm_init's with NULL.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Acked-by: James Morse <james.morse@arm.com>
---
arch/arm64/mm/trans_pgd.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
index 47b6b7029907..ded8e2ba0308 100644
--- a/arch/arm64/mm/trans_pgd.c
+++ b/arch/arm64/mm/trans_pgd.c
@@ -67,7 +67,7 @@ static int copy_pte(struct trans_pgd_info *info, pmd_t *dst_pmdp,
dst_ptep = trans_alloc(info);
if (!dst_ptep)
return -ENOMEM;
- pmd_populate_kernel(&init_mm, dst_pmdp, dst_ptep);
+ pmd_populate_kernel(NULL, dst_pmdp, dst_ptep);
dst_ptep = pte_offset_kernel(dst_pmdp, start);
src_ptep = pte_offset_kernel(src_pmdp, start);
@@ -90,7 +90,7 @@ static int copy_pmd(struct trans_pgd_info *info, pud_t *dst_pudp,
dst_pmdp = trans_alloc(info);
if (!dst_pmdp)
return -ENOMEM;
- pud_populate(&init_mm, dst_pudp, dst_pmdp);
+ pud_populate(NULL, dst_pudp, dst_pmdp);
}
dst_pmdp = pmd_offset(dst_pudp, start);
@@ -126,7 +126,7 @@ static int copy_pud(struct trans_pgd_info *info, p4d_t *dst_p4dp,
dst_pudp = trans_alloc(info);
if (!dst_pudp)
return -ENOMEM;
- p4d_populate(&init_mm, dst_p4dp, dst_pudp);
+ p4d_populate(NULL, dst_p4dp, dst_pudp);
}
dst_pudp = pud_offset(dst_p4dp, start);
@@ -241,7 +241,7 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
p4dp = trans_alloc(info);
if (!pgdp)
return -ENOMEM;
- pgd_populate(&init_mm, pgdp, p4dp);
+ pgd_populate(NULL, pgdp, p4dp);
}
p4dp = p4d_offset(pgdp, dst_addr);
@@ -249,7 +249,7 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
pudp = trans_alloc(info);
if (!pudp)
return -ENOMEM;
- p4d_populate(&init_mm, p4dp, pudp);
+ p4d_populate(NULL, p4dp, pudp);
}
pudp = pud_offset(p4dp, dst_addr);
@@ -257,7 +257,7 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
pmdp = trans_alloc(info);
if (!pmdp)
return -ENOMEM;
- pud_populate(&init_mm, pudp, pmdp);
+ pud_populate(NULL, pudp, pmdp);
}
pmdp = pmd_offset(pudp, dst_addr);
@@ -265,7 +265,7 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
ptep = trans_alloc(info);
if (!ptep)
return -ENOMEM;
- pmd_populate_kernel(&init_mm, pmdp, ptep);
+ pmd_populate_kernel(NULL, pmdp, ptep);
}
ptep = pte_offset_kernel(pmdp, dst_addr);
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 06/18] arm64: trans_pgd: pass NULL instead of init_mm to *_populate functions
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
trans_pgd_* should be independent from mm context because the tables that
are created by this code are used when there are no mm context around, as
it is between kernels. Simply replace mm_init's with NULL.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Acked-by: James Morse <james.morse@arm.com>
---
arch/arm64/mm/trans_pgd.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
index 47b6b7029907..ded8e2ba0308 100644
--- a/arch/arm64/mm/trans_pgd.c
+++ b/arch/arm64/mm/trans_pgd.c
@@ -67,7 +67,7 @@ static int copy_pte(struct trans_pgd_info *info, pmd_t *dst_pmdp,
dst_ptep = trans_alloc(info);
if (!dst_ptep)
return -ENOMEM;
- pmd_populate_kernel(&init_mm, dst_pmdp, dst_ptep);
+ pmd_populate_kernel(NULL, dst_pmdp, dst_ptep);
dst_ptep = pte_offset_kernel(dst_pmdp, start);
src_ptep = pte_offset_kernel(src_pmdp, start);
@@ -90,7 +90,7 @@ static int copy_pmd(struct trans_pgd_info *info, pud_t *dst_pudp,
dst_pmdp = trans_alloc(info);
if (!dst_pmdp)
return -ENOMEM;
- pud_populate(&init_mm, dst_pudp, dst_pmdp);
+ pud_populate(NULL, dst_pudp, dst_pmdp);
}
dst_pmdp = pmd_offset(dst_pudp, start);
@@ -126,7 +126,7 @@ static int copy_pud(struct trans_pgd_info *info, p4d_t *dst_p4dp,
dst_pudp = trans_alloc(info);
if (!dst_pudp)
return -ENOMEM;
- p4d_populate(&init_mm, dst_p4dp, dst_pudp);
+ p4d_populate(NULL, dst_p4dp, dst_pudp);
}
dst_pudp = pud_offset(dst_p4dp, start);
@@ -241,7 +241,7 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
p4dp = trans_alloc(info);
if (!pgdp)
return -ENOMEM;
- pgd_populate(&init_mm, pgdp, p4dp);
+ pgd_populate(NULL, pgdp, p4dp);
}
p4dp = p4d_offset(pgdp, dst_addr);
@@ -249,7 +249,7 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
pudp = trans_alloc(info);
if (!pudp)
return -ENOMEM;
- p4d_populate(&init_mm, p4dp, pudp);
+ p4d_populate(NULL, p4dp, pudp);
}
pudp = pud_offset(p4dp, dst_addr);
@@ -257,7 +257,7 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
pmdp = trans_alloc(info);
if (!pmdp)
return -ENOMEM;
- pud_populate(&init_mm, pudp, pmdp);
+ pud_populate(NULL, pudp, pmdp);
}
pmdp = pmd_offset(pudp, dst_addr);
@@ -265,7 +265,7 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
ptep = trans_alloc(info);
if (!ptep)
return -ENOMEM;
- pmd_populate_kernel(&init_mm, pmdp, ptep);
+ pmd_populate_kernel(NULL, pmdp, ptep);
}
ptep = pte_offset_kernel(pmdp, dst_addr);
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 07/18] arm64: mm: Always update TCR_EL1 from __cpu_set_tcr_t0sz()
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-25 19:19 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
From: James Morse <james.morse@arm.com>
Because only the idmap sets a non-standard T0SZ, __cpu_set_tcr_t0sz()
can check for platforms that need to do this using
__cpu_uses_extended_idmap() before doing its work.
The idmap is only built with enough levels, (and T0SZ bits) to map
its single page.
To allow hibernate, and then kexec to idmap their single page copy
routines, __cpu_set_tcr_t0sz() needs to consider additional users,
who may need a different number of levels/T0SZ-bits to the idmap.
(i.e. VA_BITS may be enough for the idmap, but not hibernate/kexec)
Always read TCR_EL1, and check whether any work needs doing for
this request. __cpu_uses_extended_idmap() remains as it is used
by KVM, whose idmap is also part of the kernel image.
This mostly affects the cpuidle path, where we now get an extra
system register read .
CC: Lorenzo Pieralisi <Lorenzo.Pieralisi@arm.com>
CC: Sudeep Holla <sudeep.holla@arm.com>
Signed-off-by: James Morse <james.morse@arm.com>
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/include/asm/mmu_context.h | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
index 0b3079fd28eb..70ce8c1d2b07 100644
--- a/arch/arm64/include/asm/mmu_context.h
+++ b/arch/arm64/include/asm/mmu_context.h
@@ -81,16 +81,15 @@ static inline bool __cpu_uses_extended_idmap_level(void)
}
/*
- * Set TCR.T0SZ to its default value (based on VA_BITS)
+ * Ensure TCR.T0SZ is set to the provided value.
*/
static inline void __cpu_set_tcr_t0sz(unsigned long t0sz)
{
- unsigned long tcr;
+ unsigned long tcr = read_sysreg(tcr_el1);
- if (!__cpu_uses_extended_idmap())
+ if ((tcr & TCR_T0SZ_MASK) >> TCR_T0SZ_OFFSET == t0sz)
return;
- tcr = read_sysreg(tcr_el1);
tcr &= ~TCR_T0SZ_MASK;
tcr |= t0sz << TCR_T0SZ_OFFSET;
write_sysreg(tcr, tcr_el1);
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 07/18] arm64: mm: Always update TCR_EL1 from __cpu_set_tcr_t0sz()
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
From: James Morse <james.morse@arm.com>
Because only the idmap sets a non-standard T0SZ, __cpu_set_tcr_t0sz()
can check for platforms that need to do this using
__cpu_uses_extended_idmap() before doing its work.
The idmap is only built with enough levels, (and T0SZ bits) to map
its single page.
To allow hibernate, and then kexec to idmap their single page copy
routines, __cpu_set_tcr_t0sz() needs to consider additional users,
who may need a different number of levels/T0SZ-bits to the idmap.
(i.e. VA_BITS may be enough for the idmap, but not hibernate/kexec)
Always read TCR_EL1, and check whether any work needs doing for
this request. __cpu_uses_extended_idmap() remains as it is used
by KVM, whose idmap is also part of the kernel image.
This mostly affects the cpuidle path, where we now get an extra
system register read .
CC: Lorenzo Pieralisi <Lorenzo.Pieralisi@arm.com>
CC: Sudeep Holla <sudeep.holla@arm.com>
Signed-off-by: James Morse <james.morse@arm.com>
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/include/asm/mmu_context.h | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
index 0b3079fd28eb..70ce8c1d2b07 100644
--- a/arch/arm64/include/asm/mmu_context.h
+++ b/arch/arm64/include/asm/mmu_context.h
@@ -81,16 +81,15 @@ static inline bool __cpu_uses_extended_idmap_level(void)
}
/*
- * Set TCR.T0SZ to its default value (based on VA_BITS)
+ * Ensure TCR.T0SZ is set to the provided value.
*/
static inline void __cpu_set_tcr_t0sz(unsigned long t0sz)
{
- unsigned long tcr;
+ unsigned long tcr = read_sysreg(tcr_el1);
- if (!__cpu_uses_extended_idmap())
+ if ((tcr & TCR_T0SZ_MASK) >> TCR_T0SZ_OFFSET == t0sz)
return;
- tcr = read_sysreg(tcr_el1);
tcr &= ~TCR_T0SZ_MASK;
tcr |= t0sz << TCR_T0SZ_OFFSET;
write_sysreg(tcr, tcr_el1);
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 07/18] arm64: mm: Always update TCR_EL1 from __cpu_set_tcr_t0sz()
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
From: James Morse <james.morse@arm.com>
Because only the idmap sets a non-standard T0SZ, __cpu_set_tcr_t0sz()
can check for platforms that need to do this using
__cpu_uses_extended_idmap() before doing its work.
The idmap is only built with enough levels, (and T0SZ bits) to map
its single page.
To allow hibernate, and then kexec to idmap their single page copy
routines, __cpu_set_tcr_t0sz() needs to consider additional users,
who may need a different number of levels/T0SZ-bits to the idmap.
(i.e. VA_BITS may be enough for the idmap, but not hibernate/kexec)
Always read TCR_EL1, and check whether any work needs doing for
this request. __cpu_uses_extended_idmap() remains as it is used
by KVM, whose idmap is also part of the kernel image.
This mostly affects the cpuidle path, where we now get an extra
system register read .
CC: Lorenzo Pieralisi <Lorenzo.Pieralisi@arm.com>
CC: Sudeep Holla <sudeep.holla@arm.com>
Signed-off-by: James Morse <james.morse@arm.com>
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/include/asm/mmu_context.h | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
index 0b3079fd28eb..70ce8c1d2b07 100644
--- a/arch/arm64/include/asm/mmu_context.h
+++ b/arch/arm64/include/asm/mmu_context.h
@@ -81,16 +81,15 @@ static inline bool __cpu_uses_extended_idmap_level(void)
}
/*
- * Set TCR.T0SZ to its default value (based on VA_BITS)
+ * Ensure TCR.T0SZ is set to the provided value.
*/
static inline void __cpu_set_tcr_t0sz(unsigned long t0sz)
{
- unsigned long tcr;
+ unsigned long tcr = read_sysreg(tcr_el1);
- if (!__cpu_uses_extended_idmap())
+ if ((tcr & TCR_T0SZ_MASK) >> TCR_T0SZ_OFFSET == t0sz)
return;
- tcr = read_sysreg(tcr_el1);
tcr &= ~TCR_T0SZ_MASK;
tcr |= t0sz << TCR_T0SZ_OFFSET;
write_sysreg(tcr, tcr_el1);
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 08/18] arm64: trans_pgd: hibernate: idmap the single page that holds the copy page routines
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-25 19:19 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
From: James Morse <james.morse@arm.com>
To resume from hibernate, the contents of memory are restored from
the swap image. This may overwrite any page, including the running
kernel and its page tables.
Hibernate copies the code it uses to do the restore into a single
page that it knows won't be overwritten, and maps it with page tables
built from pages that won't be overwritten.
Today the address it uses for this mapping is arbitrary, but to allow
kexec to reuse this code, it needs to be idmapped. To idmap the page
we must avoid the kernel helpers that have VA_BITS baked in.
Convert create_single_mapping() to take a single PA, and idmap it.
The page tables are built in the reverse order to normal using
pfn_pte() to stir in any bits between 52:48. T0SZ is always increased
to cover 48bits, or 52 if the copy code has bits 52:48 in its PA.
Signed-off-by: James Morse <james.morse@arm.com>
[Adopted the original patch from James to trans_pgd interface, so it can be
commonly used by both Kexec and Hibernate. Some minor clean-ups.]
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Link: https://lore.kernel.org/linux-arm-kernel/20200115143322.214247-4-james.morse@arm.com/
---
arch/arm64/include/asm/trans_pgd.h | 3 ++
arch/arm64/kernel/hibernate.c | 32 +++++++------------
arch/arm64/mm/trans_pgd.c | 49 ++++++++++++++++++++++++++++++
3 files changed, 63 insertions(+), 21 deletions(-)
diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/trans_pgd.h
index 7fbf6a3ccff7..5d08e5adf3d5 100644
--- a/arch/arm64/include/asm/trans_pgd.h
+++ b/arch/arm64/include/asm/trans_pgd.h
@@ -33,4 +33,7 @@ int trans_pgd_create_copy(struct trans_pgd_info *info, pgd_t **trans_pgd,
int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
void *page, unsigned long dst_addr, pgprot_t pgprot);
+int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans_ttbr0,
+ unsigned long *t0sz, void *page);
+
#endif /* _ASM_TRANS_TABLE_H */
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 94fc275cdd21..9df32ba0d574 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -194,7 +194,6 @@ static void *hibernate_page_alloc(void *arg)
* page system.
*/
static int create_safe_exec_page(void *src_start, size_t length,
- unsigned long dst_addr,
phys_addr_t *phys_dst_addr)
{
struct trans_pgd_info trans_info = {
@@ -203,7 +202,8 @@ static int create_safe_exec_page(void *src_start, size_t length,
};
void *page = (void *)get_safe_page(GFP_ATOMIC);
- pgd_t *trans_pgd;
+ phys_addr_t trans_ttbr0;
+ unsigned long t0sz;
int rc;
if (!page)
@@ -211,13 +211,7 @@ static int create_safe_exec_page(void *src_start, size_t length,
memcpy(page, src_start, length);
__flush_icache_range((unsigned long)page, (unsigned long)page + length);
-
- trans_pgd = (void *)get_safe_page(GFP_ATOMIC);
- if (!trans_pgd)
- return -ENOMEM;
-
- rc = trans_pgd_map_page(&trans_info, trans_pgd, page, dst_addr,
- PAGE_KERNEL_EXEC);
+ rc = trans_pgd_idmap_page(&trans_info, &trans_ttbr0, &t0sz, page);
if (rc)
return rc;
@@ -230,12 +224,15 @@ static int create_safe_exec_page(void *src_start, size_t length,
* page, but TLBs may contain stale ASID-tagged entries (e.g. for EFI
* runtime services), while for a userspace-driven test_resume cycle it
* points to userspace page tables (and we must point it at a zero page
- * ourselves). Elsewhere we only (un)install the idmap with preemption
- * disabled, so T0SZ should be as required regardless.
+ * ourselves).
+ *
+ * We change T0SZ as part of installing the idmap. This is undone by
+ * cpu_uninstall_idmap() in __cpu_suspend_exit().
*/
cpu_set_reserved_ttbr0();
local_flush_tlb_all();
- write_sysreg(phys_to_ttbr(virt_to_phys(trans_pgd)), ttbr0_el1);
+ __cpu_set_tcr_t0sz(t0sz);
+ write_sysreg(trans_ttbr0, ttbr0_el1);
isb();
*phys_dst_addr = virt_to_phys(page);
@@ -434,7 +431,6 @@ int swsusp_arch_resume(void)
void *zero_page;
size_t exit_size;
pgd_t *tmp_pg_dir;
- phys_addr_t phys_hibernate_exit;
void __noreturn (*hibernate_exit)(phys_addr_t, phys_addr_t, void *,
void *, phys_addr_t, phys_addr_t);
struct trans_pgd_info trans_info = {
@@ -462,19 +458,13 @@ int swsusp_arch_resume(void)
return -ENOMEM;
}
- /*
- * Locate the exit code in the bottom-but-one page, so that *NULL
- * still has disastrous affects.
- */
- hibernate_exit = (void *)PAGE_SIZE;
exit_size = __hibernate_exit_text_end - __hibernate_exit_text_start;
/*
* Copy swsusp_arch_suspend_exit() to a safe page. This will generate
* a new set of ttbr0 page tables and load them.
*/
rc = create_safe_exec_page(__hibernate_exit_text_start, exit_size,
- (unsigned long)hibernate_exit,
- &phys_hibernate_exit);
+ (phys_addr_t *)&hibernate_exit);
if (rc) {
pr_err("Failed to create safe executable page for hibernate_exit code.\n");
return rc;
@@ -493,7 +483,7 @@ int swsusp_arch_resume(void)
* We can skip this step if we booted at EL1, or are running with VHE.
*/
if (el2_reset_needed()) {
- phys_addr_t el2_vectors = phys_hibernate_exit; /* base */
+ phys_addr_t el2_vectors = (phys_addr_t)hibernate_exit;
el2_vectors += hibernate_el2_vectors -
__hibernate_exit_text_start; /* offset */
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
index ded8e2ba0308..527f0a39c3da 100644
--- a/arch/arm64/mm/trans_pgd.c
+++ b/arch/arm64/mm/trans_pgd.c
@@ -273,3 +273,52 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
return 0;
}
+
+/*
+ * The page we want to idmap may be outside the range covered by VA_BITS that
+ * can be built using the kernel's p?d_populate() helpers. As a one off, for a
+ * single page, we build these page tables bottom up and just assume that will
+ * need the maximum T0SZ.
+ *
+ * Returns 0 on success, and -ENOMEM on failure.
+ * On success trans_ttbr0 contains page table with idmapped page, t0sz is set to
+ * maximum T0SZ for this page.
+ */
+int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans_ttbr0,
+ unsigned long *t0sz, void *page)
+{
+ phys_addr_t dst_addr = virt_to_phys(page);
+ unsigned long pfn = __phys_to_pfn(dst_addr);
+ int max_msb = (dst_addr & GENMASK(52, 48)) ? 51 : 47;
+ int bits_mapped = PAGE_SHIFT - 4;
+ unsigned long level_mask, prev_level_entry, *levels[4];
+ int this_level, index, level_lsb, level_msb;
+
+ dst_addr &= PAGE_MASK;
+ prev_level_entry = pte_val(pfn_pte(pfn, PAGE_KERNEL_EXEC));
+
+ for (this_level = 3; this_level >= 0; this_level--) {
+ levels[this_level] = trans_alloc(info);
+ if (!levels[this_level])
+ return -ENOMEM;
+
+ level_lsb = ARM64_HW_PGTABLE_LEVEL_SHIFT(this_level);
+ level_msb = min(level_lsb + bits_mapped, max_msb);
+ level_mask = GENMASK_ULL(level_msb, level_lsb);
+
+ index = (dst_addr & level_mask) >> level_lsb;
+ *(levels[this_level] + index) = prev_level_entry;
+
+ pfn = virt_to_pfn(levels[this_level]);
+ prev_level_entry = pte_val(pfn_pte(pfn,
+ __pgprot(PMD_TYPE_TABLE)));
+
+ if (level_msb == max_msb)
+ break;
+ }
+
+ *trans_ttbr0 = phys_to_ttbr(__pfn_to_phys(pfn));
+ *t0sz = TCR_T0SZ(max_msb + 1);
+
+ return 0;
+}
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 08/18] arm64: trans_pgd: hibernate: idmap the single page that holds the copy page routines
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
From: James Morse <james.morse@arm.com>
To resume from hibernate, the contents of memory are restored from
the swap image. This may overwrite any page, including the running
kernel and its page tables.
Hibernate copies the code it uses to do the restore into a single
page that it knows won't be overwritten, and maps it with page tables
built from pages that won't be overwritten.
Today the address it uses for this mapping is arbitrary, but to allow
kexec to reuse this code, it needs to be idmapped. To idmap the page
we must avoid the kernel helpers that have VA_BITS baked in.
Convert create_single_mapping() to take a single PA, and idmap it.
The page tables are built in the reverse order to normal using
pfn_pte() to stir in any bits between 52:48. T0SZ is always increased
to cover 48bits, or 52 if the copy code has bits 52:48 in its PA.
Signed-off-by: James Morse <james.morse@arm.com>
[Adopted the original patch from James to trans_pgd interface, so it can be
commonly used by both Kexec and Hibernate. Some minor clean-ups.]
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Link: https://lore.kernel.org/linux-arm-kernel/20200115143322.214247-4-james.morse@arm.com/
---
arch/arm64/include/asm/trans_pgd.h | 3 ++
arch/arm64/kernel/hibernate.c | 32 +++++++------------
arch/arm64/mm/trans_pgd.c | 49 ++++++++++++++++++++++++++++++
3 files changed, 63 insertions(+), 21 deletions(-)
diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/trans_pgd.h
index 7fbf6a3ccff7..5d08e5adf3d5 100644
--- a/arch/arm64/include/asm/trans_pgd.h
+++ b/arch/arm64/include/asm/trans_pgd.h
@@ -33,4 +33,7 @@ int trans_pgd_create_copy(struct trans_pgd_info *info, pgd_t **trans_pgd,
int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
void *page, unsigned long dst_addr, pgprot_t pgprot);
+int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans_ttbr0,
+ unsigned long *t0sz, void *page);
+
#endif /* _ASM_TRANS_TABLE_H */
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 94fc275cdd21..9df32ba0d574 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -194,7 +194,6 @@ static void *hibernate_page_alloc(void *arg)
* page system.
*/
static int create_safe_exec_page(void *src_start, size_t length,
- unsigned long dst_addr,
phys_addr_t *phys_dst_addr)
{
struct trans_pgd_info trans_info = {
@@ -203,7 +202,8 @@ static int create_safe_exec_page(void *src_start, size_t length,
};
void *page = (void *)get_safe_page(GFP_ATOMIC);
- pgd_t *trans_pgd;
+ phys_addr_t trans_ttbr0;
+ unsigned long t0sz;
int rc;
if (!page)
@@ -211,13 +211,7 @@ static int create_safe_exec_page(void *src_start, size_t length,
memcpy(page, src_start, length);
__flush_icache_range((unsigned long)page, (unsigned long)page + length);
-
- trans_pgd = (void *)get_safe_page(GFP_ATOMIC);
- if (!trans_pgd)
- return -ENOMEM;
-
- rc = trans_pgd_map_page(&trans_info, trans_pgd, page, dst_addr,
- PAGE_KERNEL_EXEC);
+ rc = trans_pgd_idmap_page(&trans_info, &trans_ttbr0, &t0sz, page);
if (rc)
return rc;
@@ -230,12 +224,15 @@ static int create_safe_exec_page(void *src_start, size_t length,
* page, but TLBs may contain stale ASID-tagged entries (e.g. for EFI
* runtime services), while for a userspace-driven test_resume cycle it
* points to userspace page tables (and we must point it at a zero page
- * ourselves). Elsewhere we only (un)install the idmap with preemption
- * disabled, so T0SZ should be as required regardless.
+ * ourselves).
+ *
+ * We change T0SZ as part of installing the idmap. This is undone by
+ * cpu_uninstall_idmap() in __cpu_suspend_exit().
*/
cpu_set_reserved_ttbr0();
local_flush_tlb_all();
- write_sysreg(phys_to_ttbr(virt_to_phys(trans_pgd)), ttbr0_el1);
+ __cpu_set_tcr_t0sz(t0sz);
+ write_sysreg(trans_ttbr0, ttbr0_el1);
isb();
*phys_dst_addr = virt_to_phys(page);
@@ -434,7 +431,6 @@ int swsusp_arch_resume(void)
void *zero_page;
size_t exit_size;
pgd_t *tmp_pg_dir;
- phys_addr_t phys_hibernate_exit;
void __noreturn (*hibernate_exit)(phys_addr_t, phys_addr_t, void *,
void *, phys_addr_t, phys_addr_t);
struct trans_pgd_info trans_info = {
@@ -462,19 +458,13 @@ int swsusp_arch_resume(void)
return -ENOMEM;
}
- /*
- * Locate the exit code in the bottom-but-one page, so that *NULL
- * still has disastrous affects.
- */
- hibernate_exit = (void *)PAGE_SIZE;
exit_size = __hibernate_exit_text_end - __hibernate_exit_text_start;
/*
* Copy swsusp_arch_suspend_exit() to a safe page. This will generate
* a new set of ttbr0 page tables and load them.
*/
rc = create_safe_exec_page(__hibernate_exit_text_start, exit_size,
- (unsigned long)hibernate_exit,
- &phys_hibernate_exit);
+ (phys_addr_t *)&hibernate_exit);
if (rc) {
pr_err("Failed to create safe executable page for hibernate_exit code.\n");
return rc;
@@ -493,7 +483,7 @@ int swsusp_arch_resume(void)
* We can skip this step if we booted at EL1, or are running with VHE.
*/
if (el2_reset_needed()) {
- phys_addr_t el2_vectors = phys_hibernate_exit; /* base */
+ phys_addr_t el2_vectors = (phys_addr_t)hibernate_exit;
el2_vectors += hibernate_el2_vectors -
__hibernate_exit_text_start; /* offset */
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
index ded8e2ba0308..527f0a39c3da 100644
--- a/arch/arm64/mm/trans_pgd.c
+++ b/arch/arm64/mm/trans_pgd.c
@@ -273,3 +273,52 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
return 0;
}
+
+/*
+ * The page we want to idmap may be outside the range covered by VA_BITS that
+ * can be built using the kernel's p?d_populate() helpers. As a one off, for a
+ * single page, we build these page tables bottom up and just assume that will
+ * need the maximum T0SZ.
+ *
+ * Returns 0 on success, and -ENOMEM on failure.
+ * On success trans_ttbr0 contains page table with idmapped page, t0sz is set to
+ * maximum T0SZ for this page.
+ */
+int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans_ttbr0,
+ unsigned long *t0sz, void *page)
+{
+ phys_addr_t dst_addr = virt_to_phys(page);
+ unsigned long pfn = __phys_to_pfn(dst_addr);
+ int max_msb = (dst_addr & GENMASK(52, 48)) ? 51 : 47;
+ int bits_mapped = PAGE_SHIFT - 4;
+ unsigned long level_mask, prev_level_entry, *levels[4];
+ int this_level, index, level_lsb, level_msb;
+
+ dst_addr &= PAGE_MASK;
+ prev_level_entry = pte_val(pfn_pte(pfn, PAGE_KERNEL_EXEC));
+
+ for (this_level = 3; this_level >= 0; this_level--) {
+ levels[this_level] = trans_alloc(info);
+ if (!levels[this_level])
+ return -ENOMEM;
+
+ level_lsb = ARM64_HW_PGTABLE_LEVEL_SHIFT(this_level);
+ level_msb = min(level_lsb + bits_mapped, max_msb);
+ level_mask = GENMASK_ULL(level_msb, level_lsb);
+
+ index = (dst_addr & level_mask) >> level_lsb;
+ *(levels[this_level] + index) = prev_level_entry;
+
+ pfn = virt_to_pfn(levels[this_level]);
+ prev_level_entry = pte_val(pfn_pte(pfn,
+ __pgprot(PMD_TYPE_TABLE)));
+
+ if (level_msb == max_msb)
+ break;
+ }
+
+ *trans_ttbr0 = phys_to_ttbr(__pfn_to_phys(pfn));
+ *t0sz = TCR_T0SZ(max_msb + 1);
+
+ return 0;
+}
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 08/18] arm64: trans_pgd: hibernate: idmap the single page that holds the copy page routines
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
From: James Morse <james.morse@arm.com>
To resume from hibernate, the contents of memory are restored from
the swap image. This may overwrite any page, including the running
kernel and its page tables.
Hibernate copies the code it uses to do the restore into a single
page that it knows won't be overwritten, and maps it with page tables
built from pages that won't be overwritten.
Today the address it uses for this mapping is arbitrary, but to allow
kexec to reuse this code, it needs to be idmapped. To idmap the page
we must avoid the kernel helpers that have VA_BITS baked in.
Convert create_single_mapping() to take a single PA, and idmap it.
The page tables are built in the reverse order to normal using
pfn_pte() to stir in any bits between 52:48. T0SZ is always increased
to cover 48bits, or 52 if the copy code has bits 52:48 in its PA.
Signed-off-by: James Morse <james.morse@arm.com>
[Adopted the original patch from James to trans_pgd interface, so it can be
commonly used by both Kexec and Hibernate. Some minor clean-ups.]
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Link: https://lore.kernel.org/linux-arm-kernel/20200115143322.214247-4-james.morse@arm.com/
---
arch/arm64/include/asm/trans_pgd.h | 3 ++
arch/arm64/kernel/hibernate.c | 32 +++++++------------
arch/arm64/mm/trans_pgd.c | 49 ++++++++++++++++++++++++++++++
3 files changed, 63 insertions(+), 21 deletions(-)
diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/trans_pgd.h
index 7fbf6a3ccff7..5d08e5adf3d5 100644
--- a/arch/arm64/include/asm/trans_pgd.h
+++ b/arch/arm64/include/asm/trans_pgd.h
@@ -33,4 +33,7 @@ int trans_pgd_create_copy(struct trans_pgd_info *info, pgd_t **trans_pgd,
int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
void *page, unsigned long dst_addr, pgprot_t pgprot);
+int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans_ttbr0,
+ unsigned long *t0sz, void *page);
+
#endif /* _ASM_TRANS_TABLE_H */
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 94fc275cdd21..9df32ba0d574 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -194,7 +194,6 @@ static void *hibernate_page_alloc(void *arg)
* page system.
*/
static int create_safe_exec_page(void *src_start, size_t length,
- unsigned long dst_addr,
phys_addr_t *phys_dst_addr)
{
struct trans_pgd_info trans_info = {
@@ -203,7 +202,8 @@ static int create_safe_exec_page(void *src_start, size_t length,
};
void *page = (void *)get_safe_page(GFP_ATOMIC);
- pgd_t *trans_pgd;
+ phys_addr_t trans_ttbr0;
+ unsigned long t0sz;
int rc;
if (!page)
@@ -211,13 +211,7 @@ static int create_safe_exec_page(void *src_start, size_t length,
memcpy(page, src_start, length);
__flush_icache_range((unsigned long)page, (unsigned long)page + length);
-
- trans_pgd = (void *)get_safe_page(GFP_ATOMIC);
- if (!trans_pgd)
- return -ENOMEM;
-
- rc = trans_pgd_map_page(&trans_info, trans_pgd, page, dst_addr,
- PAGE_KERNEL_EXEC);
+ rc = trans_pgd_idmap_page(&trans_info, &trans_ttbr0, &t0sz, page);
if (rc)
return rc;
@@ -230,12 +224,15 @@ static int create_safe_exec_page(void *src_start, size_t length,
* page, but TLBs may contain stale ASID-tagged entries (e.g. for EFI
* runtime services), while for a userspace-driven test_resume cycle it
* points to userspace page tables (and we must point it at a zero page
- * ourselves). Elsewhere we only (un)install the idmap with preemption
- * disabled, so T0SZ should be as required regardless.
+ * ourselves).
+ *
+ * We change T0SZ as part of installing the idmap. This is undone by
+ * cpu_uninstall_idmap() in __cpu_suspend_exit().
*/
cpu_set_reserved_ttbr0();
local_flush_tlb_all();
- write_sysreg(phys_to_ttbr(virt_to_phys(trans_pgd)), ttbr0_el1);
+ __cpu_set_tcr_t0sz(t0sz);
+ write_sysreg(trans_ttbr0, ttbr0_el1);
isb();
*phys_dst_addr = virt_to_phys(page);
@@ -434,7 +431,6 @@ int swsusp_arch_resume(void)
void *zero_page;
size_t exit_size;
pgd_t *tmp_pg_dir;
- phys_addr_t phys_hibernate_exit;
void __noreturn (*hibernate_exit)(phys_addr_t, phys_addr_t, void *,
void *, phys_addr_t, phys_addr_t);
struct trans_pgd_info trans_info = {
@@ -462,19 +458,13 @@ int swsusp_arch_resume(void)
return -ENOMEM;
}
- /*
- * Locate the exit code in the bottom-but-one page, so that *NULL
- * still has disastrous affects.
- */
- hibernate_exit = (void *)PAGE_SIZE;
exit_size = __hibernate_exit_text_end - __hibernate_exit_text_start;
/*
* Copy swsusp_arch_suspend_exit() to a safe page. This will generate
* a new set of ttbr0 page tables and load them.
*/
rc = create_safe_exec_page(__hibernate_exit_text_start, exit_size,
- (unsigned long)hibernate_exit,
- &phys_hibernate_exit);
+ (phys_addr_t *)&hibernate_exit);
if (rc) {
pr_err("Failed to create safe executable page for hibernate_exit code.\n");
return rc;
@@ -493,7 +483,7 @@ int swsusp_arch_resume(void)
* We can skip this step if we booted at EL1, or are running with VHE.
*/
if (el2_reset_needed()) {
- phys_addr_t el2_vectors = phys_hibernate_exit; /* base */
+ phys_addr_t el2_vectors = (phys_addr_t)hibernate_exit;
el2_vectors += hibernate_el2_vectors -
__hibernate_exit_text_start; /* offset */
diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c
index ded8e2ba0308..527f0a39c3da 100644
--- a/arch/arm64/mm/trans_pgd.c
+++ b/arch/arm64/mm/trans_pgd.c
@@ -273,3 +273,52 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd,
return 0;
}
+
+/*
+ * The page we want to idmap may be outside the range covered by VA_BITS that
+ * can be built using the kernel's p?d_populate() helpers. As a one off, for a
+ * single page, we build these page tables bottom up and just assume that will
+ * need the maximum T0SZ.
+ *
+ * Returns 0 on success, and -ENOMEM on failure.
+ * On success trans_ttbr0 contains page table with idmapped page, t0sz is set to
+ * maximum T0SZ for this page.
+ */
+int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans_ttbr0,
+ unsigned long *t0sz, void *page)
+{
+ phys_addr_t dst_addr = virt_to_phys(page);
+ unsigned long pfn = __phys_to_pfn(dst_addr);
+ int max_msb = (dst_addr & GENMASK(52, 48)) ? 51 : 47;
+ int bits_mapped = PAGE_SHIFT - 4;
+ unsigned long level_mask, prev_level_entry, *levels[4];
+ int this_level, index, level_lsb, level_msb;
+
+ dst_addr &= PAGE_MASK;
+ prev_level_entry = pte_val(pfn_pte(pfn, PAGE_KERNEL_EXEC));
+
+ for (this_level = 3; this_level >= 0; this_level--) {
+ levels[this_level] = trans_alloc(info);
+ if (!levels[this_level])
+ return -ENOMEM;
+
+ level_lsb = ARM64_HW_PGTABLE_LEVEL_SHIFT(this_level);
+ level_msb = min(level_lsb + bits_mapped, max_msb);
+ level_mask = GENMASK_ULL(level_msb, level_lsb);
+
+ index = (dst_addr & level_mask) >> level_lsb;
+ *(levels[this_level] + index) = prev_level_entry;
+
+ pfn = virt_to_pfn(levels[this_level]);
+ prev_level_entry = pte_val(pfn_pte(pfn,
+ __pgprot(PMD_TYPE_TABLE)));
+
+ if (level_msb == max_msb)
+ break;
+ }
+
+ *trans_ttbr0 = phys_to_ttbr(__pfn_to_phys(pfn));
+ *t0sz = TCR_T0SZ(max_msb + 1);
+
+ return 0;
+}
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 09/18] arm64: kexec: move relocation function setup
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-25 19:19 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Currently, kernel relocation function is configured in machine_kexec()
at the time of kexec reboot by using control_code_page.
This operation, however, is more logical to be done during kexec_load,
and thus remove from reboot time. Move, setup of this function to
newly added machine_kexec_post_load().
Because once MMU is enabled, kexec control page will contain more than
relocation kernel, but also vector table, add pointer to the actual
function within this page arch.kern_reloc. Currently, it equals to the
beginning of page, we will add offsets later, when vector table is
added.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: James Morse <james.morse@arm.com>
---
arch/arm64/include/asm/kexec.h | 1 +
arch/arm64/kernel/machine_kexec.c | 46 +++++++++++++------------------
2 files changed, 20 insertions(+), 27 deletions(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index 61530ec3a9b1..9befcd87e9a8 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -95,6 +95,7 @@ static inline void crash_post_resume(void) {}
struct kimage_arch {
void *dtb;
phys_addr_t dtb_mem;
+ phys_addr_t kern_reloc;
/* Core ELF header buffer */
void *elf_headers;
unsigned long elf_headers_mem;
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 8096a6aa1d49..a8aaa6562429 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -42,6 +42,7 @@ static void _kexec_image_info(const char *func, int line,
pr_debug(" start: %lx\n", kimage->start);
pr_debug(" head: %lx\n", kimage->head);
pr_debug(" nr_segments: %lu\n", kimage->nr_segments);
+ pr_debug(" kern_reloc: %pa\n", &kimage->arch.kern_reloc);
for (i = 0; i < kimage->nr_segments; i++) {
pr_debug(" segment[%lu]: %016lx - %016lx, 0x%lx bytes, %lu pages\n",
@@ -58,6 +59,22 @@ void machine_kexec_cleanup(struct kimage *kimage)
/* Empty routine needed to avoid build errors. */
}
+int machine_kexec_post_load(struct kimage *kimage)
+{
+ void *reloc_code = page_to_virt(kimage->control_code_page);
+
+ memcpy(reloc_code, arm64_relocate_new_kernel,
+ arm64_relocate_new_kernel_size);
+ kimage->arch.kern_reloc = __pa(reloc_code);
+
+ /* Flush the reloc_code in preparation for its execution. */
+ __flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size);
+ flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code +
+ arm64_relocate_new_kernel_size);
+
+ return 0;
+}
+
/**
* machine_kexec_prepare - Prepare for a kexec reboot.
*
@@ -143,8 +160,6 @@ static void kexec_segment_flush(const struct kimage *kimage)
*/
void machine_kexec(struct kimage *kimage)
{
- phys_addr_t reboot_code_buffer_phys;
- void *reboot_code_buffer;
bool in_kexec_crash = (kimage == kexec_crash_image);
bool stuck_cpus = cpus_are_stuck_in_kernel();
@@ -155,31 +170,8 @@ void machine_kexec(struct kimage *kimage)
WARN(in_kexec_crash && (stuck_cpus || smp_crash_stop_failed()),
"Some CPUs may be stale, kdump will be unreliable.\n");
- reboot_code_buffer_phys = page_to_phys(kimage->control_code_page);
- reboot_code_buffer = phys_to_virt(reboot_code_buffer_phys);
-
kexec_image_info(kimage);
- /*
- * Copy arm64_relocate_new_kernel to the reboot_code_buffer for use
- * after the kernel is shut down.
- */
- memcpy(reboot_code_buffer, arm64_relocate_new_kernel,
- arm64_relocate_new_kernel_size);
-
- /* Flush the reboot_code_buffer in preparation for its execution. */
- __flush_dcache_area(reboot_code_buffer, arm64_relocate_new_kernel_size);
-
- /*
- * Although we've killed off the secondary CPUs, we don't update
- * the online mask if we're handling a crash kernel and consequently
- * need to avoid flush_icache_range(), which will attempt to IPI
- * the offline CPUs. Therefore, we must use the __* variant here.
- */
- __flush_icache_range((uintptr_t)reboot_code_buffer,
- (uintptr_t)reboot_code_buffer +
- arm64_relocate_new_kernel_size);
-
/* Flush the kimage list and its buffers. */
kexec_list_flush(kimage);
@@ -193,7 +185,7 @@ void machine_kexec(struct kimage *kimage)
/*
* cpu_soft_restart will shutdown the MMU, disable data caches, then
- * transfer control to the reboot_code_buffer which contains a copy of
+ * transfer control to the kern_reloc which contains a copy of
* the arm64_relocate_new_kernel routine. arm64_relocate_new_kernel
* uses physical addressing to relocate the new image to its final
* position and transfers control to the image entry point when the
@@ -203,7 +195,7 @@ void machine_kexec(struct kimage *kimage)
* userspace (kexec-tools).
* In kexec_file case, the kernel starts directly without purgatory.
*/
- cpu_soft_restart(reboot_code_buffer_phys, kimage->head, kimage->start,
+ cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, kimage->start,
kimage->arch.dtb_mem);
BUG(); /* Should never get here. */
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 09/18] arm64: kexec: move relocation function setup
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Currently, kernel relocation function is configured in machine_kexec()
at the time of kexec reboot by using control_code_page.
This operation, however, is more logical to be done during kexec_load,
and thus remove from reboot time. Move, setup of this function to
newly added machine_kexec_post_load().
Because once MMU is enabled, kexec control page will contain more than
relocation kernel, but also vector table, add pointer to the actual
function within this page arch.kern_reloc. Currently, it equals to the
beginning of page, we will add offsets later, when vector table is
added.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: James Morse <james.morse@arm.com>
---
arch/arm64/include/asm/kexec.h | 1 +
arch/arm64/kernel/machine_kexec.c | 46 +++++++++++++------------------
2 files changed, 20 insertions(+), 27 deletions(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index 61530ec3a9b1..9befcd87e9a8 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -95,6 +95,7 @@ static inline void crash_post_resume(void) {}
struct kimage_arch {
void *dtb;
phys_addr_t dtb_mem;
+ phys_addr_t kern_reloc;
/* Core ELF header buffer */
void *elf_headers;
unsigned long elf_headers_mem;
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 8096a6aa1d49..a8aaa6562429 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -42,6 +42,7 @@ static void _kexec_image_info(const char *func, int line,
pr_debug(" start: %lx\n", kimage->start);
pr_debug(" head: %lx\n", kimage->head);
pr_debug(" nr_segments: %lu\n", kimage->nr_segments);
+ pr_debug(" kern_reloc: %pa\n", &kimage->arch.kern_reloc);
for (i = 0; i < kimage->nr_segments; i++) {
pr_debug(" segment[%lu]: %016lx - %016lx, 0x%lx bytes, %lu pages\n",
@@ -58,6 +59,22 @@ void machine_kexec_cleanup(struct kimage *kimage)
/* Empty routine needed to avoid build errors. */
}
+int machine_kexec_post_load(struct kimage *kimage)
+{
+ void *reloc_code = page_to_virt(kimage->control_code_page);
+
+ memcpy(reloc_code, arm64_relocate_new_kernel,
+ arm64_relocate_new_kernel_size);
+ kimage->arch.kern_reloc = __pa(reloc_code);
+
+ /* Flush the reloc_code in preparation for its execution. */
+ __flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size);
+ flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code +
+ arm64_relocate_new_kernel_size);
+
+ return 0;
+}
+
/**
* machine_kexec_prepare - Prepare for a kexec reboot.
*
@@ -143,8 +160,6 @@ static void kexec_segment_flush(const struct kimage *kimage)
*/
void machine_kexec(struct kimage *kimage)
{
- phys_addr_t reboot_code_buffer_phys;
- void *reboot_code_buffer;
bool in_kexec_crash = (kimage == kexec_crash_image);
bool stuck_cpus = cpus_are_stuck_in_kernel();
@@ -155,31 +170,8 @@ void machine_kexec(struct kimage *kimage)
WARN(in_kexec_crash && (stuck_cpus || smp_crash_stop_failed()),
"Some CPUs may be stale, kdump will be unreliable.\n");
- reboot_code_buffer_phys = page_to_phys(kimage->control_code_page);
- reboot_code_buffer = phys_to_virt(reboot_code_buffer_phys);
-
kexec_image_info(kimage);
- /*
- * Copy arm64_relocate_new_kernel to the reboot_code_buffer for use
- * after the kernel is shut down.
- */
- memcpy(reboot_code_buffer, arm64_relocate_new_kernel,
- arm64_relocate_new_kernel_size);
-
- /* Flush the reboot_code_buffer in preparation for its execution. */
- __flush_dcache_area(reboot_code_buffer, arm64_relocate_new_kernel_size);
-
- /*
- * Although we've killed off the secondary CPUs, we don't update
- * the online mask if we're handling a crash kernel and consequently
- * need to avoid flush_icache_range(), which will attempt to IPI
- * the offline CPUs. Therefore, we must use the __* variant here.
- */
- __flush_icache_range((uintptr_t)reboot_code_buffer,
- (uintptr_t)reboot_code_buffer +
- arm64_relocate_new_kernel_size);
-
/* Flush the kimage list and its buffers. */
kexec_list_flush(kimage);
@@ -193,7 +185,7 @@ void machine_kexec(struct kimage *kimage)
/*
* cpu_soft_restart will shutdown the MMU, disable data caches, then
- * transfer control to the reboot_code_buffer which contains a copy of
+ * transfer control to the kern_reloc which contains a copy of
* the arm64_relocate_new_kernel routine. arm64_relocate_new_kernel
* uses physical addressing to relocate the new image to its final
* position and transfers control to the image entry point when the
@@ -203,7 +195,7 @@ void machine_kexec(struct kimage *kimage)
* userspace (kexec-tools).
* In kexec_file case, the kernel starts directly without purgatory.
*/
- cpu_soft_restart(reboot_code_buffer_phys, kimage->head, kimage->start,
+ cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, kimage->start,
kimage->arch.dtb_mem);
BUG(); /* Should never get here. */
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 09/18] arm64: kexec: move relocation function setup
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Currently, kernel relocation function is configured in machine_kexec()
at the time of kexec reboot by using control_code_page.
This operation, however, is more logical to be done during kexec_load,
and thus remove from reboot time. Move, setup of this function to
newly added machine_kexec_post_load().
Because once MMU is enabled, kexec control page will contain more than
relocation kernel, but also vector table, add pointer to the actual
function within this page arch.kern_reloc. Currently, it equals to the
beginning of page, we will add offsets later, when vector table is
added.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: James Morse <james.morse@arm.com>
---
arch/arm64/include/asm/kexec.h | 1 +
arch/arm64/kernel/machine_kexec.c | 46 +++++++++++++------------------
2 files changed, 20 insertions(+), 27 deletions(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index 61530ec3a9b1..9befcd87e9a8 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -95,6 +95,7 @@ static inline void crash_post_resume(void) {}
struct kimage_arch {
void *dtb;
phys_addr_t dtb_mem;
+ phys_addr_t kern_reloc;
/* Core ELF header buffer */
void *elf_headers;
unsigned long elf_headers_mem;
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 8096a6aa1d49..a8aaa6562429 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -42,6 +42,7 @@ static void _kexec_image_info(const char *func, int line,
pr_debug(" start: %lx\n", kimage->start);
pr_debug(" head: %lx\n", kimage->head);
pr_debug(" nr_segments: %lu\n", kimage->nr_segments);
+ pr_debug(" kern_reloc: %pa\n", &kimage->arch.kern_reloc);
for (i = 0; i < kimage->nr_segments; i++) {
pr_debug(" segment[%lu]: %016lx - %016lx, 0x%lx bytes, %lu pages\n",
@@ -58,6 +59,22 @@ void machine_kexec_cleanup(struct kimage *kimage)
/* Empty routine needed to avoid build errors. */
}
+int machine_kexec_post_load(struct kimage *kimage)
+{
+ void *reloc_code = page_to_virt(kimage->control_code_page);
+
+ memcpy(reloc_code, arm64_relocate_new_kernel,
+ arm64_relocate_new_kernel_size);
+ kimage->arch.kern_reloc = __pa(reloc_code);
+
+ /* Flush the reloc_code in preparation for its execution. */
+ __flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size);
+ flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code +
+ arm64_relocate_new_kernel_size);
+
+ return 0;
+}
+
/**
* machine_kexec_prepare - Prepare for a kexec reboot.
*
@@ -143,8 +160,6 @@ static void kexec_segment_flush(const struct kimage *kimage)
*/
void machine_kexec(struct kimage *kimage)
{
- phys_addr_t reboot_code_buffer_phys;
- void *reboot_code_buffer;
bool in_kexec_crash = (kimage == kexec_crash_image);
bool stuck_cpus = cpus_are_stuck_in_kernel();
@@ -155,31 +170,8 @@ void machine_kexec(struct kimage *kimage)
WARN(in_kexec_crash && (stuck_cpus || smp_crash_stop_failed()),
"Some CPUs may be stale, kdump will be unreliable.\n");
- reboot_code_buffer_phys = page_to_phys(kimage->control_code_page);
- reboot_code_buffer = phys_to_virt(reboot_code_buffer_phys);
-
kexec_image_info(kimage);
- /*
- * Copy arm64_relocate_new_kernel to the reboot_code_buffer for use
- * after the kernel is shut down.
- */
- memcpy(reboot_code_buffer, arm64_relocate_new_kernel,
- arm64_relocate_new_kernel_size);
-
- /* Flush the reboot_code_buffer in preparation for its execution. */
- __flush_dcache_area(reboot_code_buffer, arm64_relocate_new_kernel_size);
-
- /*
- * Although we've killed off the secondary CPUs, we don't update
- * the online mask if we're handling a crash kernel and consequently
- * need to avoid flush_icache_range(), which will attempt to IPI
- * the offline CPUs. Therefore, we must use the __* variant here.
- */
- __flush_icache_range((uintptr_t)reboot_code_buffer,
- (uintptr_t)reboot_code_buffer +
- arm64_relocate_new_kernel_size);
-
/* Flush the kimage list and its buffers. */
kexec_list_flush(kimage);
@@ -193,7 +185,7 @@ void machine_kexec(struct kimage *kimage)
/*
* cpu_soft_restart will shutdown the MMU, disable data caches, then
- * transfer control to the reboot_code_buffer which contains a copy of
+ * transfer control to the kern_reloc which contains a copy of
* the arm64_relocate_new_kernel routine. arm64_relocate_new_kernel
* uses physical addressing to relocate the new image to its final
* position and transfers control to the image entry point when the
@@ -203,7 +195,7 @@ void machine_kexec(struct kimage *kimage)
* userspace (kexec-tools).
* In kexec_file case, the kernel starts directly without purgatory.
*/
- cpu_soft_restart(reboot_code_buffer_phys, kimage->head, kimage->start,
+ cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, kimage->start,
kimage->arch.dtb_mem);
BUG(); /* Should never get here. */
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 10/18] arm64: kexec: call kexec_image_info only once
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-25 19:19 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Currently, kexec_image_info() is called during load time, and
right before kernel is being kexec'ed. There is no need to do both.
So, call it only once when segments are loaded and the physical
location of page with copy of arm64_relocate_new_kernel is known.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Acked-by: James Morse <james.morse@arm.com>
---
arch/arm64/kernel/machine_kexec.c | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index a8aaa6562429..90a335c74442 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -66,6 +66,7 @@ int machine_kexec_post_load(struct kimage *kimage)
memcpy(reloc_code, arm64_relocate_new_kernel,
arm64_relocate_new_kernel_size);
kimage->arch.kern_reloc = __pa(reloc_code);
+ kexec_image_info(kimage);
/* Flush the reloc_code in preparation for its execution. */
__flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size);
@@ -84,8 +85,6 @@ int machine_kexec_post_load(struct kimage *kimage)
*/
int machine_kexec_prepare(struct kimage *kimage)
{
- kexec_image_info(kimage);
-
if (kimage->type != KEXEC_TYPE_CRASH && cpus_are_stuck_in_kernel()) {
pr_err("Can't kexec: CPUs are stuck in the kernel.\n");
return -EBUSY;
@@ -170,8 +169,6 @@ void machine_kexec(struct kimage *kimage)
WARN(in_kexec_crash && (stuck_cpus || smp_crash_stop_failed()),
"Some CPUs may be stale, kdump will be unreliable.\n");
- kexec_image_info(kimage);
-
/* Flush the kimage list and its buffers. */
kexec_list_flush(kimage);
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 10/18] arm64: kexec: call kexec_image_info only once
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Currently, kexec_image_info() is called during load time, and
right before kernel is being kexec'ed. There is no need to do both.
So, call it only once when segments are loaded and the physical
location of page with copy of arm64_relocate_new_kernel is known.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Acked-by: James Morse <james.morse@arm.com>
---
arch/arm64/kernel/machine_kexec.c | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index a8aaa6562429..90a335c74442 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -66,6 +66,7 @@ int machine_kexec_post_load(struct kimage *kimage)
memcpy(reloc_code, arm64_relocate_new_kernel,
arm64_relocate_new_kernel_size);
kimage->arch.kern_reloc = __pa(reloc_code);
+ kexec_image_info(kimage);
/* Flush the reloc_code in preparation for its execution. */
__flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size);
@@ -84,8 +85,6 @@ int machine_kexec_post_load(struct kimage *kimage)
*/
int machine_kexec_prepare(struct kimage *kimage)
{
- kexec_image_info(kimage);
-
if (kimage->type != KEXEC_TYPE_CRASH && cpus_are_stuck_in_kernel()) {
pr_err("Can't kexec: CPUs are stuck in the kernel.\n");
return -EBUSY;
@@ -170,8 +169,6 @@ void machine_kexec(struct kimage *kimage)
WARN(in_kexec_crash && (stuck_cpus || smp_crash_stop_failed()),
"Some CPUs may be stale, kdump will be unreliable.\n");
- kexec_image_info(kimage);
-
/* Flush the kimage list and its buffers. */
kexec_list_flush(kimage);
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 10/18] arm64: kexec: call kexec_image_info only once
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Currently, kexec_image_info() is called during load time, and
right before kernel is being kexec'ed. There is no need to do both.
So, call it only once when segments are loaded and the physical
location of page with copy of arm64_relocate_new_kernel is known.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Acked-by: James Morse <james.morse@arm.com>
---
arch/arm64/kernel/machine_kexec.c | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index a8aaa6562429..90a335c74442 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -66,6 +66,7 @@ int machine_kexec_post_load(struct kimage *kimage)
memcpy(reloc_code, arm64_relocate_new_kernel,
arm64_relocate_new_kernel_size);
kimage->arch.kern_reloc = __pa(reloc_code);
+ kexec_image_info(kimage);
/* Flush the reloc_code in preparation for its execution. */
__flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size);
@@ -84,8 +85,6 @@ int machine_kexec_post_load(struct kimage *kimage)
*/
int machine_kexec_prepare(struct kimage *kimage)
{
- kexec_image_info(kimage);
-
if (kimage->type != KEXEC_TYPE_CRASH && cpus_are_stuck_in_kernel()) {
pr_err("Can't kexec: CPUs are stuck in the kernel.\n");
return -EBUSY;
@@ -170,8 +169,6 @@ void machine_kexec(struct kimage *kimage)
WARN(in_kexec_crash && (stuck_cpus || smp_crash_stop_failed()),
"Some CPUs may be stale, kdump will be unreliable.\n");
- kexec_image_info(kimage);
-
/* Flush the kimage list and its buffers. */
kexec_list_flush(kimage);
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 11/18] arm64: kexec: arm64_relocate_new_kernel clean-ups and optimizations
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-25 19:19 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
In preparation to bigger changes to arm64_relocate_new_kernel that would
enable this function to do MMU backed memory copy, do few clean-ups and
optimizations. These include:
1. Call raw_dcache_line_size() only when relocation is actually going to
happen. i.e. kdump type kexec, does not need it.
2. copy_page(dest, src, tmps...) increments dest and src by PAGE_SIZE, so
no need to store dest prior to calling copy_page and increment it
after. Also, src is not used after a copy, not need to copy either.
3. For consistency use comment on the same line with instruction when it
describes the instruction itself.
4. Some comment corrections
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/kernel/relocate_kernel.S | 36 +++++++----------------------
1 file changed, 8 insertions(+), 28 deletions(-)
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index 84eec95ec06c..462ffbc37071 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -17,28 +17,24 @@
/*
* arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it.
*
- * The memory that the old kernel occupies may be overwritten when coping the
+ * The memory that the old kernel occupies may be overwritten when copying the
* new image to its final location. To assure that the
* arm64_relocate_new_kernel routine which does that copy is not overwritten,
* all code and data needed by arm64_relocate_new_kernel must be between the
* symbols arm64_relocate_new_kernel and arm64_relocate_new_kernel_end. The
* machine_kexec() routine will copy arm64_relocate_new_kernel to the kexec
- * control_code_page, a special page which has been set up to be preserved
- * during the copy operation.
+ * safe memory that has been set up to be preserved during the copy operation.
*/
SYM_CODE_START(arm64_relocate_new_kernel)
-
/* Setup the list loop variables. */
mov x18, x2 /* x18 = dtb address */
mov x17, x1 /* x17 = kimage_start */
mov x16, x0 /* x16 = kimage_head */
- raw_dcache_line_size x15, x0 /* x15 = dcache line size */
mov x14, xzr /* x14 = entry ptr */
mov x13, xzr /* x13 = copy dest */
-
/* Check if the new image needs relocation. */
tbnz x16, IND_DONE_BIT, .Ldone
-
+ raw_dcache_line_size x15, x0 /* x15 = dcache line size */
.Lloop:
and x12, x16, PAGE_MASK /* x12 = addr */
@@ -57,34 +53,18 @@ SYM_CODE_START(arm64_relocate_new_kernel)
b.lo 2b
dsb sy
- mov x20, x13
- mov x21, x12
- copy_page x20, x21, x0, x1, x2, x3, x4, x5, x6, x7
-
- /* dest += PAGE_SIZE */
- add x13, x13, PAGE_SIZE
+ copy_page x13, x12, x0, x1, x2, x3, x4, x5, x6, x7
b .Lnext
-
.Ltest_indirection:
tbz x16, IND_INDIRECTION_BIT, .Ltest_destination
-
- /* ptr = addr */
- mov x14, x12
+ mov x14, x12 /* ptr = addr */
b .Lnext
-
.Ltest_destination:
tbz x16, IND_DESTINATION_BIT, .Lnext
-
- /* dest = addr */
- mov x13, x12
-
+ mov x13, x12 /* dest = addr */
.Lnext:
- /* entry = *ptr++ */
- ldr x16, [x14], #8
-
- /* while (!(entry & DONE)) */
- tbz x16, IND_DONE_BIT, .Lloop
-
+ ldr x16, [x14], #8 /* entry = *ptr++ */
+ tbz x16, IND_DONE_BIT, .Lloop /* while (!(entry & DONE)) */
.Ldone:
/* wait for writes from copy_page to finish */
dsb nsh
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 11/18] arm64: kexec: arm64_relocate_new_kernel clean-ups and optimizations
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
In preparation to bigger changes to arm64_relocate_new_kernel that would
enable this function to do MMU backed memory copy, do few clean-ups and
optimizations. These include:
1. Call raw_dcache_line_size() only when relocation is actually going to
happen. i.e. kdump type kexec, does not need it.
2. copy_page(dest, src, tmps...) increments dest and src by PAGE_SIZE, so
no need to store dest prior to calling copy_page and increment it
after. Also, src is not used after a copy, not need to copy either.
3. For consistency use comment on the same line with instruction when it
describes the instruction itself.
4. Some comment corrections
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/kernel/relocate_kernel.S | 36 +++++++----------------------
1 file changed, 8 insertions(+), 28 deletions(-)
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index 84eec95ec06c..462ffbc37071 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -17,28 +17,24 @@
/*
* arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it.
*
- * The memory that the old kernel occupies may be overwritten when coping the
+ * The memory that the old kernel occupies may be overwritten when copying the
* new image to its final location. To assure that the
* arm64_relocate_new_kernel routine which does that copy is not overwritten,
* all code and data needed by arm64_relocate_new_kernel must be between the
* symbols arm64_relocate_new_kernel and arm64_relocate_new_kernel_end. The
* machine_kexec() routine will copy arm64_relocate_new_kernel to the kexec
- * control_code_page, a special page which has been set up to be preserved
- * during the copy operation.
+ * safe memory that has been set up to be preserved during the copy operation.
*/
SYM_CODE_START(arm64_relocate_new_kernel)
-
/* Setup the list loop variables. */
mov x18, x2 /* x18 = dtb address */
mov x17, x1 /* x17 = kimage_start */
mov x16, x0 /* x16 = kimage_head */
- raw_dcache_line_size x15, x0 /* x15 = dcache line size */
mov x14, xzr /* x14 = entry ptr */
mov x13, xzr /* x13 = copy dest */
-
/* Check if the new image needs relocation. */
tbnz x16, IND_DONE_BIT, .Ldone
-
+ raw_dcache_line_size x15, x0 /* x15 = dcache line size */
.Lloop:
and x12, x16, PAGE_MASK /* x12 = addr */
@@ -57,34 +53,18 @@ SYM_CODE_START(arm64_relocate_new_kernel)
b.lo 2b
dsb sy
- mov x20, x13
- mov x21, x12
- copy_page x20, x21, x0, x1, x2, x3, x4, x5, x6, x7
-
- /* dest += PAGE_SIZE */
- add x13, x13, PAGE_SIZE
+ copy_page x13, x12, x0, x1, x2, x3, x4, x5, x6, x7
b .Lnext
-
.Ltest_indirection:
tbz x16, IND_INDIRECTION_BIT, .Ltest_destination
-
- /* ptr = addr */
- mov x14, x12
+ mov x14, x12 /* ptr = addr */
b .Lnext
-
.Ltest_destination:
tbz x16, IND_DESTINATION_BIT, .Lnext
-
- /* dest = addr */
- mov x13, x12
-
+ mov x13, x12 /* dest = addr */
.Lnext:
- /* entry = *ptr++ */
- ldr x16, [x14], #8
-
- /* while (!(entry & DONE)) */
- tbz x16, IND_DONE_BIT, .Lloop
-
+ ldr x16, [x14], #8 /* entry = *ptr++ */
+ tbz x16, IND_DONE_BIT, .Lloop /* while (!(entry & DONE)) */
.Ldone:
/* wait for writes from copy_page to finish */
dsb nsh
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 11/18] arm64: kexec: arm64_relocate_new_kernel clean-ups and optimizations
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
In preparation to bigger changes to arm64_relocate_new_kernel that would
enable this function to do MMU backed memory copy, do few clean-ups and
optimizations. These include:
1. Call raw_dcache_line_size() only when relocation is actually going to
happen. i.e. kdump type kexec, does not need it.
2. copy_page(dest, src, tmps...) increments dest and src by PAGE_SIZE, so
no need to store dest prior to calling copy_page and increment it
after. Also, src is not used after a copy, not need to copy either.
3. For consistency use comment on the same line with instruction when it
describes the instruction itself.
4. Some comment corrections
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/kernel/relocate_kernel.S | 36 +++++++----------------------
1 file changed, 8 insertions(+), 28 deletions(-)
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index 84eec95ec06c..462ffbc37071 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -17,28 +17,24 @@
/*
* arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it.
*
- * The memory that the old kernel occupies may be overwritten when coping the
+ * The memory that the old kernel occupies may be overwritten when copying the
* new image to its final location. To assure that the
* arm64_relocate_new_kernel routine which does that copy is not overwritten,
* all code and data needed by arm64_relocate_new_kernel must be between the
* symbols arm64_relocate_new_kernel and arm64_relocate_new_kernel_end. The
* machine_kexec() routine will copy arm64_relocate_new_kernel to the kexec
- * control_code_page, a special page which has been set up to be preserved
- * during the copy operation.
+ * safe memory that has been set up to be preserved during the copy operation.
*/
SYM_CODE_START(arm64_relocate_new_kernel)
-
/* Setup the list loop variables. */
mov x18, x2 /* x18 = dtb address */
mov x17, x1 /* x17 = kimage_start */
mov x16, x0 /* x16 = kimage_head */
- raw_dcache_line_size x15, x0 /* x15 = dcache line size */
mov x14, xzr /* x14 = entry ptr */
mov x13, xzr /* x13 = copy dest */
-
/* Check if the new image needs relocation. */
tbnz x16, IND_DONE_BIT, .Ldone
-
+ raw_dcache_line_size x15, x0 /* x15 = dcache line size */
.Lloop:
and x12, x16, PAGE_MASK /* x12 = addr */
@@ -57,34 +53,18 @@ SYM_CODE_START(arm64_relocate_new_kernel)
b.lo 2b
dsb sy
- mov x20, x13
- mov x21, x12
- copy_page x20, x21, x0, x1, x2, x3, x4, x5, x6, x7
-
- /* dest += PAGE_SIZE */
- add x13, x13, PAGE_SIZE
+ copy_page x13, x12, x0, x1, x2, x3, x4, x5, x6, x7
b .Lnext
-
.Ltest_indirection:
tbz x16, IND_INDIRECTION_BIT, .Ltest_destination
-
- /* ptr = addr */
- mov x14, x12
+ mov x14, x12 /* ptr = addr */
b .Lnext
-
.Ltest_destination:
tbz x16, IND_DESTINATION_BIT, .Lnext
-
- /* dest = addr */
- mov x13, x12
-
+ mov x13, x12 /* dest = addr */
.Lnext:
- /* entry = *ptr++ */
- ldr x16, [x14], #8
-
- /* while (!(entry & DONE)) */
- tbz x16, IND_DONE_BIT, .Lloop
-
+ ldr x16, [x14], #8 /* entry = *ptr++ */
+ tbz x16, IND_DONE_BIT, .Lloop /* while (!(entry & DONE)) */
.Ldone:
/* wait for writes from copy_page to finish */
dsb nsh
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 12/18] arm64: kexec: arm64_relocate_new_kernel don't use x0 as temp
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-25 19:19 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
x0 will contain the only argument to arm64_relocate_new_kernel; don't
use it as a temp. Reassigned registers to free-up x0 so we won't need
to copy argument, and can use it at the beginning and at the end of the
function.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: James Morse <james.morse@arm.com>
---
arch/arm64/kernel/relocate_kernel.S | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index 462ffbc37071..b78ea5de97a4 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -34,7 +34,7 @@ SYM_CODE_START(arm64_relocate_new_kernel)
mov x13, xzr /* x13 = copy dest */
/* Check if the new image needs relocation. */
tbnz x16, IND_DONE_BIT, .Ldone
- raw_dcache_line_size x15, x0 /* x15 = dcache line size */
+ raw_dcache_line_size x15, x1 /* x15 = dcache line size */
.Lloop:
and x12, x16, PAGE_MASK /* x12 = addr */
@@ -43,17 +43,17 @@ SYM_CODE_START(arm64_relocate_new_kernel)
tbz x16, IND_SOURCE_BIT, .Ltest_indirection
/* Invalidate dest page to PoC. */
- mov x0, x13
- add x20, x0, #PAGE_SIZE
+ mov x2, x13
+ add x20, x2, #PAGE_SIZE
sub x1, x15, #1
- bic x0, x0, x1
-2: dc ivac, x0
- add x0, x0, x15
- cmp x0, x20
+ bic x2, x2, x1
+2: dc ivac, x2
+ add x2, x2, x15
+ cmp x2, x20
b.lo 2b
dsb sy
- copy_page x13, x12, x0, x1, x2, x3, x4, x5, x6, x7
+ copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8
b .Lnext
.Ltest_indirection:
tbz x16, IND_INDIRECTION_BIT, .Ltest_destination
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 12/18] arm64: kexec: arm64_relocate_new_kernel don't use x0 as temp
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
x0 will contain the only argument to arm64_relocate_new_kernel; don't
use it as a temp. Reassigned registers to free-up x0 so we won't need
to copy argument, and can use it at the beginning and at the end of the
function.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: James Morse <james.morse@arm.com>
---
arch/arm64/kernel/relocate_kernel.S | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index 462ffbc37071..b78ea5de97a4 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -34,7 +34,7 @@ SYM_CODE_START(arm64_relocate_new_kernel)
mov x13, xzr /* x13 = copy dest */
/* Check if the new image needs relocation. */
tbnz x16, IND_DONE_BIT, .Ldone
- raw_dcache_line_size x15, x0 /* x15 = dcache line size */
+ raw_dcache_line_size x15, x1 /* x15 = dcache line size */
.Lloop:
and x12, x16, PAGE_MASK /* x12 = addr */
@@ -43,17 +43,17 @@ SYM_CODE_START(arm64_relocate_new_kernel)
tbz x16, IND_SOURCE_BIT, .Ltest_indirection
/* Invalidate dest page to PoC. */
- mov x0, x13
- add x20, x0, #PAGE_SIZE
+ mov x2, x13
+ add x20, x2, #PAGE_SIZE
sub x1, x15, #1
- bic x0, x0, x1
-2: dc ivac, x0
- add x0, x0, x15
- cmp x0, x20
+ bic x2, x2, x1
+2: dc ivac, x2
+ add x2, x2, x15
+ cmp x2, x20
b.lo 2b
dsb sy
- copy_page x13, x12, x0, x1, x2, x3, x4, x5, x6, x7
+ copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8
b .Lnext
.Ltest_indirection:
tbz x16, IND_INDIRECTION_BIT, .Ltest_destination
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 12/18] arm64: kexec: arm64_relocate_new_kernel don't use x0 as temp
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
x0 will contain the only argument to arm64_relocate_new_kernel; don't
use it as a temp. Reassigned registers to free-up x0 so we won't need
to copy argument, and can use it at the beginning and at the end of the
function.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
Reviewed-by: James Morse <james.morse@arm.com>
---
arch/arm64/kernel/relocate_kernel.S | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index 462ffbc37071..b78ea5de97a4 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -34,7 +34,7 @@ SYM_CODE_START(arm64_relocate_new_kernel)
mov x13, xzr /* x13 = copy dest */
/* Check if the new image needs relocation. */
tbnz x16, IND_DONE_BIT, .Ldone
- raw_dcache_line_size x15, x0 /* x15 = dcache line size */
+ raw_dcache_line_size x15, x1 /* x15 = dcache line size */
.Lloop:
and x12, x16, PAGE_MASK /* x12 = addr */
@@ -43,17 +43,17 @@ SYM_CODE_START(arm64_relocate_new_kernel)
tbz x16, IND_SOURCE_BIT, .Ltest_indirection
/* Invalidate dest page to PoC. */
- mov x0, x13
- add x20, x0, #PAGE_SIZE
+ mov x2, x13
+ add x20, x2, #PAGE_SIZE
sub x1, x15, #1
- bic x0, x0, x1
-2: dc ivac, x0
- add x0, x0, x15
- cmp x0, x20
+ bic x2, x2, x1
+2: dc ivac, x2
+ add x2, x2, x15
+ cmp x2, x20
b.lo 2b
dsb sy
- copy_page x13, x12, x0, x1, x2, x3, x4, x5, x6, x7
+ copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8
b .Lnext
.Ltest_indirection:
tbz x16, IND_INDIRECTION_BIT, .Ltest_destination
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 13/18] arm64: kexec: add expandable argument to relocation function
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-25 19:19 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Currently, kexec relocation function (arm64_relocate_new_kernel) accepts
the following arguments:
head: start of array that contains relocation information.
entry: entry point for new kernel or purgatory.
dtb_mem: first and only argument to entry.
The number of arguments cannot be easily expended, because this
function is also called from HVC_SOFT_RESTART, which preserves only
three arguments (hypervisor abi). And, also arm64_relocate_new_kernel is
written in assembly but called without stack, thus no place to move extra
arguments to free registers.
Soon, we will need to pass more arguments: once we enable MMU we
will need to pass information about page tables.
Add a new struct: kern_reloc_arg, and place it in kexec safe page (i.e
memory that is not overwritten during relocation).
Thus, make arm64_relocate_new_kernel to only take one argument, that
contains all the needed information.
Note:
Another benefit of allowing this function to accept more arguments, is that
kernel can actually accept up to 4 arguments (x0-x3), however currently
only one is used, but if in the future we will need for more (for example,
pass information about when previous kernel exited to have a precise
measurement in time spent in purgatory), we won't be easilty do that
if arm64_relocate_new_kernel can't accept more arguments.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/include/asm/kexec.h | 18 ++++++++++++++++++
arch/arm64/kernel/asm-offsets.c | 9 +++++++++
arch/arm64/kernel/cpu-reset.S | 11 +++--------
arch/arm64/kernel/cpu-reset.h | 8 +++-----
arch/arm64/kernel/machine_kexec.c | 27 +++++++++++++++++++++++++--
arch/arm64/kernel/relocate_kernel.S | 21 ++++++++-------------
6 files changed, 66 insertions(+), 28 deletions(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index 9befcd87e9a8..990185744148 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -90,12 +90,30 @@ static inline void crash_prepare_suspend(void) {}
static inline void crash_post_resume(void) {}
#endif
+/*
+ * kern_reloc_arg is passed to kernel relocation function as an argument.
+ * head kimage->head, allows to traverse through relocation segments.
+ * entry_addr kimage->start, where to jump from relocation function (new
+ * kernel, or purgatory entry address).
+ * kern_arg0 first argument to kernel is its dtb address. The other
+ * arguments are currently unused, and must be set to 0
+ */
+struct kern_reloc_arg {
+ phys_addr_t head;
+ phys_addr_t entry_addr;
+ phys_addr_t kern_arg0;
+ phys_addr_t kern_arg1;
+ phys_addr_t kern_arg2;
+ phys_addr_t kern_arg3;
+};
+
#define ARCH_HAS_KIMAGE_ARCH
struct kimage_arch {
void *dtb;
phys_addr_t dtb_mem;
phys_addr_t kern_reloc;
+ phys_addr_t kern_reloc_arg;
/* Core ELF header buffer */
void *elf_headers;
unsigned long elf_headers_mem;
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 301784463587..6067a288f568 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -23,6 +23,7 @@
#include <asm/suspend.h>
#include <linux/kbuild.h>
#include <linux/arm-smccc.h>
+#include <linux/kexec.h>
int main(void)
{
@@ -150,6 +151,14 @@ int main(void)
DEFINE(PTRAUTH_USER_KEY_APGA, offsetof(struct ptrauth_keys_user, apga));
DEFINE(PTRAUTH_KERNEL_KEY_APIA, offsetof(struct ptrauth_keys_kernel, apia));
BLANK();
+#endif
+#ifdef CONFIG_KEXEC_CORE
+ DEFINE(KEXEC_KRELOC_HEAD, offsetof(struct kern_reloc_arg, head));
+ DEFINE(KEXEC_KRELOC_ENTRY_ADDR, offsetof(struct kern_reloc_arg, entry_addr));
+ DEFINE(KEXEC_KRELOC_KERN_ARG0, offsetof(struct kern_reloc_arg, kern_arg0));
+ DEFINE(KEXEC_KRELOC_KERN_ARG1, offsetof(struct kern_reloc_arg, kern_arg1));
+ DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
+ DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
#endif
return 0;
}
diff --git a/arch/arm64/kernel/cpu-reset.S b/arch/arm64/kernel/cpu-reset.S
index 37721eb6f9a1..bbf70db43744 100644
--- a/arch/arm64/kernel/cpu-reset.S
+++ b/arch/arm64/kernel/cpu-reset.S
@@ -16,14 +16,11 @@
.pushsection .idmap.text, "awx"
/*
- * __cpu_soft_restart(el2_switch, entry, arg0, arg1, arg2) - Helper for
- * cpu_soft_restart.
+ * __cpu_soft_restart(el2_switch, entry, arg) - Helper for cpu_soft_restart.
*
* @el2_switch: Flag to indicate a switch to EL2 is needed.
* @entry: Location to jump to for soft reset.
- * arg0: First argument passed to @entry. (relocation list)
- * arg1: Second argument passed to @entry.(physical kernel entry)
- * arg2: Third argument passed to @entry. (physical dtb address)
+ * arg: Entry argument
*
* Put the CPU into the same state as it would be if it had been reset, and
* branch to what would be the reset vector. It must be executed with the
@@ -47,9 +44,7 @@ SYM_CODE_START(__cpu_soft_restart)
hvc #0 // no return
1: mov x8, x1 // entry
- mov x0, x2 // arg0
- mov x1, x3 // arg1
- mov x2, x4 // arg2
+ mov x0, x2 // arg
br x8
SYM_CODE_END(__cpu_soft_restart)
diff --git a/arch/arm64/kernel/cpu-reset.h b/arch/arm64/kernel/cpu-reset.h
index ed50e9587ad8..7a8720ff186f 100644
--- a/arch/arm64/kernel/cpu-reset.h
+++ b/arch/arm64/kernel/cpu-reset.h
@@ -11,12 +11,10 @@
#include <asm/virt.h>
void __cpu_soft_restart(unsigned long el2_switch, unsigned long entry,
- unsigned long arg0, unsigned long arg1, unsigned long arg2);
+ unsigned long arg);
static inline void __noreturn cpu_soft_restart(unsigned long entry,
- unsigned long arg0,
- unsigned long arg1,
- unsigned long arg2)
+ unsigned long arg)
{
typeof(__cpu_soft_restart) *restart;
@@ -25,7 +23,7 @@ static inline void __noreturn cpu_soft_restart(unsigned long entry,
restart = (void *)__pa_symbol(__cpu_soft_restart);
cpu_install_idmap();
- restart(el2_switch, entry, arg0, arg1, arg2);
+ restart(el2_switch, entry, arg);
unreachable();
}
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 90a335c74442..679db3f1e0c5 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -43,6 +43,7 @@ static void _kexec_image_info(const char *func, int line,
pr_debug(" head: %lx\n", kimage->head);
pr_debug(" nr_segments: %lu\n", kimage->nr_segments);
pr_debug(" kern_reloc: %pa\n", &kimage->arch.kern_reloc);
+ pr_debug(" kern_reloc_arg: %pa\n", &kimage->arch.kern_reloc_arg);
for (i = 0; i < kimage->nr_segments; i++) {
pr_debug(" segment[%lu]: %016lx - %016lx, 0x%lx bytes, %lu pages\n",
@@ -59,19 +60,42 @@ void machine_kexec_cleanup(struct kimage *kimage)
/* Empty routine needed to avoid build errors. */
}
+/* Allocates pages for kexec page table */
+static void *kexec_page_alloc(void *arg)
+{
+ struct kimage *kimage = (struct kimage *)arg;
+ struct page *page = kimage_alloc_control_pages(kimage, 0);
+
+ if (!page)
+ return NULL;
+
+ memset(page_address(page), 0, PAGE_SIZE);
+
+ return page_address(page);
+}
+
int machine_kexec_post_load(struct kimage *kimage)
{
void *reloc_code = page_to_virt(kimage->control_code_page);
+ struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage);
+
+ if (!kern_reloc_arg)
+ return -ENOMEM;
memcpy(reloc_code, arm64_relocate_new_kernel,
arm64_relocate_new_kernel_size);
kimage->arch.kern_reloc = __pa(reloc_code);
+ kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg);
+ kern_reloc_arg->head = kimage->head;
+ kern_reloc_arg->entry_addr = kimage->start;
+ kern_reloc_arg->kern_arg0 = kimage->arch.dtb_mem;
kexec_image_info(kimage);
/* Flush the reloc_code in preparation for its execution. */
__flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size);
flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code +
arm64_relocate_new_kernel_size);
+ __flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg));
return 0;
}
@@ -192,8 +216,7 @@ void machine_kexec(struct kimage *kimage)
* userspace (kexec-tools).
* In kexec_file case, the kernel starts directly without purgatory.
*/
- cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, kimage->start,
- kimage->arch.dtb_mem);
+ cpu_soft_restart(kimage->arch.kern_reloc, kimage->arch.kern_reloc_arg);
BUG(); /* Should never get here. */
}
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index b78ea5de97a4..c92228aeddca 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -8,7 +8,7 @@
#include <linux/kexec.h>
#include <linux/linkage.h>
-
+#include <asm/asm-offsets.h>
#include <asm/assembler.h>
#include <asm/kexec.h>
#include <asm/page.h>
@@ -26,13 +26,8 @@
* safe memory that has been set up to be preserved during the copy operation.
*/
SYM_CODE_START(arm64_relocate_new_kernel)
- /* Setup the list loop variables. */
- mov x18, x2 /* x18 = dtb address */
- mov x17, x1 /* x17 = kimage_start */
- mov x16, x0 /* x16 = kimage_head */
- mov x14, xzr /* x14 = entry ptr */
- mov x13, xzr /* x13 = copy dest */
/* Check if the new image needs relocation. */
+ ldr x16, [x0, #KEXEC_KRELOC_HEAD] /* x16 = kimage_head */
tbnz x16, IND_DONE_BIT, .Ldone
raw_dcache_line_size x15, x1 /* x15 = dcache line size */
.Lloop:
@@ -73,12 +68,12 @@ SYM_CODE_START(arm64_relocate_new_kernel)
isb
/* Start new image. */
- mov x0, x18
- mov x1, xzr
- mov x2, xzr
- mov x3, xzr
- br x17
-
+ ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 = kimage_start */
+ ldr x3, [x0, #KEXEC_KRELOC_KERN_ARG3]
+ ldr x2, [x0, #KEXEC_KRELOC_KERN_ARG2]
+ ldr x1, [x0, #KEXEC_KRELOC_KERN_ARG1]
+ ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */
+ br x4
SYM_CODE_END(arm64_relocate_new_kernel)
.align 3 /* To keep the 64-bit values below naturally aligned. */
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 13/18] arm64: kexec: add expandable argument to relocation function
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Currently, kexec relocation function (arm64_relocate_new_kernel) accepts
the following arguments:
head: start of array that contains relocation information.
entry: entry point for new kernel or purgatory.
dtb_mem: first and only argument to entry.
The number of arguments cannot be easily expended, because this
function is also called from HVC_SOFT_RESTART, which preserves only
three arguments (hypervisor abi). And, also arm64_relocate_new_kernel is
written in assembly but called without stack, thus no place to move extra
arguments to free registers.
Soon, we will need to pass more arguments: once we enable MMU we
will need to pass information about page tables.
Add a new struct: kern_reloc_arg, and place it in kexec safe page (i.e
memory that is not overwritten during relocation).
Thus, make arm64_relocate_new_kernel to only take one argument, that
contains all the needed information.
Note:
Another benefit of allowing this function to accept more arguments, is that
kernel can actually accept up to 4 arguments (x0-x3), however currently
only one is used, but if in the future we will need for more (for example,
pass information about when previous kernel exited to have a precise
measurement in time spent in purgatory), we won't be easilty do that
if arm64_relocate_new_kernel can't accept more arguments.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/include/asm/kexec.h | 18 ++++++++++++++++++
arch/arm64/kernel/asm-offsets.c | 9 +++++++++
arch/arm64/kernel/cpu-reset.S | 11 +++--------
arch/arm64/kernel/cpu-reset.h | 8 +++-----
arch/arm64/kernel/machine_kexec.c | 27 +++++++++++++++++++++++++--
arch/arm64/kernel/relocate_kernel.S | 21 ++++++++-------------
6 files changed, 66 insertions(+), 28 deletions(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index 9befcd87e9a8..990185744148 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -90,12 +90,30 @@ static inline void crash_prepare_suspend(void) {}
static inline void crash_post_resume(void) {}
#endif
+/*
+ * kern_reloc_arg is passed to kernel relocation function as an argument.
+ * head kimage->head, allows to traverse through relocation segments.
+ * entry_addr kimage->start, where to jump from relocation function (new
+ * kernel, or purgatory entry address).
+ * kern_arg0 first argument to kernel is its dtb address. The other
+ * arguments are currently unused, and must be set to 0
+ */
+struct kern_reloc_arg {
+ phys_addr_t head;
+ phys_addr_t entry_addr;
+ phys_addr_t kern_arg0;
+ phys_addr_t kern_arg1;
+ phys_addr_t kern_arg2;
+ phys_addr_t kern_arg3;
+};
+
#define ARCH_HAS_KIMAGE_ARCH
struct kimage_arch {
void *dtb;
phys_addr_t dtb_mem;
phys_addr_t kern_reloc;
+ phys_addr_t kern_reloc_arg;
/* Core ELF header buffer */
void *elf_headers;
unsigned long elf_headers_mem;
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 301784463587..6067a288f568 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -23,6 +23,7 @@
#include <asm/suspend.h>
#include <linux/kbuild.h>
#include <linux/arm-smccc.h>
+#include <linux/kexec.h>
int main(void)
{
@@ -150,6 +151,14 @@ int main(void)
DEFINE(PTRAUTH_USER_KEY_APGA, offsetof(struct ptrauth_keys_user, apga));
DEFINE(PTRAUTH_KERNEL_KEY_APIA, offsetof(struct ptrauth_keys_kernel, apia));
BLANK();
+#endif
+#ifdef CONFIG_KEXEC_CORE
+ DEFINE(KEXEC_KRELOC_HEAD, offsetof(struct kern_reloc_arg, head));
+ DEFINE(KEXEC_KRELOC_ENTRY_ADDR, offsetof(struct kern_reloc_arg, entry_addr));
+ DEFINE(KEXEC_KRELOC_KERN_ARG0, offsetof(struct kern_reloc_arg, kern_arg0));
+ DEFINE(KEXEC_KRELOC_KERN_ARG1, offsetof(struct kern_reloc_arg, kern_arg1));
+ DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
+ DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
#endif
return 0;
}
diff --git a/arch/arm64/kernel/cpu-reset.S b/arch/arm64/kernel/cpu-reset.S
index 37721eb6f9a1..bbf70db43744 100644
--- a/arch/arm64/kernel/cpu-reset.S
+++ b/arch/arm64/kernel/cpu-reset.S
@@ -16,14 +16,11 @@
.pushsection .idmap.text, "awx"
/*
- * __cpu_soft_restart(el2_switch, entry, arg0, arg1, arg2) - Helper for
- * cpu_soft_restart.
+ * __cpu_soft_restart(el2_switch, entry, arg) - Helper for cpu_soft_restart.
*
* @el2_switch: Flag to indicate a switch to EL2 is needed.
* @entry: Location to jump to for soft reset.
- * arg0: First argument passed to @entry. (relocation list)
- * arg1: Second argument passed to @entry.(physical kernel entry)
- * arg2: Third argument passed to @entry. (physical dtb address)
+ * arg: Entry argument
*
* Put the CPU into the same state as it would be if it had been reset, and
* branch to what would be the reset vector. It must be executed with the
@@ -47,9 +44,7 @@ SYM_CODE_START(__cpu_soft_restart)
hvc #0 // no return
1: mov x8, x1 // entry
- mov x0, x2 // arg0
- mov x1, x3 // arg1
- mov x2, x4 // arg2
+ mov x0, x2 // arg
br x8
SYM_CODE_END(__cpu_soft_restart)
diff --git a/arch/arm64/kernel/cpu-reset.h b/arch/arm64/kernel/cpu-reset.h
index ed50e9587ad8..7a8720ff186f 100644
--- a/arch/arm64/kernel/cpu-reset.h
+++ b/arch/arm64/kernel/cpu-reset.h
@@ -11,12 +11,10 @@
#include <asm/virt.h>
void __cpu_soft_restart(unsigned long el2_switch, unsigned long entry,
- unsigned long arg0, unsigned long arg1, unsigned long arg2);
+ unsigned long arg);
static inline void __noreturn cpu_soft_restart(unsigned long entry,
- unsigned long arg0,
- unsigned long arg1,
- unsigned long arg2)
+ unsigned long arg)
{
typeof(__cpu_soft_restart) *restart;
@@ -25,7 +23,7 @@ static inline void __noreturn cpu_soft_restart(unsigned long entry,
restart = (void *)__pa_symbol(__cpu_soft_restart);
cpu_install_idmap();
- restart(el2_switch, entry, arg0, arg1, arg2);
+ restart(el2_switch, entry, arg);
unreachable();
}
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 90a335c74442..679db3f1e0c5 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -43,6 +43,7 @@ static void _kexec_image_info(const char *func, int line,
pr_debug(" head: %lx\n", kimage->head);
pr_debug(" nr_segments: %lu\n", kimage->nr_segments);
pr_debug(" kern_reloc: %pa\n", &kimage->arch.kern_reloc);
+ pr_debug(" kern_reloc_arg: %pa\n", &kimage->arch.kern_reloc_arg);
for (i = 0; i < kimage->nr_segments; i++) {
pr_debug(" segment[%lu]: %016lx - %016lx, 0x%lx bytes, %lu pages\n",
@@ -59,19 +60,42 @@ void machine_kexec_cleanup(struct kimage *kimage)
/* Empty routine needed to avoid build errors. */
}
+/* Allocates pages for kexec page table */
+static void *kexec_page_alloc(void *arg)
+{
+ struct kimage *kimage = (struct kimage *)arg;
+ struct page *page = kimage_alloc_control_pages(kimage, 0);
+
+ if (!page)
+ return NULL;
+
+ memset(page_address(page), 0, PAGE_SIZE);
+
+ return page_address(page);
+}
+
int machine_kexec_post_load(struct kimage *kimage)
{
void *reloc_code = page_to_virt(kimage->control_code_page);
+ struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage);
+
+ if (!kern_reloc_arg)
+ return -ENOMEM;
memcpy(reloc_code, arm64_relocate_new_kernel,
arm64_relocate_new_kernel_size);
kimage->arch.kern_reloc = __pa(reloc_code);
+ kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg);
+ kern_reloc_arg->head = kimage->head;
+ kern_reloc_arg->entry_addr = kimage->start;
+ kern_reloc_arg->kern_arg0 = kimage->arch.dtb_mem;
kexec_image_info(kimage);
/* Flush the reloc_code in preparation for its execution. */
__flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size);
flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code +
arm64_relocate_new_kernel_size);
+ __flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg));
return 0;
}
@@ -192,8 +216,7 @@ void machine_kexec(struct kimage *kimage)
* userspace (kexec-tools).
* In kexec_file case, the kernel starts directly without purgatory.
*/
- cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, kimage->start,
- kimage->arch.dtb_mem);
+ cpu_soft_restart(kimage->arch.kern_reloc, kimage->arch.kern_reloc_arg);
BUG(); /* Should never get here. */
}
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index b78ea5de97a4..c92228aeddca 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -8,7 +8,7 @@
#include <linux/kexec.h>
#include <linux/linkage.h>
-
+#include <asm/asm-offsets.h>
#include <asm/assembler.h>
#include <asm/kexec.h>
#include <asm/page.h>
@@ -26,13 +26,8 @@
* safe memory that has been set up to be preserved during the copy operation.
*/
SYM_CODE_START(arm64_relocate_new_kernel)
- /* Setup the list loop variables. */
- mov x18, x2 /* x18 = dtb address */
- mov x17, x1 /* x17 = kimage_start */
- mov x16, x0 /* x16 = kimage_head */
- mov x14, xzr /* x14 = entry ptr */
- mov x13, xzr /* x13 = copy dest */
/* Check if the new image needs relocation. */
+ ldr x16, [x0, #KEXEC_KRELOC_HEAD] /* x16 = kimage_head */
tbnz x16, IND_DONE_BIT, .Ldone
raw_dcache_line_size x15, x1 /* x15 = dcache line size */
.Lloop:
@@ -73,12 +68,12 @@ SYM_CODE_START(arm64_relocate_new_kernel)
isb
/* Start new image. */
- mov x0, x18
- mov x1, xzr
- mov x2, xzr
- mov x3, xzr
- br x17
-
+ ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 = kimage_start */
+ ldr x3, [x0, #KEXEC_KRELOC_KERN_ARG3]
+ ldr x2, [x0, #KEXEC_KRELOC_KERN_ARG2]
+ ldr x1, [x0, #KEXEC_KRELOC_KERN_ARG1]
+ ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */
+ br x4
SYM_CODE_END(arm64_relocate_new_kernel)
.align 3 /* To keep the 64-bit values below naturally aligned. */
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 13/18] arm64: kexec: add expandable argument to relocation function
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Currently, kexec relocation function (arm64_relocate_new_kernel) accepts
the following arguments:
head: start of array that contains relocation information.
entry: entry point for new kernel or purgatory.
dtb_mem: first and only argument to entry.
The number of arguments cannot be easily expended, because this
function is also called from HVC_SOFT_RESTART, which preserves only
three arguments (hypervisor abi). And, also arm64_relocate_new_kernel is
written in assembly but called without stack, thus no place to move extra
arguments to free registers.
Soon, we will need to pass more arguments: once we enable MMU we
will need to pass information about page tables.
Add a new struct: kern_reloc_arg, and place it in kexec safe page (i.e
memory that is not overwritten during relocation).
Thus, make arm64_relocate_new_kernel to only take one argument, that
contains all the needed information.
Note:
Another benefit of allowing this function to accept more arguments, is that
kernel can actually accept up to 4 arguments (x0-x3), however currently
only one is used, but if in the future we will need for more (for example,
pass information about when previous kernel exited to have a precise
measurement in time spent in purgatory), we won't be easilty do that
if arm64_relocate_new_kernel can't accept more arguments.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/include/asm/kexec.h | 18 ++++++++++++++++++
arch/arm64/kernel/asm-offsets.c | 9 +++++++++
arch/arm64/kernel/cpu-reset.S | 11 +++--------
arch/arm64/kernel/cpu-reset.h | 8 +++-----
arch/arm64/kernel/machine_kexec.c | 27 +++++++++++++++++++++++++--
arch/arm64/kernel/relocate_kernel.S | 21 ++++++++-------------
6 files changed, 66 insertions(+), 28 deletions(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index 9befcd87e9a8..990185744148 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -90,12 +90,30 @@ static inline void crash_prepare_suspend(void) {}
static inline void crash_post_resume(void) {}
#endif
+/*
+ * kern_reloc_arg is passed to kernel relocation function as an argument.
+ * head kimage->head, allows to traverse through relocation segments.
+ * entry_addr kimage->start, where to jump from relocation function (new
+ * kernel, or purgatory entry address).
+ * kern_arg0 first argument to kernel is its dtb address. The other
+ * arguments are currently unused, and must be set to 0
+ */
+struct kern_reloc_arg {
+ phys_addr_t head;
+ phys_addr_t entry_addr;
+ phys_addr_t kern_arg0;
+ phys_addr_t kern_arg1;
+ phys_addr_t kern_arg2;
+ phys_addr_t kern_arg3;
+};
+
#define ARCH_HAS_KIMAGE_ARCH
struct kimage_arch {
void *dtb;
phys_addr_t dtb_mem;
phys_addr_t kern_reloc;
+ phys_addr_t kern_reloc_arg;
/* Core ELF header buffer */
void *elf_headers;
unsigned long elf_headers_mem;
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 301784463587..6067a288f568 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -23,6 +23,7 @@
#include <asm/suspend.h>
#include <linux/kbuild.h>
#include <linux/arm-smccc.h>
+#include <linux/kexec.h>
int main(void)
{
@@ -150,6 +151,14 @@ int main(void)
DEFINE(PTRAUTH_USER_KEY_APGA, offsetof(struct ptrauth_keys_user, apga));
DEFINE(PTRAUTH_KERNEL_KEY_APIA, offsetof(struct ptrauth_keys_kernel, apia));
BLANK();
+#endif
+#ifdef CONFIG_KEXEC_CORE
+ DEFINE(KEXEC_KRELOC_HEAD, offsetof(struct kern_reloc_arg, head));
+ DEFINE(KEXEC_KRELOC_ENTRY_ADDR, offsetof(struct kern_reloc_arg, entry_addr));
+ DEFINE(KEXEC_KRELOC_KERN_ARG0, offsetof(struct kern_reloc_arg, kern_arg0));
+ DEFINE(KEXEC_KRELOC_KERN_ARG1, offsetof(struct kern_reloc_arg, kern_arg1));
+ DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
+ DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
#endif
return 0;
}
diff --git a/arch/arm64/kernel/cpu-reset.S b/arch/arm64/kernel/cpu-reset.S
index 37721eb6f9a1..bbf70db43744 100644
--- a/arch/arm64/kernel/cpu-reset.S
+++ b/arch/arm64/kernel/cpu-reset.S
@@ -16,14 +16,11 @@
.pushsection .idmap.text, "awx"
/*
- * __cpu_soft_restart(el2_switch, entry, arg0, arg1, arg2) - Helper for
- * cpu_soft_restart.
+ * __cpu_soft_restart(el2_switch, entry, arg) - Helper for cpu_soft_restart.
*
* @el2_switch: Flag to indicate a switch to EL2 is needed.
* @entry: Location to jump to for soft reset.
- * arg0: First argument passed to @entry. (relocation list)
- * arg1: Second argument passed to @entry.(physical kernel entry)
- * arg2: Third argument passed to @entry. (physical dtb address)
+ * arg: Entry argument
*
* Put the CPU into the same state as it would be if it had been reset, and
* branch to what would be the reset vector. It must be executed with the
@@ -47,9 +44,7 @@ SYM_CODE_START(__cpu_soft_restart)
hvc #0 // no return
1: mov x8, x1 // entry
- mov x0, x2 // arg0
- mov x1, x3 // arg1
- mov x2, x4 // arg2
+ mov x0, x2 // arg
br x8
SYM_CODE_END(__cpu_soft_restart)
diff --git a/arch/arm64/kernel/cpu-reset.h b/arch/arm64/kernel/cpu-reset.h
index ed50e9587ad8..7a8720ff186f 100644
--- a/arch/arm64/kernel/cpu-reset.h
+++ b/arch/arm64/kernel/cpu-reset.h
@@ -11,12 +11,10 @@
#include <asm/virt.h>
void __cpu_soft_restart(unsigned long el2_switch, unsigned long entry,
- unsigned long arg0, unsigned long arg1, unsigned long arg2);
+ unsigned long arg);
static inline void __noreturn cpu_soft_restart(unsigned long entry,
- unsigned long arg0,
- unsigned long arg1,
- unsigned long arg2)
+ unsigned long arg)
{
typeof(__cpu_soft_restart) *restart;
@@ -25,7 +23,7 @@ static inline void __noreturn cpu_soft_restart(unsigned long entry,
restart = (void *)__pa_symbol(__cpu_soft_restart);
cpu_install_idmap();
- restart(el2_switch, entry, arg0, arg1, arg2);
+ restart(el2_switch, entry, arg);
unreachable();
}
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 90a335c74442..679db3f1e0c5 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -43,6 +43,7 @@ static void _kexec_image_info(const char *func, int line,
pr_debug(" head: %lx\n", kimage->head);
pr_debug(" nr_segments: %lu\n", kimage->nr_segments);
pr_debug(" kern_reloc: %pa\n", &kimage->arch.kern_reloc);
+ pr_debug(" kern_reloc_arg: %pa\n", &kimage->arch.kern_reloc_arg);
for (i = 0; i < kimage->nr_segments; i++) {
pr_debug(" segment[%lu]: %016lx - %016lx, 0x%lx bytes, %lu pages\n",
@@ -59,19 +60,42 @@ void machine_kexec_cleanup(struct kimage *kimage)
/* Empty routine needed to avoid build errors. */
}
+/* Allocates pages for kexec page table */
+static void *kexec_page_alloc(void *arg)
+{
+ struct kimage *kimage = (struct kimage *)arg;
+ struct page *page = kimage_alloc_control_pages(kimage, 0);
+
+ if (!page)
+ return NULL;
+
+ memset(page_address(page), 0, PAGE_SIZE);
+
+ return page_address(page);
+}
+
int machine_kexec_post_load(struct kimage *kimage)
{
void *reloc_code = page_to_virt(kimage->control_code_page);
+ struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage);
+
+ if (!kern_reloc_arg)
+ return -ENOMEM;
memcpy(reloc_code, arm64_relocate_new_kernel,
arm64_relocate_new_kernel_size);
kimage->arch.kern_reloc = __pa(reloc_code);
+ kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg);
+ kern_reloc_arg->head = kimage->head;
+ kern_reloc_arg->entry_addr = kimage->start;
+ kern_reloc_arg->kern_arg0 = kimage->arch.dtb_mem;
kexec_image_info(kimage);
/* Flush the reloc_code in preparation for its execution. */
__flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size);
flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code +
arm64_relocate_new_kernel_size);
+ __flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg));
return 0;
}
@@ -192,8 +216,7 @@ void machine_kexec(struct kimage *kimage)
* userspace (kexec-tools).
* In kexec_file case, the kernel starts directly without purgatory.
*/
- cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, kimage->start,
- kimage->arch.dtb_mem);
+ cpu_soft_restart(kimage->arch.kern_reloc, kimage->arch.kern_reloc_arg);
BUG(); /* Should never get here. */
}
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index b78ea5de97a4..c92228aeddca 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -8,7 +8,7 @@
#include <linux/kexec.h>
#include <linux/linkage.h>
-
+#include <asm/asm-offsets.h>
#include <asm/assembler.h>
#include <asm/kexec.h>
#include <asm/page.h>
@@ -26,13 +26,8 @@
* safe memory that has been set up to be preserved during the copy operation.
*/
SYM_CODE_START(arm64_relocate_new_kernel)
- /* Setup the list loop variables. */
- mov x18, x2 /* x18 = dtb address */
- mov x17, x1 /* x17 = kimage_start */
- mov x16, x0 /* x16 = kimage_head */
- mov x14, xzr /* x14 = entry ptr */
- mov x13, xzr /* x13 = copy dest */
/* Check if the new image needs relocation. */
+ ldr x16, [x0, #KEXEC_KRELOC_HEAD] /* x16 = kimage_head */
tbnz x16, IND_DONE_BIT, .Ldone
raw_dcache_line_size x15, x1 /* x15 = dcache line size */
.Lloop:
@@ -73,12 +68,12 @@ SYM_CODE_START(arm64_relocate_new_kernel)
isb
/* Start new image. */
- mov x0, x18
- mov x1, xzr
- mov x2, xzr
- mov x3, xzr
- br x17
-
+ ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 = kimage_start */
+ ldr x3, [x0, #KEXEC_KRELOC_KERN_ARG3]
+ ldr x2, [x0, #KEXEC_KRELOC_KERN_ARG2]
+ ldr x1, [x0, #KEXEC_KRELOC_KERN_ARG1]
+ ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */
+ br x4
SYM_CODE_END(arm64_relocate_new_kernel)
.align 3 /* To keep the 64-bit values below naturally aligned. */
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 14/18] arm64: kexec: use ld script for relocation function
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-25 19:19 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Currently, relocation code declares start and end variables
which are used to compute it size.
The better way to do this is to use ld script incited, and put relocation
function in its own section.
Soon, relocation function will share the same page with EL2 vectors. So,
proper marking is needed.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/include/asm/kexec.h | 4 ++++
arch/arm64/include/asm/sections.h | 1 +
arch/arm64/kernel/machine_kexec.c | 17 ++++++++---------
arch/arm64/kernel/relocate_kernel.S | 15 ++-------------
arch/arm64/kernel/vmlinux.lds.S | 19 +++++++++++++++++++
5 files changed, 34 insertions(+), 22 deletions(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index 990185744148..7f4f9abdf049 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -90,6 +90,10 @@ static inline void crash_prepare_suspend(void) {}
static inline void crash_post_resume(void) {}
#endif
+#if defined(CONFIG_KEXEC_CORE)
+extern const char arm64_relocate_new_kernel[];
+#endif
+
/*
* kern_reloc_arg is passed to kernel relocation function as an argument.
* head kimage->head, allows to traverse through relocation segments.
diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h
index 8ff579361731..ae873eb22205 100644
--- a/arch/arm64/include/asm/sections.h
+++ b/arch/arm64/include/asm/sections.h
@@ -19,5 +19,6 @@ extern char __exittext_begin[], __exittext_end[];
extern char __irqentry_text_start[], __irqentry_text_end[];
extern char __mmuoff_data_start[], __mmuoff_data_end[];
extern char __entry_tramp_text_start[], __entry_tramp_text_end[];
+extern char __relocate_new_kernel_start[], __relocate_new_kernel_end[];
#endif /* __ASM_SECTIONS_H */
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 679db3f1e0c5..361a4d082093 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -20,13 +20,10 @@
#include <asm/mmu.h>
#include <asm/mmu_context.h>
#include <asm/page.h>
+#include <asm/sections.h>
#include "cpu-reset.h"
-/* Global variables for the arm64_relocate_new_kernel routine. */
-extern const unsigned char arm64_relocate_new_kernel[];
-extern const unsigned long arm64_relocate_new_kernel_size;
-
/**
* kexec_image_info - For debugging output.
*/
@@ -78,13 +75,15 @@ int machine_kexec_post_load(struct kimage *kimage)
{
void *reloc_code = page_to_virt(kimage->control_code_page);
struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage);
+ long func_offset, reloc_size;
if (!kern_reloc_arg)
return -ENOMEM;
- memcpy(reloc_code, arm64_relocate_new_kernel,
- arm64_relocate_new_kernel_size);
- kimage->arch.kern_reloc = __pa(reloc_code);
+ func_offset = arm64_relocate_new_kernel - __relocate_new_kernel_start;
+ reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start;
+ memcpy(reloc_code, __relocate_new_kernel_start, reloc_size);
+ kimage->arch.kern_reloc = __pa(reloc_code) + func_offset;
kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg);
kern_reloc_arg->head = kimage->head;
kern_reloc_arg->entry_addr = kimage->start;
@@ -92,9 +91,9 @@ int machine_kexec_post_load(struct kimage *kimage)
kexec_image_info(kimage);
/* Flush the reloc_code in preparation for its execution. */
- __flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size);
+ __flush_dcache_area(reloc_code, reloc_size);
flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code +
- arm64_relocate_new_kernel_size);
+ reloc_size);
__flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg));
return 0;
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index c92228aeddca..d2a4a0b0d76b 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -14,6 +14,7 @@
#include <asm/page.h>
#include <asm/sysreg.h>
+.pushsection ".kexec_relocate.text", "ax"
/*
* arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it.
*
@@ -75,16 +76,4 @@ SYM_CODE_START(arm64_relocate_new_kernel)
ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */
br x4
SYM_CODE_END(arm64_relocate_new_kernel)
-
-.align 3 /* To keep the 64-bit values below naturally aligned. */
-
-.Lcopy_end:
-.org KEXEC_CONTROL_PAGE_SIZE
-
-/*
- * arm64_relocate_new_kernel_size - Number of bytes to copy to the
- * control_code_page.
- */
-.globl arm64_relocate_new_kernel_size
-arm64_relocate_new_kernel_size:
- .quad .Lcopy_end - arm64_relocate_new_kernel
+.popsection
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index 4c0b0c89ad59..33b0d3c9fd3b 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -12,6 +12,7 @@
#include <asm/cache.h>
#include <asm/hyp_image.h>
#include <asm/kernel-pgtable.h>
+#include <asm/kexec.h>
#include <asm/memory.h>
#include <asm/page.h>
@@ -82,6 +83,16 @@ jiffies = jiffies_64;
#define HIBERNATE_TEXT
#endif
+#ifdef CONFIG_KEXEC_CORE
+#define KEXEC_TEXT \
+ . = ALIGN(SZ_4K); \
+ __relocate_new_kernel_start = .; \
+ *(.kexec_relocate.text) \
+ __relocate_new_kernel_end = .;
+#else
+#define KEXEC_TEXT
+#endif
+
#ifdef CONFIG_UNMAP_KERNEL_AT_EL0
#define TRAMP_TEXT \
. = ALIGN(PAGE_SIZE); \
@@ -142,6 +153,7 @@ SECTIONS
HYPERVISOR_TEXT
IDMAP_TEXT
HIBERNATE_TEXT
+ KEXEC_TEXT
TRAMP_TEXT
*(.fixup)
*(.gnu.warning)
@@ -316,3 +328,10 @@ ASSERT((__entry_tramp_text_end - __entry_tramp_text_start) == PAGE_SIZE,
* If padding is applied before .head.text, virt<->phys conversions will fail.
*/
ASSERT(_text == KIMAGE_VADDR, "HEAD is misaligned")
+
+#ifdef CONFIG_KEXEC_CORE
+/* kexec relocation code should fit into one KEXEC_CONTROL_PAGE_SIZE */
+ASSERT(__relocate_new_kernel_end - (__relocate_new_kernel_start & ~(SZ_4K - 1))
+ <= SZ_4K, "kexec relocation code is too big or misaligned")
+ASSERT(KEXEC_CONTROL_PAGE_SIZE >= SZ_4K, "KEXEC_CONTROL_PAGE_SIZE is brokern")
+#endif
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 14/18] arm64: kexec: use ld script for relocation function
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Currently, relocation code declares start and end variables
which are used to compute it size.
The better way to do this is to use ld script incited, and put relocation
function in its own section.
Soon, relocation function will share the same page with EL2 vectors. So,
proper marking is needed.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/include/asm/kexec.h | 4 ++++
arch/arm64/include/asm/sections.h | 1 +
arch/arm64/kernel/machine_kexec.c | 17 ++++++++---------
arch/arm64/kernel/relocate_kernel.S | 15 ++-------------
arch/arm64/kernel/vmlinux.lds.S | 19 +++++++++++++++++++
5 files changed, 34 insertions(+), 22 deletions(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index 990185744148..7f4f9abdf049 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -90,6 +90,10 @@ static inline void crash_prepare_suspend(void) {}
static inline void crash_post_resume(void) {}
#endif
+#if defined(CONFIG_KEXEC_CORE)
+extern const char arm64_relocate_new_kernel[];
+#endif
+
/*
* kern_reloc_arg is passed to kernel relocation function as an argument.
* head kimage->head, allows to traverse through relocation segments.
diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h
index 8ff579361731..ae873eb22205 100644
--- a/arch/arm64/include/asm/sections.h
+++ b/arch/arm64/include/asm/sections.h
@@ -19,5 +19,6 @@ extern char __exittext_begin[], __exittext_end[];
extern char __irqentry_text_start[], __irqentry_text_end[];
extern char __mmuoff_data_start[], __mmuoff_data_end[];
extern char __entry_tramp_text_start[], __entry_tramp_text_end[];
+extern char __relocate_new_kernel_start[], __relocate_new_kernel_end[];
#endif /* __ASM_SECTIONS_H */
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 679db3f1e0c5..361a4d082093 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -20,13 +20,10 @@
#include <asm/mmu.h>
#include <asm/mmu_context.h>
#include <asm/page.h>
+#include <asm/sections.h>
#include "cpu-reset.h"
-/* Global variables for the arm64_relocate_new_kernel routine. */
-extern const unsigned char arm64_relocate_new_kernel[];
-extern const unsigned long arm64_relocate_new_kernel_size;
-
/**
* kexec_image_info - For debugging output.
*/
@@ -78,13 +75,15 @@ int machine_kexec_post_load(struct kimage *kimage)
{
void *reloc_code = page_to_virt(kimage->control_code_page);
struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage);
+ long func_offset, reloc_size;
if (!kern_reloc_arg)
return -ENOMEM;
- memcpy(reloc_code, arm64_relocate_new_kernel,
- arm64_relocate_new_kernel_size);
- kimage->arch.kern_reloc = __pa(reloc_code);
+ func_offset = arm64_relocate_new_kernel - __relocate_new_kernel_start;
+ reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start;
+ memcpy(reloc_code, __relocate_new_kernel_start, reloc_size);
+ kimage->arch.kern_reloc = __pa(reloc_code) + func_offset;
kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg);
kern_reloc_arg->head = kimage->head;
kern_reloc_arg->entry_addr = kimage->start;
@@ -92,9 +91,9 @@ int machine_kexec_post_load(struct kimage *kimage)
kexec_image_info(kimage);
/* Flush the reloc_code in preparation for its execution. */
- __flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size);
+ __flush_dcache_area(reloc_code, reloc_size);
flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code +
- arm64_relocate_new_kernel_size);
+ reloc_size);
__flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg));
return 0;
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index c92228aeddca..d2a4a0b0d76b 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -14,6 +14,7 @@
#include <asm/page.h>
#include <asm/sysreg.h>
+.pushsection ".kexec_relocate.text", "ax"
/*
* arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it.
*
@@ -75,16 +76,4 @@ SYM_CODE_START(arm64_relocate_new_kernel)
ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */
br x4
SYM_CODE_END(arm64_relocate_new_kernel)
-
-.align 3 /* To keep the 64-bit values below naturally aligned. */
-
-.Lcopy_end:
-.org KEXEC_CONTROL_PAGE_SIZE
-
-/*
- * arm64_relocate_new_kernel_size - Number of bytes to copy to the
- * control_code_page.
- */
-.globl arm64_relocate_new_kernel_size
-arm64_relocate_new_kernel_size:
- .quad .Lcopy_end - arm64_relocate_new_kernel
+.popsection
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index 4c0b0c89ad59..33b0d3c9fd3b 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -12,6 +12,7 @@
#include <asm/cache.h>
#include <asm/hyp_image.h>
#include <asm/kernel-pgtable.h>
+#include <asm/kexec.h>
#include <asm/memory.h>
#include <asm/page.h>
@@ -82,6 +83,16 @@ jiffies = jiffies_64;
#define HIBERNATE_TEXT
#endif
+#ifdef CONFIG_KEXEC_CORE
+#define KEXEC_TEXT \
+ . = ALIGN(SZ_4K); \
+ __relocate_new_kernel_start = .; \
+ *(.kexec_relocate.text) \
+ __relocate_new_kernel_end = .;
+#else
+#define KEXEC_TEXT
+#endif
+
#ifdef CONFIG_UNMAP_KERNEL_AT_EL0
#define TRAMP_TEXT \
. = ALIGN(PAGE_SIZE); \
@@ -142,6 +153,7 @@ SECTIONS
HYPERVISOR_TEXT
IDMAP_TEXT
HIBERNATE_TEXT
+ KEXEC_TEXT
TRAMP_TEXT
*(.fixup)
*(.gnu.warning)
@@ -316,3 +328,10 @@ ASSERT((__entry_tramp_text_end - __entry_tramp_text_start) == PAGE_SIZE,
* If padding is applied before .head.text, virt<->phys conversions will fail.
*/
ASSERT(_text == KIMAGE_VADDR, "HEAD is misaligned")
+
+#ifdef CONFIG_KEXEC_CORE
+/* kexec relocation code should fit into one KEXEC_CONTROL_PAGE_SIZE */
+ASSERT(__relocate_new_kernel_end - (__relocate_new_kernel_start & ~(SZ_4K - 1))
+ <= SZ_4K, "kexec relocation code is too big or misaligned")
+ASSERT(KEXEC_CONTROL_PAGE_SIZE >= SZ_4K, "KEXEC_CONTROL_PAGE_SIZE is brokern")
+#endif
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 14/18] arm64: kexec: use ld script for relocation function
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Currently, relocation code declares start and end variables
which are used to compute it size.
The better way to do this is to use ld script incited, and put relocation
function in its own section.
Soon, relocation function will share the same page with EL2 vectors. So,
proper marking is needed.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/include/asm/kexec.h | 4 ++++
arch/arm64/include/asm/sections.h | 1 +
arch/arm64/kernel/machine_kexec.c | 17 ++++++++---------
arch/arm64/kernel/relocate_kernel.S | 15 ++-------------
arch/arm64/kernel/vmlinux.lds.S | 19 +++++++++++++++++++
5 files changed, 34 insertions(+), 22 deletions(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index 990185744148..7f4f9abdf049 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -90,6 +90,10 @@ static inline void crash_prepare_suspend(void) {}
static inline void crash_post_resume(void) {}
#endif
+#if defined(CONFIG_KEXEC_CORE)
+extern const char arm64_relocate_new_kernel[];
+#endif
+
/*
* kern_reloc_arg is passed to kernel relocation function as an argument.
* head kimage->head, allows to traverse through relocation segments.
diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h
index 8ff579361731..ae873eb22205 100644
--- a/arch/arm64/include/asm/sections.h
+++ b/arch/arm64/include/asm/sections.h
@@ -19,5 +19,6 @@ extern char __exittext_begin[], __exittext_end[];
extern char __irqentry_text_start[], __irqentry_text_end[];
extern char __mmuoff_data_start[], __mmuoff_data_end[];
extern char __entry_tramp_text_start[], __entry_tramp_text_end[];
+extern char __relocate_new_kernel_start[], __relocate_new_kernel_end[];
#endif /* __ASM_SECTIONS_H */
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 679db3f1e0c5..361a4d082093 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -20,13 +20,10 @@
#include <asm/mmu.h>
#include <asm/mmu_context.h>
#include <asm/page.h>
+#include <asm/sections.h>
#include "cpu-reset.h"
-/* Global variables for the arm64_relocate_new_kernel routine. */
-extern const unsigned char arm64_relocate_new_kernel[];
-extern const unsigned long arm64_relocate_new_kernel_size;
-
/**
* kexec_image_info - For debugging output.
*/
@@ -78,13 +75,15 @@ int machine_kexec_post_load(struct kimage *kimage)
{
void *reloc_code = page_to_virt(kimage->control_code_page);
struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage);
+ long func_offset, reloc_size;
if (!kern_reloc_arg)
return -ENOMEM;
- memcpy(reloc_code, arm64_relocate_new_kernel,
- arm64_relocate_new_kernel_size);
- kimage->arch.kern_reloc = __pa(reloc_code);
+ func_offset = arm64_relocate_new_kernel - __relocate_new_kernel_start;
+ reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start;
+ memcpy(reloc_code, __relocate_new_kernel_start, reloc_size);
+ kimage->arch.kern_reloc = __pa(reloc_code) + func_offset;
kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg);
kern_reloc_arg->head = kimage->head;
kern_reloc_arg->entry_addr = kimage->start;
@@ -92,9 +91,9 @@ int machine_kexec_post_load(struct kimage *kimage)
kexec_image_info(kimage);
/* Flush the reloc_code in preparation for its execution. */
- __flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size);
+ __flush_dcache_area(reloc_code, reloc_size);
flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code +
- arm64_relocate_new_kernel_size);
+ reloc_size);
__flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg));
return 0;
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index c92228aeddca..d2a4a0b0d76b 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -14,6 +14,7 @@
#include <asm/page.h>
#include <asm/sysreg.h>
+.pushsection ".kexec_relocate.text", "ax"
/*
* arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it.
*
@@ -75,16 +76,4 @@ SYM_CODE_START(arm64_relocate_new_kernel)
ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */
br x4
SYM_CODE_END(arm64_relocate_new_kernel)
-
-.align 3 /* To keep the 64-bit values below naturally aligned. */
-
-.Lcopy_end:
-.org KEXEC_CONTROL_PAGE_SIZE
-
-/*
- * arm64_relocate_new_kernel_size - Number of bytes to copy to the
- * control_code_page.
- */
-.globl arm64_relocate_new_kernel_size
-arm64_relocate_new_kernel_size:
- .quad .Lcopy_end - arm64_relocate_new_kernel
+.popsection
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index 4c0b0c89ad59..33b0d3c9fd3b 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -12,6 +12,7 @@
#include <asm/cache.h>
#include <asm/hyp_image.h>
#include <asm/kernel-pgtable.h>
+#include <asm/kexec.h>
#include <asm/memory.h>
#include <asm/page.h>
@@ -82,6 +83,16 @@ jiffies = jiffies_64;
#define HIBERNATE_TEXT
#endif
+#ifdef CONFIG_KEXEC_CORE
+#define KEXEC_TEXT \
+ . = ALIGN(SZ_4K); \
+ __relocate_new_kernel_start = .; \
+ *(.kexec_relocate.text) \
+ __relocate_new_kernel_end = .;
+#else
+#define KEXEC_TEXT
+#endif
+
#ifdef CONFIG_UNMAP_KERNEL_AT_EL0
#define TRAMP_TEXT \
. = ALIGN(PAGE_SIZE); \
@@ -142,6 +153,7 @@ SECTIONS
HYPERVISOR_TEXT
IDMAP_TEXT
HIBERNATE_TEXT
+ KEXEC_TEXT
TRAMP_TEXT
*(.fixup)
*(.gnu.warning)
@@ -316,3 +328,10 @@ ASSERT((__entry_tramp_text_end - __entry_tramp_text_start) == PAGE_SIZE,
* If padding is applied before .head.text, virt<->phys conversions will fail.
*/
ASSERT(_text == KIMAGE_VADDR, "HEAD is misaligned")
+
+#ifdef CONFIG_KEXEC_CORE
+/* kexec relocation code should fit into one KEXEC_CONTROL_PAGE_SIZE */
+ASSERT(__relocate_new_kernel_end - (__relocate_new_kernel_start & ~(SZ_4K - 1))
+ <= SZ_4K, "kexec relocation code is too big or misaligned")
+ASSERT(KEXEC_CONTROL_PAGE_SIZE >= SZ_4K, "KEXEC_CONTROL_PAGE_SIZE is brokern")
+#endif
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 15/18] arm64: kexec: kexec may require EL2 vectors
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-25 19:19 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
If we have a EL2 mode without VHE, the EL2 vectors are needed in order
to switch to EL2 and jump to new world with hypervisor privileges.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/include/asm/kexec.h | 5 +++++
arch/arm64/kernel/asm-offsets.c | 1 +
arch/arm64/kernel/machine_kexec.c | 9 +++++++-
arch/arm64/kernel/relocate_kernel.S | 35 +++++++++++++++++++++++++++++
4 files changed, 49 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index 7f4f9abdf049..b96d8a6aac80 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -92,6 +92,7 @@ static inline void crash_post_resume(void) {}
#if defined(CONFIG_KEXEC_CORE)
extern const char arm64_relocate_new_kernel[];
+extern const char arm64_kexec_el2_vectors[];
#endif
/*
@@ -101,6 +102,9 @@ extern const char arm64_relocate_new_kernel[];
* kernel, or purgatory entry address).
* kern_arg0 first argument to kernel is its dtb address. The other
* arguments are currently unused, and must be set to 0
+ * el2_vector If present means that relocation routine will go to EL1
+ * from EL2 to do the copy, and then back to EL2 to do the jump
+ * to new world.
*/
struct kern_reloc_arg {
phys_addr_t head;
@@ -109,6 +113,7 @@ struct kern_reloc_arg {
phys_addr_t kern_arg1;
phys_addr_t kern_arg2;
phys_addr_t kern_arg3;
+ phys_addr_t el2_vector;
};
#define ARCH_HAS_KIMAGE_ARCH
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 6067a288f568..8a9475be1b62 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -159,6 +159,7 @@ int main(void)
DEFINE(KEXEC_KRELOC_KERN_ARG1, offsetof(struct kern_reloc_arg, kern_arg1));
DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
+ DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector));
#endif
return 0;
}
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 361a4d082093..41d1e3ca13f8 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -75,19 +75,26 @@ int machine_kexec_post_load(struct kimage *kimage)
{
void *reloc_code = page_to_virt(kimage->control_code_page);
struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage);
- long func_offset, reloc_size;
+ long func_offset, vector_offset, reloc_size;
if (!kern_reloc_arg)
return -ENOMEM;
func_offset = arm64_relocate_new_kernel - __relocate_new_kernel_start;
reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start;
+ vector_offset = arm64_kexec_el2_vectors - __relocate_new_kernel_start;
+
memcpy(reloc_code, __relocate_new_kernel_start, reloc_size);
kimage->arch.kern_reloc = __pa(reloc_code) + func_offset;
kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg);
kern_reloc_arg->head = kimage->head;
kern_reloc_arg->entry_addr = kimage->start;
kern_reloc_arg->kern_arg0 = kimage->arch.dtb_mem;
+
+ /* Setup vector table only when EL2 is available, but no VHE */
+ if (is_hyp_mode_available() && !is_kernel_in_hyp_mode())
+ kern_reloc_arg->el2_vector = __pa(reloc_code) + vector_offset;
+
kexec_image_info(kimage);
/* Flush the reloc_code in preparation for its execution. */
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index d2a4a0b0d76b..c6178b1a4e60 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -14,6 +14,17 @@
#include <asm/page.h>
#include <asm/sysreg.h>
+.macro el1_sync_64
+ .align 7
+ br x4 /* Jump to new world from el2 */
+.endm
+
+.macro invalid_vector label
+\label:
+ .align 7
+ b \label
+.endm
+
.pushsection ".kexec_relocate.text", "ax"
/*
* arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it.
@@ -76,4 +87,28 @@ SYM_CODE_START(arm64_relocate_new_kernel)
ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */
br x4
SYM_CODE_END(arm64_relocate_new_kernel)
+
+/* el2 vectors - switch el2 here while we restore the memory image. */
+ .align 11
+SYM_CODE_START(arm64_kexec_el2_vectors)
+ invalid_vector el2_sync_invalid_sp0 /* Synchronous EL2t */
+ invalid_vector el2_irq_invalid_sp0 /* IRQ EL2t */
+ invalid_vector el2_fiq_invalid_sp0 /* FIQ EL2t */
+ invalid_vector el2_error_invalid_sp0 /* Error EL2t */
+
+ invalid_vector el2_sync_invalid_spx /* Synchronous EL2h */
+ invalid_vector el2_irq_invalid_spx /* IRQ EL2h */
+ invalid_vector el2_fiq_invalid_spx /* FIQ EL2h */
+ invalid_vector el2_error_invalid_spx /* Error EL2h */
+
+ el1_sync_64 /* Synchronous 64-bit EL1 */
+ invalid_vector el1_irq_invalid_64 /* IRQ 64-bit EL1 */
+ invalid_vector el1_fiq_invalid_64 /* FIQ 64-bit EL1 */
+ invalid_vector el1_error_invalid_64 /* Error 64-bit EL1 */
+
+ invalid_vector el1_sync_invalid_32 /* Synchronous 32-bit EL1 */
+ invalid_vector el1_irq_invalid_32 /* IRQ 32-bit EL1 */
+ invalid_vector el1_fiq_invalid_32 /* FIQ 32-bit EL1 */
+ invalid_vector el1_error_invalid_32 /* Error 32-bit EL1 */
+SYM_CODE_END(arm64_kexec_el2_vectors)
.popsection
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 15/18] arm64: kexec: kexec may require EL2 vectors
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
If we have a EL2 mode without VHE, the EL2 vectors are needed in order
to switch to EL2 and jump to new world with hypervisor privileges.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/include/asm/kexec.h | 5 +++++
arch/arm64/kernel/asm-offsets.c | 1 +
arch/arm64/kernel/machine_kexec.c | 9 +++++++-
arch/arm64/kernel/relocate_kernel.S | 35 +++++++++++++++++++++++++++++
4 files changed, 49 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index 7f4f9abdf049..b96d8a6aac80 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -92,6 +92,7 @@ static inline void crash_post_resume(void) {}
#if defined(CONFIG_KEXEC_CORE)
extern const char arm64_relocate_new_kernel[];
+extern const char arm64_kexec_el2_vectors[];
#endif
/*
@@ -101,6 +102,9 @@ extern const char arm64_relocate_new_kernel[];
* kernel, or purgatory entry address).
* kern_arg0 first argument to kernel is its dtb address. The other
* arguments are currently unused, and must be set to 0
+ * el2_vector If present means that relocation routine will go to EL1
+ * from EL2 to do the copy, and then back to EL2 to do the jump
+ * to new world.
*/
struct kern_reloc_arg {
phys_addr_t head;
@@ -109,6 +113,7 @@ struct kern_reloc_arg {
phys_addr_t kern_arg1;
phys_addr_t kern_arg2;
phys_addr_t kern_arg3;
+ phys_addr_t el2_vector;
};
#define ARCH_HAS_KIMAGE_ARCH
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 6067a288f568..8a9475be1b62 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -159,6 +159,7 @@ int main(void)
DEFINE(KEXEC_KRELOC_KERN_ARG1, offsetof(struct kern_reloc_arg, kern_arg1));
DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
+ DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector));
#endif
return 0;
}
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 361a4d082093..41d1e3ca13f8 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -75,19 +75,26 @@ int machine_kexec_post_load(struct kimage *kimage)
{
void *reloc_code = page_to_virt(kimage->control_code_page);
struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage);
- long func_offset, reloc_size;
+ long func_offset, vector_offset, reloc_size;
if (!kern_reloc_arg)
return -ENOMEM;
func_offset = arm64_relocate_new_kernel - __relocate_new_kernel_start;
reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start;
+ vector_offset = arm64_kexec_el2_vectors - __relocate_new_kernel_start;
+
memcpy(reloc_code, __relocate_new_kernel_start, reloc_size);
kimage->arch.kern_reloc = __pa(reloc_code) + func_offset;
kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg);
kern_reloc_arg->head = kimage->head;
kern_reloc_arg->entry_addr = kimage->start;
kern_reloc_arg->kern_arg0 = kimage->arch.dtb_mem;
+
+ /* Setup vector table only when EL2 is available, but no VHE */
+ if (is_hyp_mode_available() && !is_kernel_in_hyp_mode())
+ kern_reloc_arg->el2_vector = __pa(reloc_code) + vector_offset;
+
kexec_image_info(kimage);
/* Flush the reloc_code in preparation for its execution. */
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index d2a4a0b0d76b..c6178b1a4e60 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -14,6 +14,17 @@
#include <asm/page.h>
#include <asm/sysreg.h>
+.macro el1_sync_64
+ .align 7
+ br x4 /* Jump to new world from el2 */
+.endm
+
+.macro invalid_vector label
+\label:
+ .align 7
+ b \label
+.endm
+
.pushsection ".kexec_relocate.text", "ax"
/*
* arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it.
@@ -76,4 +87,28 @@ SYM_CODE_START(arm64_relocate_new_kernel)
ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */
br x4
SYM_CODE_END(arm64_relocate_new_kernel)
+
+/* el2 vectors - switch el2 here while we restore the memory image. */
+ .align 11
+SYM_CODE_START(arm64_kexec_el2_vectors)
+ invalid_vector el2_sync_invalid_sp0 /* Synchronous EL2t */
+ invalid_vector el2_irq_invalid_sp0 /* IRQ EL2t */
+ invalid_vector el2_fiq_invalid_sp0 /* FIQ EL2t */
+ invalid_vector el2_error_invalid_sp0 /* Error EL2t */
+
+ invalid_vector el2_sync_invalid_spx /* Synchronous EL2h */
+ invalid_vector el2_irq_invalid_spx /* IRQ EL2h */
+ invalid_vector el2_fiq_invalid_spx /* FIQ EL2h */
+ invalid_vector el2_error_invalid_spx /* Error EL2h */
+
+ el1_sync_64 /* Synchronous 64-bit EL1 */
+ invalid_vector el1_irq_invalid_64 /* IRQ 64-bit EL1 */
+ invalid_vector el1_fiq_invalid_64 /* FIQ 64-bit EL1 */
+ invalid_vector el1_error_invalid_64 /* Error 64-bit EL1 */
+
+ invalid_vector el1_sync_invalid_32 /* Synchronous 32-bit EL1 */
+ invalid_vector el1_irq_invalid_32 /* IRQ 32-bit EL1 */
+ invalid_vector el1_fiq_invalid_32 /* FIQ 32-bit EL1 */
+ invalid_vector el1_error_invalid_32 /* Error 32-bit EL1 */
+SYM_CODE_END(arm64_kexec_el2_vectors)
.popsection
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 15/18] arm64: kexec: kexec may require EL2 vectors
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
If we have a EL2 mode without VHE, the EL2 vectors are needed in order
to switch to EL2 and jump to new world with hypervisor privileges.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/include/asm/kexec.h | 5 +++++
arch/arm64/kernel/asm-offsets.c | 1 +
arch/arm64/kernel/machine_kexec.c | 9 +++++++-
arch/arm64/kernel/relocate_kernel.S | 35 +++++++++++++++++++++++++++++
4 files changed, 49 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index 7f4f9abdf049..b96d8a6aac80 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -92,6 +92,7 @@ static inline void crash_post_resume(void) {}
#if defined(CONFIG_KEXEC_CORE)
extern const char arm64_relocate_new_kernel[];
+extern const char arm64_kexec_el2_vectors[];
#endif
/*
@@ -101,6 +102,9 @@ extern const char arm64_relocate_new_kernel[];
* kernel, or purgatory entry address).
* kern_arg0 first argument to kernel is its dtb address. The other
* arguments are currently unused, and must be set to 0
+ * el2_vector If present means that relocation routine will go to EL1
+ * from EL2 to do the copy, and then back to EL2 to do the jump
+ * to new world.
*/
struct kern_reloc_arg {
phys_addr_t head;
@@ -109,6 +113,7 @@ struct kern_reloc_arg {
phys_addr_t kern_arg1;
phys_addr_t kern_arg2;
phys_addr_t kern_arg3;
+ phys_addr_t el2_vector;
};
#define ARCH_HAS_KIMAGE_ARCH
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 6067a288f568..8a9475be1b62 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -159,6 +159,7 @@ int main(void)
DEFINE(KEXEC_KRELOC_KERN_ARG1, offsetof(struct kern_reloc_arg, kern_arg1));
DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
+ DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector));
#endif
return 0;
}
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 361a4d082093..41d1e3ca13f8 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -75,19 +75,26 @@ int machine_kexec_post_load(struct kimage *kimage)
{
void *reloc_code = page_to_virt(kimage->control_code_page);
struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage);
- long func_offset, reloc_size;
+ long func_offset, vector_offset, reloc_size;
if (!kern_reloc_arg)
return -ENOMEM;
func_offset = arm64_relocate_new_kernel - __relocate_new_kernel_start;
reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start;
+ vector_offset = arm64_kexec_el2_vectors - __relocate_new_kernel_start;
+
memcpy(reloc_code, __relocate_new_kernel_start, reloc_size);
kimage->arch.kern_reloc = __pa(reloc_code) + func_offset;
kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg);
kern_reloc_arg->head = kimage->head;
kern_reloc_arg->entry_addr = kimage->start;
kern_reloc_arg->kern_arg0 = kimage->arch.dtb_mem;
+
+ /* Setup vector table only when EL2 is available, but no VHE */
+ if (is_hyp_mode_available() && !is_kernel_in_hyp_mode())
+ kern_reloc_arg->el2_vector = __pa(reloc_code) + vector_offset;
+
kexec_image_info(kimage);
/* Flush the reloc_code in preparation for its execution. */
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index d2a4a0b0d76b..c6178b1a4e60 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -14,6 +14,17 @@
#include <asm/page.h>
#include <asm/sysreg.h>
+.macro el1_sync_64
+ .align 7
+ br x4 /* Jump to new world from el2 */
+.endm
+
+.macro invalid_vector label
+\label:
+ .align 7
+ b \label
+.endm
+
.pushsection ".kexec_relocate.text", "ax"
/*
* arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it.
@@ -76,4 +87,28 @@ SYM_CODE_START(arm64_relocate_new_kernel)
ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */
br x4
SYM_CODE_END(arm64_relocate_new_kernel)
+
+/* el2 vectors - switch el2 here while we restore the memory image. */
+ .align 11
+SYM_CODE_START(arm64_kexec_el2_vectors)
+ invalid_vector el2_sync_invalid_sp0 /* Synchronous EL2t */
+ invalid_vector el2_irq_invalid_sp0 /* IRQ EL2t */
+ invalid_vector el2_fiq_invalid_sp0 /* FIQ EL2t */
+ invalid_vector el2_error_invalid_sp0 /* Error EL2t */
+
+ invalid_vector el2_sync_invalid_spx /* Synchronous EL2h */
+ invalid_vector el2_irq_invalid_spx /* IRQ EL2h */
+ invalid_vector el2_fiq_invalid_spx /* FIQ EL2h */
+ invalid_vector el2_error_invalid_spx /* Error EL2h */
+
+ el1_sync_64 /* Synchronous 64-bit EL1 */
+ invalid_vector el1_irq_invalid_64 /* IRQ 64-bit EL1 */
+ invalid_vector el1_fiq_invalid_64 /* FIQ 64-bit EL1 */
+ invalid_vector el1_error_invalid_64 /* Error 64-bit EL1 */
+
+ invalid_vector el1_sync_invalid_32 /* Synchronous 32-bit EL1 */
+ invalid_vector el1_irq_invalid_32 /* IRQ 32-bit EL1 */
+ invalid_vector el1_fiq_invalid_32 /* FIQ 32-bit EL1 */
+ invalid_vector el1_error_invalid_32 /* Error 32-bit EL1 */
+SYM_CODE_END(arm64_kexec_el2_vectors)
.popsection
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for kexec
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-25 19:19 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Configure a page table located in kexec-safe memory that has
the following mappings:
1. identity mapping for text of relocation function with executable
permission.
2. va mappings for all source ranges
3. va mappings for all destination ranges.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/include/asm/kexec.h | 12 ++++
arch/arm64/kernel/asm-offsets.c | 6 ++
arch/arm64/kernel/machine_kexec.c | 91 ++++++++++++++++++++++++++++++-
3 files changed, 108 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index b96d8a6aac80..049cde429b1b 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -105,6 +105,12 @@ extern const char arm64_kexec_el2_vectors[];
* el2_vector If present means that relocation routine will go to EL1
* from EL2 to do the copy, and then back to EL2 to do the jump
* to new world.
+ * trans_ttbr0 idmap for relocation function and its argument
+ * trans_ttbr1 map for source/destination addresses.
+ * trans_t0sz t0sz for idmap page in trans_ttbr0
+ * src_addr start address for source pages.
+ * dst_addr start address for destination pages.
+ * copy_len Number of bytes that need to be copied
*/
struct kern_reloc_arg {
phys_addr_t head;
@@ -114,6 +120,12 @@ struct kern_reloc_arg {
phys_addr_t kern_arg2;
phys_addr_t kern_arg3;
phys_addr_t el2_vector;
+ phys_addr_t trans_ttbr0;
+ phys_addr_t trans_ttbr1;
+ unsigned long trans_t0sz;
+ unsigned long src_addr;
+ unsigned long dst_addr;
+ unsigned long copy_len;
};
#define ARCH_HAS_KIMAGE_ARCH
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 8a9475be1b62..06278611451d 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -160,6 +160,12 @@ int main(void)
DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector));
+ DEFINE(KEXEC_KRELOC_TRANS_TTBR0, offsetof(struct kern_reloc_arg, trans_ttbr0));
+ DEFINE(KEXEC_KRELOC_TRANS_TTBR1, offsetof(struct kern_reloc_arg, trans_ttbr1));
+ DEFINE(KEXEC_KRELOC_TRANS_T0SZ, offsetof(struct kern_reloc_arg, trans_t0sz));
+ DEFINE(KEXEC_KRELOC_SRC_ADDR, offsetof(struct kern_reloc_arg, src_addr));
+ DEFINE(KEXEC_KRELOC_DST_ADDR, offsetof(struct kern_reloc_arg, dst_addr));
+ DEFINE(KEXEC_KRELOC_COPY_LEN, offsetof(struct kern_reloc_arg, copy_len));
#endif
return 0;
}
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 41d1e3ca13f8..dc1b7e5a54fb 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -21,6 +21,7 @@
#include <asm/mmu_context.h>
#include <asm/page.h>
#include <asm/sections.h>
+#include <asm/trans_pgd.h>
#include "cpu-reset.h"
@@ -71,11 +72,91 @@ static void *kexec_page_alloc(void *arg)
return page_address(page);
}
+/*
+ * Map source segments starting from src_va, and map destination
+ * segments starting from dst_va, and return size of copy in
+ * *copy_len argument.
+ * Relocation function essentially needs to do:
+ * memcpy(dst_va, src_va, copy_len);
+ */
+static int map_segments(struct kimage *kimage, pgd_t *pgdp,
+ struct trans_pgd_info *info,
+ unsigned long src_va,
+ unsigned long dst_va,
+ unsigned long *copy_len)
+{
+ unsigned long *ptr = 0;
+ unsigned long dest = 0;
+ unsigned long len = 0;
+ unsigned long entry, addr;
+ int rc;
+
+ for (entry = kimage->head; !(entry & IND_DONE); entry = *ptr++) {
+ addr = entry & PAGE_MASK;
+
+ switch (entry & IND_FLAGS) {
+ case IND_DESTINATION:
+ dest = addr;
+ break;
+ case IND_INDIRECTION:
+ ptr = __va(addr);
+ if (rc)
+ return rc;
+ break;
+ case IND_SOURCE:
+ rc = trans_pgd_map_page(info, pgdp, __va(addr),
+ src_va, PAGE_KERNEL);
+ if (rc)
+ return rc;
+ rc = trans_pgd_map_page(info, pgdp, __va(dest),
+ dst_va, PAGE_KERNEL);
+ if (rc)
+ return rc;
+ dest += PAGE_SIZE;
+ src_va += PAGE_SIZE;
+ dst_va += PAGE_SIZE;
+ len += PAGE_SIZE;
+ }
+ }
+ *copy_len = len;
+
+ return 0;
+}
+
+static int mmu_relocate_setup(struct kimage *kimage, void *reloc_code,
+ struct kern_reloc_arg *kern_reloc_arg)
+{
+ struct trans_pgd_info info = {
+ .trans_alloc_page = kexec_page_alloc,
+ .trans_alloc_arg = kimage,
+ };
+ pgd_t *trans_pgd = kexec_page_alloc(kimage);
+ int rc;
+
+ if (!trans_pgd)
+ return -ENOMEM;
+
+ /* idmap relocation function */
+ rc = trans_pgd_idmap_page(&info, &kern_reloc_arg->trans_ttbr0,
+ &kern_reloc_arg->trans_t0sz, reloc_code);
+ if (rc)
+ return rc;
+
+ kern_reloc_arg->src_addr = _PAGE_OFFSET(VA_BITS_MIN);
+ kern_reloc_arg->dst_addr = _PAGE_OFFSET(VA_BITS_MIN - 1);
+ kern_reloc_arg->trans_ttbr1 = phys_to_ttbr(__pa(trans_pgd));
+
+ rc = map_segments(kimage, trans_pgd, &info, kern_reloc_arg->src_addr,
+ kern_reloc_arg->dst_addr, &kern_reloc_arg->copy_len);
+ return rc;
+}
+
int machine_kexec_post_load(struct kimage *kimage)
{
void *reloc_code = page_to_virt(kimage->control_code_page);
struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage);
long func_offset, vector_offset, reloc_size;
+ int rc = 0;
if (!kern_reloc_arg)
return -ENOMEM;
@@ -95,6 +176,14 @@ int machine_kexec_post_load(struct kimage *kimage)
if (is_hyp_mode_available() && !is_kernel_in_hyp_mode())
kern_reloc_arg->el2_vector = __pa(reloc_code) + vector_offset;
+ /*
+ * If relocation is not needed, we do not need to enable MMU in
+ * relocation routine, therefore do not create page tables for
+ * scenarios such as crash kernel
+ */
+ if (!(kimage->head & IND_DONE))
+ rc = mmu_relocate_setup(kimage, reloc_code, kern_reloc_arg);
+
kexec_image_info(kimage);
/* Flush the reloc_code in preparation for its execution. */
@@ -103,7 +192,7 @@ int machine_kexec_post_load(struct kimage *kimage)
reloc_size);
__flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg));
- return 0;
+ return rc;
}
/**
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for kexec
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Configure a page table located in kexec-safe memory that has
the following mappings:
1. identity mapping for text of relocation function with executable
permission.
2. va mappings for all source ranges
3. va mappings for all destination ranges.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/include/asm/kexec.h | 12 ++++
arch/arm64/kernel/asm-offsets.c | 6 ++
arch/arm64/kernel/machine_kexec.c | 91 ++++++++++++++++++++++++++++++-
3 files changed, 108 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index b96d8a6aac80..049cde429b1b 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -105,6 +105,12 @@ extern const char arm64_kexec_el2_vectors[];
* el2_vector If present means that relocation routine will go to EL1
* from EL2 to do the copy, and then back to EL2 to do the jump
* to new world.
+ * trans_ttbr0 idmap for relocation function and its argument
+ * trans_ttbr1 map for source/destination addresses.
+ * trans_t0sz t0sz for idmap page in trans_ttbr0
+ * src_addr start address for source pages.
+ * dst_addr start address for destination pages.
+ * copy_len Number of bytes that need to be copied
*/
struct kern_reloc_arg {
phys_addr_t head;
@@ -114,6 +120,12 @@ struct kern_reloc_arg {
phys_addr_t kern_arg2;
phys_addr_t kern_arg3;
phys_addr_t el2_vector;
+ phys_addr_t trans_ttbr0;
+ phys_addr_t trans_ttbr1;
+ unsigned long trans_t0sz;
+ unsigned long src_addr;
+ unsigned long dst_addr;
+ unsigned long copy_len;
};
#define ARCH_HAS_KIMAGE_ARCH
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 8a9475be1b62..06278611451d 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -160,6 +160,12 @@ int main(void)
DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector));
+ DEFINE(KEXEC_KRELOC_TRANS_TTBR0, offsetof(struct kern_reloc_arg, trans_ttbr0));
+ DEFINE(KEXEC_KRELOC_TRANS_TTBR1, offsetof(struct kern_reloc_arg, trans_ttbr1));
+ DEFINE(KEXEC_KRELOC_TRANS_T0SZ, offsetof(struct kern_reloc_arg, trans_t0sz));
+ DEFINE(KEXEC_KRELOC_SRC_ADDR, offsetof(struct kern_reloc_arg, src_addr));
+ DEFINE(KEXEC_KRELOC_DST_ADDR, offsetof(struct kern_reloc_arg, dst_addr));
+ DEFINE(KEXEC_KRELOC_COPY_LEN, offsetof(struct kern_reloc_arg, copy_len));
#endif
return 0;
}
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 41d1e3ca13f8..dc1b7e5a54fb 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -21,6 +21,7 @@
#include <asm/mmu_context.h>
#include <asm/page.h>
#include <asm/sections.h>
+#include <asm/trans_pgd.h>
#include "cpu-reset.h"
@@ -71,11 +72,91 @@ static void *kexec_page_alloc(void *arg)
return page_address(page);
}
+/*
+ * Map source segments starting from src_va, and map destination
+ * segments starting from dst_va, and return size of copy in
+ * *copy_len argument.
+ * Relocation function essentially needs to do:
+ * memcpy(dst_va, src_va, copy_len);
+ */
+static int map_segments(struct kimage *kimage, pgd_t *pgdp,
+ struct trans_pgd_info *info,
+ unsigned long src_va,
+ unsigned long dst_va,
+ unsigned long *copy_len)
+{
+ unsigned long *ptr = 0;
+ unsigned long dest = 0;
+ unsigned long len = 0;
+ unsigned long entry, addr;
+ int rc;
+
+ for (entry = kimage->head; !(entry & IND_DONE); entry = *ptr++) {
+ addr = entry & PAGE_MASK;
+
+ switch (entry & IND_FLAGS) {
+ case IND_DESTINATION:
+ dest = addr;
+ break;
+ case IND_INDIRECTION:
+ ptr = __va(addr);
+ if (rc)
+ return rc;
+ break;
+ case IND_SOURCE:
+ rc = trans_pgd_map_page(info, pgdp, __va(addr),
+ src_va, PAGE_KERNEL);
+ if (rc)
+ return rc;
+ rc = trans_pgd_map_page(info, pgdp, __va(dest),
+ dst_va, PAGE_KERNEL);
+ if (rc)
+ return rc;
+ dest += PAGE_SIZE;
+ src_va += PAGE_SIZE;
+ dst_va += PAGE_SIZE;
+ len += PAGE_SIZE;
+ }
+ }
+ *copy_len = len;
+
+ return 0;
+}
+
+static int mmu_relocate_setup(struct kimage *kimage, void *reloc_code,
+ struct kern_reloc_arg *kern_reloc_arg)
+{
+ struct trans_pgd_info info = {
+ .trans_alloc_page = kexec_page_alloc,
+ .trans_alloc_arg = kimage,
+ };
+ pgd_t *trans_pgd = kexec_page_alloc(kimage);
+ int rc;
+
+ if (!trans_pgd)
+ return -ENOMEM;
+
+ /* idmap relocation function */
+ rc = trans_pgd_idmap_page(&info, &kern_reloc_arg->trans_ttbr0,
+ &kern_reloc_arg->trans_t0sz, reloc_code);
+ if (rc)
+ return rc;
+
+ kern_reloc_arg->src_addr = _PAGE_OFFSET(VA_BITS_MIN);
+ kern_reloc_arg->dst_addr = _PAGE_OFFSET(VA_BITS_MIN - 1);
+ kern_reloc_arg->trans_ttbr1 = phys_to_ttbr(__pa(trans_pgd));
+
+ rc = map_segments(kimage, trans_pgd, &info, kern_reloc_arg->src_addr,
+ kern_reloc_arg->dst_addr, &kern_reloc_arg->copy_len);
+ return rc;
+}
+
int machine_kexec_post_load(struct kimage *kimage)
{
void *reloc_code = page_to_virt(kimage->control_code_page);
struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage);
long func_offset, vector_offset, reloc_size;
+ int rc = 0;
if (!kern_reloc_arg)
return -ENOMEM;
@@ -95,6 +176,14 @@ int machine_kexec_post_load(struct kimage *kimage)
if (is_hyp_mode_available() && !is_kernel_in_hyp_mode())
kern_reloc_arg->el2_vector = __pa(reloc_code) + vector_offset;
+ /*
+ * If relocation is not needed, we do not need to enable MMU in
+ * relocation routine, therefore do not create page tables for
+ * scenarios such as crash kernel
+ */
+ if (!(kimage->head & IND_DONE))
+ rc = mmu_relocate_setup(kimage, reloc_code, kern_reloc_arg);
+
kexec_image_info(kimage);
/* Flush the reloc_code in preparation for its execution. */
@@ -103,7 +192,7 @@ int machine_kexec_post_load(struct kimage *kimage)
reloc_size);
__flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg));
- return 0;
+ return rc;
}
/**
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for kexec
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Configure a page table located in kexec-safe memory that has
the following mappings:
1. identity mapping for text of relocation function with executable
permission.
2. va mappings for all source ranges
3. va mappings for all destination ranges.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/include/asm/kexec.h | 12 ++++
arch/arm64/kernel/asm-offsets.c | 6 ++
arch/arm64/kernel/machine_kexec.c | 91 ++++++++++++++++++++++++++++++-
3 files changed, 108 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index b96d8a6aac80..049cde429b1b 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -105,6 +105,12 @@ extern const char arm64_kexec_el2_vectors[];
* el2_vector If present means that relocation routine will go to EL1
* from EL2 to do the copy, and then back to EL2 to do the jump
* to new world.
+ * trans_ttbr0 idmap for relocation function and its argument
+ * trans_ttbr1 map for source/destination addresses.
+ * trans_t0sz t0sz for idmap page in trans_ttbr0
+ * src_addr start address for source pages.
+ * dst_addr start address for destination pages.
+ * copy_len Number of bytes that need to be copied
*/
struct kern_reloc_arg {
phys_addr_t head;
@@ -114,6 +120,12 @@ struct kern_reloc_arg {
phys_addr_t kern_arg2;
phys_addr_t kern_arg3;
phys_addr_t el2_vector;
+ phys_addr_t trans_ttbr0;
+ phys_addr_t trans_ttbr1;
+ unsigned long trans_t0sz;
+ unsigned long src_addr;
+ unsigned long dst_addr;
+ unsigned long copy_len;
};
#define ARCH_HAS_KIMAGE_ARCH
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 8a9475be1b62..06278611451d 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -160,6 +160,12 @@ int main(void)
DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector));
+ DEFINE(KEXEC_KRELOC_TRANS_TTBR0, offsetof(struct kern_reloc_arg, trans_ttbr0));
+ DEFINE(KEXEC_KRELOC_TRANS_TTBR1, offsetof(struct kern_reloc_arg, trans_ttbr1));
+ DEFINE(KEXEC_KRELOC_TRANS_T0SZ, offsetof(struct kern_reloc_arg, trans_t0sz));
+ DEFINE(KEXEC_KRELOC_SRC_ADDR, offsetof(struct kern_reloc_arg, src_addr));
+ DEFINE(KEXEC_KRELOC_DST_ADDR, offsetof(struct kern_reloc_arg, dst_addr));
+ DEFINE(KEXEC_KRELOC_COPY_LEN, offsetof(struct kern_reloc_arg, copy_len));
#endif
return 0;
}
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 41d1e3ca13f8..dc1b7e5a54fb 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -21,6 +21,7 @@
#include <asm/mmu_context.h>
#include <asm/page.h>
#include <asm/sections.h>
+#include <asm/trans_pgd.h>
#include "cpu-reset.h"
@@ -71,11 +72,91 @@ static void *kexec_page_alloc(void *arg)
return page_address(page);
}
+/*
+ * Map source segments starting from src_va, and map destination
+ * segments starting from dst_va, and return size of copy in
+ * *copy_len argument.
+ * Relocation function essentially needs to do:
+ * memcpy(dst_va, src_va, copy_len);
+ */
+static int map_segments(struct kimage *kimage, pgd_t *pgdp,
+ struct trans_pgd_info *info,
+ unsigned long src_va,
+ unsigned long dst_va,
+ unsigned long *copy_len)
+{
+ unsigned long *ptr = 0;
+ unsigned long dest = 0;
+ unsigned long len = 0;
+ unsigned long entry, addr;
+ int rc;
+
+ for (entry = kimage->head; !(entry & IND_DONE); entry = *ptr++) {
+ addr = entry & PAGE_MASK;
+
+ switch (entry & IND_FLAGS) {
+ case IND_DESTINATION:
+ dest = addr;
+ break;
+ case IND_INDIRECTION:
+ ptr = __va(addr);
+ if (rc)
+ return rc;
+ break;
+ case IND_SOURCE:
+ rc = trans_pgd_map_page(info, pgdp, __va(addr),
+ src_va, PAGE_KERNEL);
+ if (rc)
+ return rc;
+ rc = trans_pgd_map_page(info, pgdp, __va(dest),
+ dst_va, PAGE_KERNEL);
+ if (rc)
+ return rc;
+ dest += PAGE_SIZE;
+ src_va += PAGE_SIZE;
+ dst_va += PAGE_SIZE;
+ len += PAGE_SIZE;
+ }
+ }
+ *copy_len = len;
+
+ return 0;
+}
+
+static int mmu_relocate_setup(struct kimage *kimage, void *reloc_code,
+ struct kern_reloc_arg *kern_reloc_arg)
+{
+ struct trans_pgd_info info = {
+ .trans_alloc_page = kexec_page_alloc,
+ .trans_alloc_arg = kimage,
+ };
+ pgd_t *trans_pgd = kexec_page_alloc(kimage);
+ int rc;
+
+ if (!trans_pgd)
+ return -ENOMEM;
+
+ /* idmap relocation function */
+ rc = trans_pgd_idmap_page(&info, &kern_reloc_arg->trans_ttbr0,
+ &kern_reloc_arg->trans_t0sz, reloc_code);
+ if (rc)
+ return rc;
+
+ kern_reloc_arg->src_addr = _PAGE_OFFSET(VA_BITS_MIN);
+ kern_reloc_arg->dst_addr = _PAGE_OFFSET(VA_BITS_MIN - 1);
+ kern_reloc_arg->trans_ttbr1 = phys_to_ttbr(__pa(trans_pgd));
+
+ rc = map_segments(kimage, trans_pgd, &info, kern_reloc_arg->src_addr,
+ kern_reloc_arg->dst_addr, &kern_reloc_arg->copy_len);
+ return rc;
+}
+
int machine_kexec_post_load(struct kimage *kimage)
{
void *reloc_code = page_to_virt(kimage->control_code_page);
struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage);
long func_offset, vector_offset, reloc_size;
+ int rc = 0;
if (!kern_reloc_arg)
return -ENOMEM;
@@ -95,6 +176,14 @@ int machine_kexec_post_load(struct kimage *kimage)
if (is_hyp_mode_available() && !is_kernel_in_hyp_mode())
kern_reloc_arg->el2_vector = __pa(reloc_code) + vector_offset;
+ /*
+ * If relocation is not needed, we do not need to enable MMU in
+ * relocation routine, therefore do not create page tables for
+ * scenarios such as crash kernel
+ */
+ if (!(kimage->head & IND_DONE))
+ rc = mmu_relocate_setup(kimage, reloc_code, kern_reloc_arg);
+
kexec_image_info(kimage);
/* Flush the reloc_code in preparation for its execution. */
@@ -103,7 +192,7 @@ int machine_kexec_post_load(struct kimage *kimage)
reloc_size);
__flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg));
- return 0;
+ return rc;
}
/**
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* Re: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for kexec
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-26 0:21 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-26 0:21 UTC (permalink / raw)
To: jmorris, sashal, ebiederm, kexec, linux-kernel, corbet,
catalin.marinas, will, linux-arm-kernel, maz, james.morse,
vladimir.murzin, matthias.bgg, linux-mm, mark.rutland,
steve.capper, rfontana, tglx, selindag, tyhicks
I forgot to make changes to arch/arm64/Kconfig. The correct patch is
below.
---
From a2bc374320d7c7efd3c40644ad3d6d59a024b301 Mon Sep 17 00:00:00 2001
From: Pavel Tatashin <pasha.tatashin@soleen.com>
Date: Mon, 29 Jul 2019 21:24:25 -0400
Subject: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for
kexec
Configure a page table located in kexec-safe memory that has
the following mappings:
1. identity mapping for text of relocation function with executable
permission.
2. va mappings for all source ranges
3. va mappings for all destination ranges.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/Kconfig | 2 +-
arch/arm64/include/asm/kexec.h | 12 ++++
arch/arm64/kernel/asm-offsets.c | 6 ++
arch/arm64/kernel/machine_kexec.c | 91 ++++++++++++++++++++++++++++++-
4 files changed, 109 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index fc0ed9d6e011..440abd0c0ee1 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1134,7 +1134,7 @@ config CRASH_DUMP
config TRANS_TABLE
def_bool y
- depends on HIBERNATION
+ depends on HIBERNATION || KEXEC_CORE
config XEN_DOM0
def_bool y
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index b96d8a6aac80..049cde429b1b 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -105,6 +105,12 @@ extern const char arm64_kexec_el2_vectors[];
* el2_vector If present means that relocation routine will go to EL1
* from EL2 to do the copy, and then back to EL2 to do the jump
* to new world.
+ * trans_ttbr0 idmap for relocation function and its argument
+ * trans_ttbr1 map for source/destination addresses.
+ * trans_t0sz t0sz for idmap page in trans_ttbr0
+ * src_addr start address for source pages.
+ * dst_addr start address for destination pages.
+ * copy_len Number of bytes that need to be copied
*/
struct kern_reloc_arg {
phys_addr_t head;
@@ -114,6 +120,12 @@ struct kern_reloc_arg {
phys_addr_t kern_arg2;
phys_addr_t kern_arg3;
phys_addr_t el2_vector;
+ phys_addr_t trans_ttbr0;
+ phys_addr_t trans_ttbr1;
+ unsigned long trans_t0sz;
+ unsigned long src_addr;
+ unsigned long dst_addr;
+ unsigned long copy_len;
};
#define ARCH_HAS_KIMAGE_ARCH
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 8a9475be1b62..06278611451d 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -160,6 +160,12 @@ int main(void)
DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector));
+ DEFINE(KEXEC_KRELOC_TRANS_TTBR0, offsetof(struct kern_reloc_arg, trans_ttbr0));
+ DEFINE(KEXEC_KRELOC_TRANS_TTBR1, offsetof(struct kern_reloc_arg, trans_ttbr1));
+ DEFINE(KEXEC_KRELOC_TRANS_T0SZ, offsetof(struct kern_reloc_arg, trans_t0sz));
+ DEFINE(KEXEC_KRELOC_SRC_ADDR, offsetof(struct kern_reloc_arg, src_addr));
+ DEFINE(KEXEC_KRELOC_DST_ADDR, offsetof(struct kern_reloc_arg, dst_addr));
+ DEFINE(KEXEC_KRELOC_COPY_LEN, offsetof(struct kern_reloc_arg, copy_len));
#endif
return 0;
}
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 41d1e3ca13f8..dc1b7e5a54fb 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -21,6 +21,7 @@
#include <asm/mmu_context.h>
#include <asm/page.h>
#include <asm/sections.h>
+#include <asm/trans_pgd.h>
#include "cpu-reset.h"
@@ -71,11 +72,91 @@ static void *kexec_page_alloc(void *arg)
return page_address(page);
}
+/*
+ * Map source segments starting from src_va, and map destination
+ * segments starting from dst_va, and return size of copy in
+ * *copy_len argument.
+ * Relocation function essentially needs to do:
+ * memcpy(dst_va, src_va, copy_len);
+ */
+static int map_segments(struct kimage *kimage, pgd_t *pgdp,
+ struct trans_pgd_info *info,
+ unsigned long src_va,
+ unsigned long dst_va,
+ unsigned long *copy_len)
+{
+ unsigned long *ptr = 0;
+ unsigned long dest = 0;
+ unsigned long len = 0;
+ unsigned long entry, addr;
+ int rc;
+
+ for (entry = kimage->head; !(entry & IND_DONE); entry = *ptr++) {
+ addr = entry & PAGE_MASK;
+
+ switch (entry & IND_FLAGS) {
+ case IND_DESTINATION:
+ dest = addr;
+ break;
+ case IND_INDIRECTION:
+ ptr = __va(addr);
+ if (rc)
+ return rc;
+ break;
+ case IND_SOURCE:
+ rc = trans_pgd_map_page(info, pgdp, __va(addr),
+ src_va, PAGE_KERNEL);
+ if (rc)
+ return rc;
+ rc = trans_pgd_map_page(info, pgdp, __va(dest),
+ dst_va, PAGE_KERNEL);
+ if (rc)
+ return rc;
+ dest += PAGE_SIZE;
+ src_va += PAGE_SIZE;
+ dst_va += PAGE_SIZE;
+ len += PAGE_SIZE;
+ }
+ }
+ *copy_len = len;
+
+ return 0;
+}
+
+static int mmu_relocate_setup(struct kimage *kimage, void *reloc_code,
+ struct kern_reloc_arg *kern_reloc_arg)
+{
+ struct trans_pgd_info info = {
+ .trans_alloc_page = kexec_page_alloc,
+ .trans_alloc_arg = kimage,
+ };
+ pgd_t *trans_pgd = kexec_page_alloc(kimage);
+ int rc;
+
+ if (!trans_pgd)
+ return -ENOMEM;
+
+ /* idmap relocation function */
+ rc = trans_pgd_idmap_page(&info, &kern_reloc_arg->trans_ttbr0,
+ &kern_reloc_arg->trans_t0sz, reloc_code);
+ if (rc)
+ return rc;
+
+ kern_reloc_arg->src_addr = _PAGE_OFFSET(VA_BITS_MIN);
+ kern_reloc_arg->dst_addr = _PAGE_OFFSET(VA_BITS_MIN - 1);
+ kern_reloc_arg->trans_ttbr1 = phys_to_ttbr(__pa(trans_pgd));
+
+ rc = map_segments(kimage, trans_pgd, &info, kern_reloc_arg->src_addr,
+ kern_reloc_arg->dst_addr, &kern_reloc_arg->copy_len);
+ return rc;
+}
+
int machine_kexec_post_load(struct kimage *kimage)
{
void *reloc_code = page_to_virt(kimage->control_code_page);
struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage);
long func_offset, vector_offset, reloc_size;
+ int rc = 0;
if (!kern_reloc_arg)
return -ENOMEM;
@@ -95,6 +176,14 @@ int machine_kexec_post_load(struct kimage *kimage)
if (is_hyp_mode_available() && !is_kernel_in_hyp_mode())
kern_reloc_arg->el2_vector = __pa(reloc_code) + vector_offset;
+ /*
+ * If relocation is not needed, we do not need to enable MMU in
+ * relocation routine, therefore do not create page tables for
+ * scenarios such as crash kernel
+ */
+ if (!(kimage->head & IND_DONE))
+ rc = mmu_relocate_setup(kimage, reloc_code, kern_reloc_arg);
+
kexec_image_info(kimage);
/* Flush the reloc_code in preparation for its execution. */
@@ -103,7 +192,7 @@ int machine_kexec_post_load(struct kimage *kimage)
reloc_size);
__flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg));
- return 0;
+ return rc;
}
/**
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* Re: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for kexec
@ 2021-01-26 0:21 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-26 0:21 UTC (permalink / raw)
To: jmorris, sashal, ebiederm, kexec, linux-kernel, corbet,
catalin.marinas, will, linux-arm-kernel, maz, james.morse,
vladimir.murzin, matthias.bgg, linux-mm, mark.rutland,
steve.capper, rfontana, tglx, selindag, tyhicks
I forgot to make changes to arch/arm64/Kconfig. The correct patch is
below.
---
From a2bc374320d7c7efd3c40644ad3d6d59a024b301 Mon Sep 17 00:00:00 2001
From: Pavel Tatashin <pasha.tatashin@soleen.com>
Date: Mon, 29 Jul 2019 21:24:25 -0400
Subject: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for
kexec
Configure a page table located in kexec-safe memory that has
the following mappings:
1. identity mapping for text of relocation function with executable
permission.
2. va mappings for all source ranges
3. va mappings for all destination ranges.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/Kconfig | 2 +-
arch/arm64/include/asm/kexec.h | 12 ++++
arch/arm64/kernel/asm-offsets.c | 6 ++
arch/arm64/kernel/machine_kexec.c | 91 ++++++++++++++++++++++++++++++-
4 files changed, 109 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index fc0ed9d6e011..440abd0c0ee1 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1134,7 +1134,7 @@ config CRASH_DUMP
config TRANS_TABLE
def_bool y
- depends on HIBERNATION
+ depends on HIBERNATION || KEXEC_CORE
config XEN_DOM0
def_bool y
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index b96d8a6aac80..049cde429b1b 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -105,6 +105,12 @@ extern const char arm64_kexec_el2_vectors[];
* el2_vector If present means that relocation routine will go to EL1
* from EL2 to do the copy, and then back to EL2 to do the jump
* to new world.
+ * trans_ttbr0 idmap for relocation function and its argument
+ * trans_ttbr1 map for source/destination addresses.
+ * trans_t0sz t0sz for idmap page in trans_ttbr0
+ * src_addr start address for source pages.
+ * dst_addr start address for destination pages.
+ * copy_len Number of bytes that need to be copied
*/
struct kern_reloc_arg {
phys_addr_t head;
@@ -114,6 +120,12 @@ struct kern_reloc_arg {
phys_addr_t kern_arg2;
phys_addr_t kern_arg3;
phys_addr_t el2_vector;
+ phys_addr_t trans_ttbr0;
+ phys_addr_t trans_ttbr1;
+ unsigned long trans_t0sz;
+ unsigned long src_addr;
+ unsigned long dst_addr;
+ unsigned long copy_len;
};
#define ARCH_HAS_KIMAGE_ARCH
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 8a9475be1b62..06278611451d 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -160,6 +160,12 @@ int main(void)
DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector));
+ DEFINE(KEXEC_KRELOC_TRANS_TTBR0, offsetof(struct kern_reloc_arg, trans_ttbr0));
+ DEFINE(KEXEC_KRELOC_TRANS_TTBR1, offsetof(struct kern_reloc_arg, trans_ttbr1));
+ DEFINE(KEXEC_KRELOC_TRANS_T0SZ, offsetof(struct kern_reloc_arg, trans_t0sz));
+ DEFINE(KEXEC_KRELOC_SRC_ADDR, offsetof(struct kern_reloc_arg, src_addr));
+ DEFINE(KEXEC_KRELOC_DST_ADDR, offsetof(struct kern_reloc_arg, dst_addr));
+ DEFINE(KEXEC_KRELOC_COPY_LEN, offsetof(struct kern_reloc_arg, copy_len));
#endif
return 0;
}
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 41d1e3ca13f8..dc1b7e5a54fb 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -21,6 +21,7 @@
#include <asm/mmu_context.h>
#include <asm/page.h>
#include <asm/sections.h>
+#include <asm/trans_pgd.h>
#include "cpu-reset.h"
@@ -71,11 +72,91 @@ static void *kexec_page_alloc(void *arg)
return page_address(page);
}
+/*
+ * Map source segments starting from src_va, and map destination
+ * segments starting from dst_va, and return size of copy in
+ * *copy_len argument.
+ * Relocation function essentially needs to do:
+ * memcpy(dst_va, src_va, copy_len);
+ */
+static int map_segments(struct kimage *kimage, pgd_t *pgdp,
+ struct trans_pgd_info *info,
+ unsigned long src_va,
+ unsigned long dst_va,
+ unsigned long *copy_len)
+{
+ unsigned long *ptr = 0;
+ unsigned long dest = 0;
+ unsigned long len = 0;
+ unsigned long entry, addr;
+ int rc;
+
+ for (entry = kimage->head; !(entry & IND_DONE); entry = *ptr++) {
+ addr = entry & PAGE_MASK;
+
+ switch (entry & IND_FLAGS) {
+ case IND_DESTINATION:
+ dest = addr;
+ break;
+ case IND_INDIRECTION:
+ ptr = __va(addr);
+ if (rc)
+ return rc;
+ break;
+ case IND_SOURCE:
+ rc = trans_pgd_map_page(info, pgdp, __va(addr),
+ src_va, PAGE_KERNEL);
+ if (rc)
+ return rc;
+ rc = trans_pgd_map_page(info, pgdp, __va(dest),
+ dst_va, PAGE_KERNEL);
+ if (rc)
+ return rc;
+ dest += PAGE_SIZE;
+ src_va += PAGE_SIZE;
+ dst_va += PAGE_SIZE;
+ len += PAGE_SIZE;
+ }
+ }
+ *copy_len = len;
+
+ return 0;
+}
+
+static int mmu_relocate_setup(struct kimage *kimage, void *reloc_code,
+ struct kern_reloc_arg *kern_reloc_arg)
+{
+ struct trans_pgd_info info = {
+ .trans_alloc_page = kexec_page_alloc,
+ .trans_alloc_arg = kimage,
+ };
+ pgd_t *trans_pgd = kexec_page_alloc(kimage);
+ int rc;
+
+ if (!trans_pgd)
+ return -ENOMEM;
+
+ /* idmap relocation function */
+ rc = trans_pgd_idmap_page(&info, &kern_reloc_arg->trans_ttbr0,
+ &kern_reloc_arg->trans_t0sz, reloc_code);
+ if (rc)
+ return rc;
+
+ kern_reloc_arg->src_addr = _PAGE_OFFSET(VA_BITS_MIN);
+ kern_reloc_arg->dst_addr = _PAGE_OFFSET(VA_BITS_MIN - 1);
+ kern_reloc_arg->trans_ttbr1 = phys_to_ttbr(__pa(trans_pgd));
+
+ rc = map_segments(kimage, trans_pgd, &info, kern_reloc_arg->src_addr,
+ kern_reloc_arg->dst_addr, &kern_reloc_arg->copy_len);
+ return rc;
+}
+
int machine_kexec_post_load(struct kimage *kimage)
{
void *reloc_code = page_to_virt(kimage->control_code_page);
struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage);
long func_offset, vector_offset, reloc_size;
+ int rc = 0;
if (!kern_reloc_arg)
return -ENOMEM;
@@ -95,6 +176,14 @@ int machine_kexec_post_load(struct kimage *kimage)
if (is_hyp_mode_available() && !is_kernel_in_hyp_mode())
kern_reloc_arg->el2_vector = __pa(reloc_code) + vector_offset;
+ /*
+ * If relocation is not needed, we do not need to enable MMU in
+ * relocation routine, therefore do not create page tables for
+ * scenarios such as crash kernel
+ */
+ if (!(kimage->head & IND_DONE))
+ rc = mmu_relocate_setup(kimage, reloc_code, kern_reloc_arg);
+
kexec_image_info(kimage);
/* Flush the reloc_code in preparation for its execution. */
@@ -103,7 +192,7 @@ int machine_kexec_post_load(struct kimage *kimage)
reloc_size);
__flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg));
- return 0;
+ return rc;
}
/**
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* Re: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for kexec
@ 2021-01-26 0:21 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-26 0:21 UTC (permalink / raw)
To: jmorris, sashal, ebiederm, kexec, linux-kernel, corbet,
catalin.marinas, will, linux-arm-kernel, maz, james.morse,
vladimir.murzin, matthias.bgg, linux-mm, mark.rutland,
steve.capper, rfontana, tglx, selindag, tyhicks
I forgot to make changes to arch/arm64/Kconfig. The correct patch is
below.
---
From a2bc374320d7c7efd3c40644ad3d6d59a024b301 Mon Sep 17 00:00:00 2001
From: Pavel Tatashin <pasha.tatashin@soleen.com>
Date: Mon, 29 Jul 2019 21:24:25 -0400
Subject: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for
kexec
Configure a page table located in kexec-safe memory that has
the following mappings:
1. identity mapping for text of relocation function with executable
permission.
2. va mappings for all source ranges
3. va mappings for all destination ranges.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/Kconfig | 2 +-
arch/arm64/include/asm/kexec.h | 12 ++++
arch/arm64/kernel/asm-offsets.c | 6 ++
arch/arm64/kernel/machine_kexec.c | 91 ++++++++++++++++++++++++++++++-
4 files changed, 109 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index fc0ed9d6e011..440abd0c0ee1 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1134,7 +1134,7 @@ config CRASH_DUMP
config TRANS_TABLE
def_bool y
- depends on HIBERNATION
+ depends on HIBERNATION || KEXEC_CORE
config XEN_DOM0
def_bool y
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index b96d8a6aac80..049cde429b1b 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -105,6 +105,12 @@ extern const char arm64_kexec_el2_vectors[];
* el2_vector If present means that relocation routine will go to EL1
* from EL2 to do the copy, and then back to EL2 to do the jump
* to new world.
+ * trans_ttbr0 idmap for relocation function and its argument
+ * trans_ttbr1 map for source/destination addresses.
+ * trans_t0sz t0sz for idmap page in trans_ttbr0
+ * src_addr start address for source pages.
+ * dst_addr start address for destination pages.
+ * copy_len Number of bytes that need to be copied
*/
struct kern_reloc_arg {
phys_addr_t head;
@@ -114,6 +120,12 @@ struct kern_reloc_arg {
phys_addr_t kern_arg2;
phys_addr_t kern_arg3;
phys_addr_t el2_vector;
+ phys_addr_t trans_ttbr0;
+ phys_addr_t trans_ttbr1;
+ unsigned long trans_t0sz;
+ unsigned long src_addr;
+ unsigned long dst_addr;
+ unsigned long copy_len;
};
#define ARCH_HAS_KIMAGE_ARCH
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 8a9475be1b62..06278611451d 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -160,6 +160,12 @@ int main(void)
DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector));
+ DEFINE(KEXEC_KRELOC_TRANS_TTBR0, offsetof(struct kern_reloc_arg, trans_ttbr0));
+ DEFINE(KEXEC_KRELOC_TRANS_TTBR1, offsetof(struct kern_reloc_arg, trans_ttbr1));
+ DEFINE(KEXEC_KRELOC_TRANS_T0SZ, offsetof(struct kern_reloc_arg, trans_t0sz));
+ DEFINE(KEXEC_KRELOC_SRC_ADDR, offsetof(struct kern_reloc_arg, src_addr));
+ DEFINE(KEXEC_KRELOC_DST_ADDR, offsetof(struct kern_reloc_arg, dst_addr));
+ DEFINE(KEXEC_KRELOC_COPY_LEN, offsetof(struct kern_reloc_arg, copy_len));
#endif
return 0;
}
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 41d1e3ca13f8..dc1b7e5a54fb 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -21,6 +21,7 @@
#include <asm/mmu_context.h>
#include <asm/page.h>
#include <asm/sections.h>
+#include <asm/trans_pgd.h>
#include "cpu-reset.h"
@@ -71,11 +72,91 @@ static void *kexec_page_alloc(void *arg)
return page_address(page);
}
+/*
+ * Map source segments starting from src_va, and map destination
+ * segments starting from dst_va, and return size of copy in
+ * *copy_len argument.
+ * Relocation function essentially needs to do:
+ * memcpy(dst_va, src_va, copy_len);
+ */
+static int map_segments(struct kimage *kimage, pgd_t *pgdp,
+ struct trans_pgd_info *info,
+ unsigned long src_va,
+ unsigned long dst_va,
+ unsigned long *copy_len)
+{
+ unsigned long *ptr = 0;
+ unsigned long dest = 0;
+ unsigned long len = 0;
+ unsigned long entry, addr;
+ int rc;
+
+ for (entry = kimage->head; !(entry & IND_DONE); entry = *ptr++) {
+ addr = entry & PAGE_MASK;
+
+ switch (entry & IND_FLAGS) {
+ case IND_DESTINATION:
+ dest = addr;
+ break;
+ case IND_INDIRECTION:
+ ptr = __va(addr);
+ if (rc)
+ return rc;
+ break;
+ case IND_SOURCE:
+ rc = trans_pgd_map_page(info, pgdp, __va(addr),
+ src_va, PAGE_KERNEL);
+ if (rc)
+ return rc;
+ rc = trans_pgd_map_page(info, pgdp, __va(dest),
+ dst_va, PAGE_KERNEL);
+ if (rc)
+ return rc;
+ dest += PAGE_SIZE;
+ src_va += PAGE_SIZE;
+ dst_va += PAGE_SIZE;
+ len += PAGE_SIZE;
+ }
+ }
+ *copy_len = len;
+
+ return 0;
+}
+
+static int mmu_relocate_setup(struct kimage *kimage, void *reloc_code,
+ struct kern_reloc_arg *kern_reloc_arg)
+{
+ struct trans_pgd_info info = {
+ .trans_alloc_page = kexec_page_alloc,
+ .trans_alloc_arg = kimage,
+ };
+ pgd_t *trans_pgd = kexec_page_alloc(kimage);
+ int rc;
+
+ if (!trans_pgd)
+ return -ENOMEM;
+
+ /* idmap relocation function */
+ rc = trans_pgd_idmap_page(&info, &kern_reloc_arg->trans_ttbr0,
+ &kern_reloc_arg->trans_t0sz, reloc_code);
+ if (rc)
+ return rc;
+
+ kern_reloc_arg->src_addr = _PAGE_OFFSET(VA_BITS_MIN);
+ kern_reloc_arg->dst_addr = _PAGE_OFFSET(VA_BITS_MIN - 1);
+ kern_reloc_arg->trans_ttbr1 = phys_to_ttbr(__pa(trans_pgd));
+
+ rc = map_segments(kimage, trans_pgd, &info, kern_reloc_arg->src_addr,
+ kern_reloc_arg->dst_addr, &kern_reloc_arg->copy_len);
+ return rc;
+}
+
int machine_kexec_post_load(struct kimage *kimage)
{
void *reloc_code = page_to_virt(kimage->control_code_page);
struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage);
long func_offset, vector_offset, reloc_size;
+ int rc = 0;
if (!kern_reloc_arg)
return -ENOMEM;
@@ -95,6 +176,14 @@ int machine_kexec_post_load(struct kimage *kimage)
if (is_hyp_mode_available() && !is_kernel_in_hyp_mode())
kern_reloc_arg->el2_vector = __pa(reloc_code) + vector_offset;
+ /*
+ * If relocation is not needed, we do not need to enable MMU in
+ * relocation routine, therefore do not create page tables for
+ * scenarios such as crash kernel
+ */
+ if (!(kimage->head & IND_DONE))
+ rc = mmu_relocate_setup(kimage, reloc_code, kern_reloc_arg);
+
kexec_image_info(kimage);
/* Flush the reloc_code in preparation for its execution. */
@@ -103,7 +192,7 @@ int machine_kexec_post_load(struct kimage *kimage)
reloc_size);
__flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg));
- return 0;
+ return rc;
}
/**
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* Re: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for kexec
2021-01-26 0:21 ` Pavel Tatashin
(?)
@ 2021-01-27 16:42 ` Tyler Hicks
-1 siblings, 0 replies; 84+ messages in thread
From: Tyler Hicks @ 2021-01-27 16:42 UTC (permalink / raw)
To: Pavel Tatashin
Cc: jmorris, sashal, ebiederm, kexec, linux-kernel, corbet,
catalin.marinas, will, linux-arm-kernel, maz, james.morse,
vladimir.murzin, matthias.bgg, linux-mm, mark.rutland,
steve.capper, rfontana, tglx, selindag
On 2021-01-25 19:21:22, Pavel Tatashin wrote:
> I forgot to make changes to arch/arm64/Kconfig. The correct patch is
> below.
>
> ---
>
> From a2bc374320d7c7efd3c40644ad3d6d59a024b301 Mon Sep 17 00:00:00 2001
> From: Pavel Tatashin <pasha.tatashin@soleen.com>
> Date: Mon, 29 Jul 2019 21:24:25 -0400
> Subject: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for
> kexec
>
> Configure a page table located in kexec-safe memory that has
> the following mappings:
>
> 1. identity mapping for text of relocation function with executable
> permission.
> 2. va mappings for all source ranges
> 3. va mappings for all destination ranges.
>
> Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
> ---
> arch/arm64/Kconfig | 2 +-
> arch/arm64/include/asm/kexec.h | 12 ++++
> arch/arm64/kernel/asm-offsets.c | 6 ++
> arch/arm64/kernel/machine_kexec.c | 91 ++++++++++++++++++++++++++++++-
> 4 files changed, 109 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index fc0ed9d6e011..440abd0c0ee1 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -1134,7 +1134,7 @@ config CRASH_DUMP
>
> config TRANS_TABLE
> def_bool y
> - depends on HIBERNATION
> + depends on HIBERNATION || KEXEC_CORE
>
> config XEN_DOM0
> def_bool y
> diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
> index b96d8a6aac80..049cde429b1b 100644
> --- a/arch/arm64/include/asm/kexec.h
> +++ b/arch/arm64/include/asm/kexec.h
> @@ -105,6 +105,12 @@ extern const char arm64_kexec_el2_vectors[];
> * el2_vector If present means that relocation routine will go to EL1
> * from EL2 to do the copy, and then back to EL2 to do the jump
> * to new world.
> + * trans_ttbr0 idmap for relocation function and its argument
> + * trans_ttbr1 map for source/destination addresses.
> + * trans_t0sz t0sz for idmap page in trans_ttbr0
> + * src_addr start address for source pages.
> + * dst_addr start address for destination pages.
> + * copy_len Number of bytes that need to be copied
> */
> struct kern_reloc_arg {
> phys_addr_t head;
> @@ -114,6 +120,12 @@ struct kern_reloc_arg {
> phys_addr_t kern_arg2;
> phys_addr_t kern_arg3;
> phys_addr_t el2_vector;
> + phys_addr_t trans_ttbr0;
> + phys_addr_t trans_ttbr1;
> + unsigned long trans_t0sz;
> + unsigned long src_addr;
> + unsigned long dst_addr;
> + unsigned long copy_len;
> };
>
> #define ARCH_HAS_KIMAGE_ARCH
> diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
> index 8a9475be1b62..06278611451d 100644
> --- a/arch/arm64/kernel/asm-offsets.c
> +++ b/arch/arm64/kernel/asm-offsets.c
> @@ -160,6 +160,12 @@ int main(void)
> DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
> DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
> DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector));
> + DEFINE(KEXEC_KRELOC_TRANS_TTBR0, offsetof(struct kern_reloc_arg, trans_ttbr0));
> + DEFINE(KEXEC_KRELOC_TRANS_TTBR1, offsetof(struct kern_reloc_arg, trans_ttbr1));
> + DEFINE(KEXEC_KRELOC_TRANS_T0SZ, offsetof(struct kern_reloc_arg, trans_t0sz));
> + DEFINE(KEXEC_KRELOC_SRC_ADDR, offsetof(struct kern_reloc_arg, src_addr));
> + DEFINE(KEXEC_KRELOC_DST_ADDR, offsetof(struct kern_reloc_arg, dst_addr));
> + DEFINE(KEXEC_KRELOC_COPY_LEN, offsetof(struct kern_reloc_arg, copy_len));
> #endif
> return 0;
> }
> diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
> index 41d1e3ca13f8..dc1b7e5a54fb 100644
> --- a/arch/arm64/kernel/machine_kexec.c
> +++ b/arch/arm64/kernel/machine_kexec.c
> @@ -21,6 +21,7 @@
> #include <asm/mmu_context.h>
> #include <asm/page.h>
> #include <asm/sections.h>
> +#include <asm/trans_pgd.h>
>
> #include "cpu-reset.h"
>
> @@ -71,11 +72,91 @@ static void *kexec_page_alloc(void *arg)
> return page_address(page);
> }
>
> +/*
> + * Map source segments starting from src_va, and map destination
> + * segments starting from dst_va, and return size of copy in
> + * *copy_len argument.
> + * Relocation function essentially needs to do:
> + * memcpy(dst_va, src_va, copy_len);
> + */
> +static int map_segments(struct kimage *kimage, pgd_t *pgdp,
> + struct trans_pgd_info *info,
> + unsigned long src_va,
> + unsigned long dst_va,
> + unsigned long *copy_len)
> +{
> + unsigned long *ptr = 0;
> + unsigned long dest = 0;
> + unsigned long len = 0;
> + unsigned long entry, addr;
> + int rc;
> +
> + for (entry = kimage->head; !(entry & IND_DONE); entry = *ptr++) {
> + addr = entry & PAGE_MASK;
> +
> + switch (entry & IND_FLAGS) {
> + case IND_DESTINATION:
> + dest = addr;
> + break;
> + case IND_INDIRECTION:
> + ptr = __va(addr);
> + if (rc)
We're checking a potentially uninitialized value here. I think this
whole "if (rc) return rc;" conditional can go away.
Tyler
> + return rc;
> + break;
> + case IND_SOURCE:
> + rc = trans_pgd_map_page(info, pgdp, __va(addr),
> + src_va, PAGE_KERNEL);
> + if (rc)
> + return rc;
> + rc = trans_pgd_map_page(info, pgdp, __va(dest),
> + dst_va, PAGE_KERNEL);
> + if (rc)
> + return rc;
> + dest += PAGE_SIZE;
> + src_va += PAGE_SIZE;
> + dst_va += PAGE_SIZE;
> + len += PAGE_SIZE;
> + }
> + }
> + *copy_len = len;
> +
> + return 0;
> +}
> +
> +static int mmu_relocate_setup(struct kimage *kimage, void *reloc_code,
> + struct kern_reloc_arg *kern_reloc_arg)
> +{
> + struct trans_pgd_info info = {
> + .trans_alloc_page = kexec_page_alloc,
> + .trans_alloc_arg = kimage,
> + };
> + pgd_t *trans_pgd = kexec_page_alloc(kimage);
> + int rc;
> +
> + if (!trans_pgd)
> + return -ENOMEM;
> +
> + /* idmap relocation function */
> + rc = trans_pgd_idmap_page(&info, &kern_reloc_arg->trans_ttbr0,
> + &kern_reloc_arg->trans_t0sz, reloc_code);
> + if (rc)
> + return rc;
> +
> + kern_reloc_arg->src_addr = _PAGE_OFFSET(VA_BITS_MIN);
> + kern_reloc_arg->dst_addr = _PAGE_OFFSET(VA_BITS_MIN - 1);
> + kern_reloc_arg->trans_ttbr1 = phys_to_ttbr(__pa(trans_pgd));
> +
> + rc = map_segments(kimage, trans_pgd, &info, kern_reloc_arg->src_addr,
> + kern_reloc_arg->dst_addr, &kern_reloc_arg->copy_len);
> + return rc;
> +}
> +
> int machine_kexec_post_load(struct kimage *kimage)
> {
> void *reloc_code = page_to_virt(kimage->control_code_page);
> struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage);
> long func_offset, vector_offset, reloc_size;
> + int rc = 0;
>
> if (!kern_reloc_arg)
> return -ENOMEM;
> @@ -95,6 +176,14 @@ int machine_kexec_post_load(struct kimage *kimage)
> if (is_hyp_mode_available() && !is_kernel_in_hyp_mode())
> kern_reloc_arg->el2_vector = __pa(reloc_code) + vector_offset;
>
> + /*
> + * If relocation is not needed, we do not need to enable MMU in
> + * relocation routine, therefore do not create page tables for
> + * scenarios such as crash kernel
> + */
> + if (!(kimage->head & IND_DONE))
> + rc = mmu_relocate_setup(kimage, reloc_code, kern_reloc_arg);
> +
> kexec_image_info(kimage);
>
> /* Flush the reloc_code in preparation for its execution. */
> @@ -103,7 +192,7 @@ int machine_kexec_post_load(struct kimage *kimage)
> reloc_size);
> __flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg));
>
> - return 0;
> + return rc;
> }
>
> /**
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for kexec
@ 2021-01-27 16:42 ` Tyler Hicks
0 siblings, 0 replies; 84+ messages in thread
From: Tyler Hicks @ 2021-01-27 16:42 UTC (permalink / raw)
To: Pavel Tatashin
Cc: sashal, mark.rutland, vladimir.murzin, corbet, catalin.marinas,
selindag, steve.capper, kexec, linux-kernel, jmorris, linux-mm,
james.morse, ebiederm, maz, matthias.bgg, rfontana, will, tglx,
linux-arm-kernel
On 2021-01-25 19:21:22, Pavel Tatashin wrote:
> I forgot to make changes to arch/arm64/Kconfig. The correct patch is
> below.
>
> ---
>
> From a2bc374320d7c7efd3c40644ad3d6d59a024b301 Mon Sep 17 00:00:00 2001
> From: Pavel Tatashin <pasha.tatashin@soleen.com>
> Date: Mon, 29 Jul 2019 21:24:25 -0400
> Subject: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for
> kexec
>
> Configure a page table located in kexec-safe memory that has
> the following mappings:
>
> 1. identity mapping for text of relocation function with executable
> permission.
> 2. va mappings for all source ranges
> 3. va mappings for all destination ranges.
>
> Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
> ---
> arch/arm64/Kconfig | 2 +-
> arch/arm64/include/asm/kexec.h | 12 ++++
> arch/arm64/kernel/asm-offsets.c | 6 ++
> arch/arm64/kernel/machine_kexec.c | 91 ++++++++++++++++++++++++++++++-
> 4 files changed, 109 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index fc0ed9d6e011..440abd0c0ee1 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -1134,7 +1134,7 @@ config CRASH_DUMP
>
> config TRANS_TABLE
> def_bool y
> - depends on HIBERNATION
> + depends on HIBERNATION || KEXEC_CORE
>
> config XEN_DOM0
> def_bool y
> diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
> index b96d8a6aac80..049cde429b1b 100644
> --- a/arch/arm64/include/asm/kexec.h
> +++ b/arch/arm64/include/asm/kexec.h
> @@ -105,6 +105,12 @@ extern const char arm64_kexec_el2_vectors[];
> * el2_vector If present means that relocation routine will go to EL1
> * from EL2 to do the copy, and then back to EL2 to do the jump
> * to new world.
> + * trans_ttbr0 idmap for relocation function and its argument
> + * trans_ttbr1 map for source/destination addresses.
> + * trans_t0sz t0sz for idmap page in trans_ttbr0
> + * src_addr start address for source pages.
> + * dst_addr start address for destination pages.
> + * copy_len Number of bytes that need to be copied
> */
> struct kern_reloc_arg {
> phys_addr_t head;
> @@ -114,6 +120,12 @@ struct kern_reloc_arg {
> phys_addr_t kern_arg2;
> phys_addr_t kern_arg3;
> phys_addr_t el2_vector;
> + phys_addr_t trans_ttbr0;
> + phys_addr_t trans_ttbr1;
> + unsigned long trans_t0sz;
> + unsigned long src_addr;
> + unsigned long dst_addr;
> + unsigned long copy_len;
> };
>
> #define ARCH_HAS_KIMAGE_ARCH
> diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
> index 8a9475be1b62..06278611451d 100644
> --- a/arch/arm64/kernel/asm-offsets.c
> +++ b/arch/arm64/kernel/asm-offsets.c
> @@ -160,6 +160,12 @@ int main(void)
> DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
> DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
> DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector));
> + DEFINE(KEXEC_KRELOC_TRANS_TTBR0, offsetof(struct kern_reloc_arg, trans_ttbr0));
> + DEFINE(KEXEC_KRELOC_TRANS_TTBR1, offsetof(struct kern_reloc_arg, trans_ttbr1));
> + DEFINE(KEXEC_KRELOC_TRANS_T0SZ, offsetof(struct kern_reloc_arg, trans_t0sz));
> + DEFINE(KEXEC_KRELOC_SRC_ADDR, offsetof(struct kern_reloc_arg, src_addr));
> + DEFINE(KEXEC_KRELOC_DST_ADDR, offsetof(struct kern_reloc_arg, dst_addr));
> + DEFINE(KEXEC_KRELOC_COPY_LEN, offsetof(struct kern_reloc_arg, copy_len));
> #endif
> return 0;
> }
> diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
> index 41d1e3ca13f8..dc1b7e5a54fb 100644
> --- a/arch/arm64/kernel/machine_kexec.c
> +++ b/arch/arm64/kernel/machine_kexec.c
> @@ -21,6 +21,7 @@
> #include <asm/mmu_context.h>
> #include <asm/page.h>
> #include <asm/sections.h>
> +#include <asm/trans_pgd.h>
>
> #include "cpu-reset.h"
>
> @@ -71,11 +72,91 @@ static void *kexec_page_alloc(void *arg)
> return page_address(page);
> }
>
> +/*
> + * Map source segments starting from src_va, and map destination
> + * segments starting from dst_va, and return size of copy in
> + * *copy_len argument.
> + * Relocation function essentially needs to do:
> + * memcpy(dst_va, src_va, copy_len);
> + */
> +static int map_segments(struct kimage *kimage, pgd_t *pgdp,
> + struct trans_pgd_info *info,
> + unsigned long src_va,
> + unsigned long dst_va,
> + unsigned long *copy_len)
> +{
> + unsigned long *ptr = 0;
> + unsigned long dest = 0;
> + unsigned long len = 0;
> + unsigned long entry, addr;
> + int rc;
> +
> + for (entry = kimage->head; !(entry & IND_DONE); entry = *ptr++) {
> + addr = entry & PAGE_MASK;
> +
> + switch (entry & IND_FLAGS) {
> + case IND_DESTINATION:
> + dest = addr;
> + break;
> + case IND_INDIRECTION:
> + ptr = __va(addr);
> + if (rc)
We're checking a potentially uninitialized value here. I think this
whole "if (rc) return rc;" conditional can go away.
Tyler
> + return rc;
> + break;
> + case IND_SOURCE:
> + rc = trans_pgd_map_page(info, pgdp, __va(addr),
> + src_va, PAGE_KERNEL);
> + if (rc)
> + return rc;
> + rc = trans_pgd_map_page(info, pgdp, __va(dest),
> + dst_va, PAGE_KERNEL);
> + if (rc)
> + return rc;
> + dest += PAGE_SIZE;
> + src_va += PAGE_SIZE;
> + dst_va += PAGE_SIZE;
> + len += PAGE_SIZE;
> + }
> + }
> + *copy_len = len;
> +
> + return 0;
> +}
> +
> +static int mmu_relocate_setup(struct kimage *kimage, void *reloc_code,
> + struct kern_reloc_arg *kern_reloc_arg)
> +{
> + struct trans_pgd_info info = {
> + .trans_alloc_page = kexec_page_alloc,
> + .trans_alloc_arg = kimage,
> + };
> + pgd_t *trans_pgd = kexec_page_alloc(kimage);
> + int rc;
> +
> + if (!trans_pgd)
> + return -ENOMEM;
> +
> + /* idmap relocation function */
> + rc = trans_pgd_idmap_page(&info, &kern_reloc_arg->trans_ttbr0,
> + &kern_reloc_arg->trans_t0sz, reloc_code);
> + if (rc)
> + return rc;
> +
> + kern_reloc_arg->src_addr = _PAGE_OFFSET(VA_BITS_MIN);
> + kern_reloc_arg->dst_addr = _PAGE_OFFSET(VA_BITS_MIN - 1);
> + kern_reloc_arg->trans_ttbr1 = phys_to_ttbr(__pa(trans_pgd));
> +
> + rc = map_segments(kimage, trans_pgd, &info, kern_reloc_arg->src_addr,
> + kern_reloc_arg->dst_addr, &kern_reloc_arg->copy_len);
> + return rc;
> +}
> +
> int machine_kexec_post_load(struct kimage *kimage)
> {
> void *reloc_code = page_to_virt(kimage->control_code_page);
> struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage);
> long func_offset, vector_offset, reloc_size;
> + int rc = 0;
>
> if (!kern_reloc_arg)
> return -ENOMEM;
> @@ -95,6 +176,14 @@ int machine_kexec_post_load(struct kimage *kimage)
> if (is_hyp_mode_available() && !is_kernel_in_hyp_mode())
> kern_reloc_arg->el2_vector = __pa(reloc_code) + vector_offset;
>
> + /*
> + * If relocation is not needed, we do not need to enable MMU in
> + * relocation routine, therefore do not create page tables for
> + * scenarios such as crash kernel
> + */
> + if (!(kimage->head & IND_DONE))
> + rc = mmu_relocate_setup(kimage, reloc_code, kern_reloc_arg);
> +
> kexec_image_info(kimage);
>
> /* Flush the reloc_code in preparation for its execution. */
> @@ -103,7 +192,7 @@ int machine_kexec_post_load(struct kimage *kimage)
> reloc_size);
> __flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg));
>
> - return 0;
> + return rc;
> }
>
> /**
> --
> 2.25.1
>
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for kexec
@ 2021-01-27 16:42 ` Tyler Hicks
0 siblings, 0 replies; 84+ messages in thread
From: Tyler Hicks @ 2021-01-27 16:42 UTC (permalink / raw)
To: Pavel Tatashin
Cc: sashal, mark.rutland, vladimir.murzin, corbet, catalin.marinas,
selindag, steve.capper, kexec, linux-kernel, jmorris, linux-mm,
james.morse, ebiederm, maz, matthias.bgg, rfontana, will, tglx,
linux-arm-kernel
On 2021-01-25 19:21:22, Pavel Tatashin wrote:
> I forgot to make changes to arch/arm64/Kconfig. The correct patch is
> below.
>
> ---
>
> From a2bc374320d7c7efd3c40644ad3d6d59a024b301 Mon Sep 17 00:00:00 2001
> From: Pavel Tatashin <pasha.tatashin@soleen.com>
> Date: Mon, 29 Jul 2019 21:24:25 -0400
> Subject: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for
> kexec
>
> Configure a page table located in kexec-safe memory that has
> the following mappings:
>
> 1. identity mapping for text of relocation function with executable
> permission.
> 2. va mappings for all source ranges
> 3. va mappings for all destination ranges.
>
> Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
> ---
> arch/arm64/Kconfig | 2 +-
> arch/arm64/include/asm/kexec.h | 12 ++++
> arch/arm64/kernel/asm-offsets.c | 6 ++
> arch/arm64/kernel/machine_kexec.c | 91 ++++++++++++++++++++++++++++++-
> 4 files changed, 109 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index fc0ed9d6e011..440abd0c0ee1 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -1134,7 +1134,7 @@ config CRASH_DUMP
>
> config TRANS_TABLE
> def_bool y
> - depends on HIBERNATION
> + depends on HIBERNATION || KEXEC_CORE
>
> config XEN_DOM0
> def_bool y
> diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
> index b96d8a6aac80..049cde429b1b 100644
> --- a/arch/arm64/include/asm/kexec.h
> +++ b/arch/arm64/include/asm/kexec.h
> @@ -105,6 +105,12 @@ extern const char arm64_kexec_el2_vectors[];
> * el2_vector If present means that relocation routine will go to EL1
> * from EL2 to do the copy, and then back to EL2 to do the jump
> * to new world.
> + * trans_ttbr0 idmap for relocation function and its argument
> + * trans_ttbr1 map for source/destination addresses.
> + * trans_t0sz t0sz for idmap page in trans_ttbr0
> + * src_addr start address for source pages.
> + * dst_addr start address for destination pages.
> + * copy_len Number of bytes that need to be copied
> */
> struct kern_reloc_arg {
> phys_addr_t head;
> @@ -114,6 +120,12 @@ struct kern_reloc_arg {
> phys_addr_t kern_arg2;
> phys_addr_t kern_arg3;
> phys_addr_t el2_vector;
> + phys_addr_t trans_ttbr0;
> + phys_addr_t trans_ttbr1;
> + unsigned long trans_t0sz;
> + unsigned long src_addr;
> + unsigned long dst_addr;
> + unsigned long copy_len;
> };
>
> #define ARCH_HAS_KIMAGE_ARCH
> diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
> index 8a9475be1b62..06278611451d 100644
> --- a/arch/arm64/kernel/asm-offsets.c
> +++ b/arch/arm64/kernel/asm-offsets.c
> @@ -160,6 +160,12 @@ int main(void)
> DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
> DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
> DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector));
> + DEFINE(KEXEC_KRELOC_TRANS_TTBR0, offsetof(struct kern_reloc_arg, trans_ttbr0));
> + DEFINE(KEXEC_KRELOC_TRANS_TTBR1, offsetof(struct kern_reloc_arg, trans_ttbr1));
> + DEFINE(KEXEC_KRELOC_TRANS_T0SZ, offsetof(struct kern_reloc_arg, trans_t0sz));
> + DEFINE(KEXEC_KRELOC_SRC_ADDR, offsetof(struct kern_reloc_arg, src_addr));
> + DEFINE(KEXEC_KRELOC_DST_ADDR, offsetof(struct kern_reloc_arg, dst_addr));
> + DEFINE(KEXEC_KRELOC_COPY_LEN, offsetof(struct kern_reloc_arg, copy_len));
> #endif
> return 0;
> }
> diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
> index 41d1e3ca13f8..dc1b7e5a54fb 100644
> --- a/arch/arm64/kernel/machine_kexec.c
> +++ b/arch/arm64/kernel/machine_kexec.c
> @@ -21,6 +21,7 @@
> #include <asm/mmu_context.h>
> #include <asm/page.h>
> #include <asm/sections.h>
> +#include <asm/trans_pgd.h>
>
> #include "cpu-reset.h"
>
> @@ -71,11 +72,91 @@ static void *kexec_page_alloc(void *arg)
> return page_address(page);
> }
>
> +/*
> + * Map source segments starting from src_va, and map destination
> + * segments starting from dst_va, and return size of copy in
> + * *copy_len argument.
> + * Relocation function essentially needs to do:
> + * memcpy(dst_va, src_va, copy_len);
> + */
> +static int map_segments(struct kimage *kimage, pgd_t *pgdp,
> + struct trans_pgd_info *info,
> + unsigned long src_va,
> + unsigned long dst_va,
> + unsigned long *copy_len)
> +{
> + unsigned long *ptr = 0;
> + unsigned long dest = 0;
> + unsigned long len = 0;
> + unsigned long entry, addr;
> + int rc;
> +
> + for (entry = kimage->head; !(entry & IND_DONE); entry = *ptr++) {
> + addr = entry & PAGE_MASK;
> +
> + switch (entry & IND_FLAGS) {
> + case IND_DESTINATION:
> + dest = addr;
> + break;
> + case IND_INDIRECTION:
> + ptr = __va(addr);
> + if (rc)
We're checking a potentially uninitialized value here. I think this
whole "if (rc) return rc;" conditional can go away.
Tyler
> + return rc;
> + break;
> + case IND_SOURCE:
> + rc = trans_pgd_map_page(info, pgdp, __va(addr),
> + src_va, PAGE_KERNEL);
> + if (rc)
> + return rc;
> + rc = trans_pgd_map_page(info, pgdp, __va(dest),
> + dst_va, PAGE_KERNEL);
> + if (rc)
> + return rc;
> + dest += PAGE_SIZE;
> + src_va += PAGE_SIZE;
> + dst_va += PAGE_SIZE;
> + len += PAGE_SIZE;
> + }
> + }
> + *copy_len = len;
> +
> + return 0;
> +}
> +
> +static int mmu_relocate_setup(struct kimage *kimage, void *reloc_code,
> + struct kern_reloc_arg *kern_reloc_arg)
> +{
> + struct trans_pgd_info info = {
> + .trans_alloc_page = kexec_page_alloc,
> + .trans_alloc_arg = kimage,
> + };
> + pgd_t *trans_pgd = kexec_page_alloc(kimage);
> + int rc;
> +
> + if (!trans_pgd)
> + return -ENOMEM;
> +
> + /* idmap relocation function */
> + rc = trans_pgd_idmap_page(&info, &kern_reloc_arg->trans_ttbr0,
> + &kern_reloc_arg->trans_t0sz, reloc_code);
> + if (rc)
> + return rc;
> +
> + kern_reloc_arg->src_addr = _PAGE_OFFSET(VA_BITS_MIN);
> + kern_reloc_arg->dst_addr = _PAGE_OFFSET(VA_BITS_MIN - 1);
> + kern_reloc_arg->trans_ttbr1 = phys_to_ttbr(__pa(trans_pgd));
> +
> + rc = map_segments(kimage, trans_pgd, &info, kern_reloc_arg->src_addr,
> + kern_reloc_arg->dst_addr, &kern_reloc_arg->copy_len);
> + return rc;
> +}
> +
> int machine_kexec_post_load(struct kimage *kimage)
> {
> void *reloc_code = page_to_virt(kimage->control_code_page);
> struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage);
> long func_offset, vector_offset, reloc_size;
> + int rc = 0;
>
> if (!kern_reloc_arg)
> return -ENOMEM;
> @@ -95,6 +176,14 @@ int machine_kexec_post_load(struct kimage *kimage)
> if (is_hyp_mode_available() && !is_kernel_in_hyp_mode())
> kern_reloc_arg->el2_vector = __pa(reloc_code) + vector_offset;
>
> + /*
> + * If relocation is not needed, we do not need to enable MMU in
> + * relocation routine, therefore do not create page tables for
> + * scenarios such as crash kernel
> + */
> + if (!(kimage->head & IND_DONE))
> + rc = mmu_relocate_setup(kimage, reloc_code, kern_reloc_arg);
> +
> kexec_image_info(kimage);
>
> /* Flush the reloc_code in preparation for its execution. */
> @@ -103,7 +192,7 @@ int machine_kexec_post_load(struct kimage *kimage)
> reloc_size);
> __flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg));
>
> - return 0;
> + return rc;
> }
>
> /**
> --
> 2.25.1
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for kexec
2021-01-27 16:42 ` Tyler Hicks
(?)
(?)
@ 2021-01-27 16:47 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-27 16:47 UTC (permalink / raw)
To: Tyler Hicks
Cc: James Morris, Sasha Levin, Eric W. Biederman, kexec mailing list,
LKML, Jonathan Corbet, Catalin Marinas, Will Deacon, Linux ARM,
Marc Zyngier, James Morse, Vladimir Murzin, Matthias Brugger,
linux-mm, Mark Rutland, steve.capper, rfontana, Thomas Gleixner,
Selin Dag
On Wed, Jan 27, 2021 at 11:42 AM Tyler Hicks
<tyhicks@linux.microsoft.com> wrote:
>
> On 2021-01-25 19:21:22, Pavel Tatashin wrote:
> > I forgot to make changes to arch/arm64/Kconfig. The correct patch is
> > below.
> >
> > ---
> >
> > From a2bc374320d7c7efd3c40644ad3d6d59a024b301 Mon Sep 17 00:00:00 2001
> > From: Pavel Tatashin <pasha.tatashin@soleen.com>
> > Date: Mon, 29 Jul 2019 21:24:25 -0400
> > Subject: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for
> > kexec
> >
> > Configure a page table located in kexec-safe memory that has
> > the following mappings:
> >
> > 1. identity mapping for text of relocation function with executable
> > permission.
> > 2. va mappings for all source ranges
> > 3. va mappings for all destination ranges.
> >
> > Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
> > ---
> > arch/arm64/Kconfig | 2 +-
> > arch/arm64/include/asm/kexec.h | 12 ++++
> > arch/arm64/kernel/asm-offsets.c | 6 ++
> > arch/arm64/kernel/machine_kexec.c | 91 ++++++++++++++++++++++++++++++-
> > 4 files changed, 109 insertions(+), 2 deletions(-)
> >
> > diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> > index fc0ed9d6e011..440abd0c0ee1 100644
> > --- a/arch/arm64/Kconfig
> > +++ b/arch/arm64/Kconfig
> > @@ -1134,7 +1134,7 @@ config CRASH_DUMP
> >
> > config TRANS_TABLE
> > def_bool y
> > - depends on HIBERNATION
> > + depends on HIBERNATION || KEXEC_CORE
> >
> > config XEN_DOM0
> > def_bool y
> > diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
> > index b96d8a6aac80..049cde429b1b 100644
> > --- a/arch/arm64/include/asm/kexec.h
> > +++ b/arch/arm64/include/asm/kexec.h
> > @@ -105,6 +105,12 @@ extern const char arm64_kexec_el2_vectors[];
> > * el2_vector If present means that relocation routine will go to EL1
> > * from EL2 to do the copy, and then back to EL2 to do the jump
> > * to new world.
> > + * trans_ttbr0 idmap for relocation function and its argument
> > + * trans_ttbr1 map for source/destination addresses.
> > + * trans_t0sz t0sz for idmap page in trans_ttbr0
> > + * src_addr start address for source pages.
> > + * dst_addr start address for destination pages.
> > + * copy_len Number of bytes that need to be copied
> > */
> > struct kern_reloc_arg {
> > phys_addr_t head;
> > @@ -114,6 +120,12 @@ struct kern_reloc_arg {
> > phys_addr_t kern_arg2;
> > phys_addr_t kern_arg3;
> > phys_addr_t el2_vector;
> > + phys_addr_t trans_ttbr0;
> > + phys_addr_t trans_ttbr1;
> > + unsigned long trans_t0sz;
> > + unsigned long src_addr;
> > + unsigned long dst_addr;
> > + unsigned long copy_len;
> > };
> >
> > #define ARCH_HAS_KIMAGE_ARCH
> > diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
> > index 8a9475be1b62..06278611451d 100644
> > --- a/arch/arm64/kernel/asm-offsets.c
> > +++ b/arch/arm64/kernel/asm-offsets.c
> > @@ -160,6 +160,12 @@ int main(void)
> > DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
> > DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
> > DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector));
> > + DEFINE(KEXEC_KRELOC_TRANS_TTBR0, offsetof(struct kern_reloc_arg, trans_ttbr0));
> > + DEFINE(KEXEC_KRELOC_TRANS_TTBR1, offsetof(struct kern_reloc_arg, trans_ttbr1));
> > + DEFINE(KEXEC_KRELOC_TRANS_T0SZ, offsetof(struct kern_reloc_arg, trans_t0sz));
> > + DEFINE(KEXEC_KRELOC_SRC_ADDR, offsetof(struct kern_reloc_arg, src_addr));
> > + DEFINE(KEXEC_KRELOC_DST_ADDR, offsetof(struct kern_reloc_arg, dst_addr));
> > + DEFINE(KEXEC_KRELOC_COPY_LEN, offsetof(struct kern_reloc_arg, copy_len));
> > #endif
> > return 0;
> > }
> > diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
> > index 41d1e3ca13f8..dc1b7e5a54fb 100644
> > --- a/arch/arm64/kernel/machine_kexec.c
> > +++ b/arch/arm64/kernel/machine_kexec.c
> > @@ -21,6 +21,7 @@
> > #include <asm/mmu_context.h>
> > #include <asm/page.h>
> > #include <asm/sections.h>
> > +#include <asm/trans_pgd.h>
> >
> > #include "cpu-reset.h"
> >
> > @@ -71,11 +72,91 @@ static void *kexec_page_alloc(void *arg)
> > return page_address(page);
> > }
> >
> > +/*
> > + * Map source segments starting from src_va, and map destination
> > + * segments starting from dst_va, and return size of copy in
> > + * *copy_len argument.
> > + * Relocation function essentially needs to do:
> > + * memcpy(dst_va, src_va, copy_len);
> > + */
> > +static int map_segments(struct kimage *kimage, pgd_t *pgdp,
> > + struct trans_pgd_info *info,
> > + unsigned long src_va,
> > + unsigned long dst_va,
> > + unsigned long *copy_len)
> > +{
> > + unsigned long *ptr = 0;
> > + unsigned long dest = 0;
> > + unsigned long len = 0;
> > + unsigned long entry, addr;
> > + int rc;
> > +
> > + for (entry = kimage->head; !(entry & IND_DONE); entry = *ptr++) {
> > + addr = entry & PAGE_MASK;
> > +
> > + switch (entry & IND_FLAGS) {
> > + case IND_DESTINATION:
> > + dest = addr;
> > + break;
> > + case IND_INDIRECTION:
> > + ptr = __va(addr);
> > + if (rc)
>
> We're checking a potentially uninitialized value here. I think this
> whole "if (rc) return rc;" conditional can go away.
Thank you Tyler. This is indeed not needed.
Pasha
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for kexec
@ 2021-01-27 16:47 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-27 16:47 UTC (permalink / raw)
To: Tyler Hicks
Cc: Sasha Levin, Mark Rutland, Vladimir Murzin, Jonathan Corbet,
Catalin Marinas, Selin Dag, steve.capper, kexec mailing list,
LKML, James Morris, linux-mm, James Morse, Eric W. Biederman,
Marc Zyngier, Matthias Brugger, rfontana, Will Deacon,
Thomas Gleixner, Linux ARM
On Wed, Jan 27, 2021 at 11:42 AM Tyler Hicks
<tyhicks@linux.microsoft.com> wrote:
>
> On 2021-01-25 19:21:22, Pavel Tatashin wrote:
> > I forgot to make changes to arch/arm64/Kconfig. The correct patch is
> > below.
> >
> > ---
> >
> > From a2bc374320d7c7efd3c40644ad3d6d59a024b301 Mon Sep 17 00:00:00 2001
> > From: Pavel Tatashin <pasha.tatashin@soleen.com>
> > Date: Mon, 29 Jul 2019 21:24:25 -0400
> > Subject: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for
> > kexec
> >
> > Configure a page table located in kexec-safe memory that has
> > the following mappings:
> >
> > 1. identity mapping for text of relocation function with executable
> > permission.
> > 2. va mappings for all source ranges
> > 3. va mappings for all destination ranges.
> >
> > Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
> > ---
> > arch/arm64/Kconfig | 2 +-
> > arch/arm64/include/asm/kexec.h | 12 ++++
> > arch/arm64/kernel/asm-offsets.c | 6 ++
> > arch/arm64/kernel/machine_kexec.c | 91 ++++++++++++++++++++++++++++++-
> > 4 files changed, 109 insertions(+), 2 deletions(-)
> >
> > diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> > index fc0ed9d6e011..440abd0c0ee1 100644
> > --- a/arch/arm64/Kconfig
> > +++ b/arch/arm64/Kconfig
> > @@ -1134,7 +1134,7 @@ config CRASH_DUMP
> >
> > config TRANS_TABLE
> > def_bool y
> > - depends on HIBERNATION
> > + depends on HIBERNATION || KEXEC_CORE
> >
> > config XEN_DOM0
> > def_bool y
> > diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
> > index b96d8a6aac80..049cde429b1b 100644
> > --- a/arch/arm64/include/asm/kexec.h
> > +++ b/arch/arm64/include/asm/kexec.h
> > @@ -105,6 +105,12 @@ extern const char arm64_kexec_el2_vectors[];
> > * el2_vector If present means that relocation routine will go to EL1
> > * from EL2 to do the copy, and then back to EL2 to do the jump
> > * to new world.
> > + * trans_ttbr0 idmap for relocation function and its argument
> > + * trans_ttbr1 map for source/destination addresses.
> > + * trans_t0sz t0sz for idmap page in trans_ttbr0
> > + * src_addr start address for source pages.
> > + * dst_addr start address for destination pages.
> > + * copy_len Number of bytes that need to be copied
> > */
> > struct kern_reloc_arg {
> > phys_addr_t head;
> > @@ -114,6 +120,12 @@ struct kern_reloc_arg {
> > phys_addr_t kern_arg2;
> > phys_addr_t kern_arg3;
> > phys_addr_t el2_vector;
> > + phys_addr_t trans_ttbr0;
> > + phys_addr_t trans_ttbr1;
> > + unsigned long trans_t0sz;
> > + unsigned long src_addr;
> > + unsigned long dst_addr;
> > + unsigned long copy_len;
> > };
> >
> > #define ARCH_HAS_KIMAGE_ARCH
> > diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
> > index 8a9475be1b62..06278611451d 100644
> > --- a/arch/arm64/kernel/asm-offsets.c
> > +++ b/arch/arm64/kernel/asm-offsets.c
> > @@ -160,6 +160,12 @@ int main(void)
> > DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
> > DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
> > DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector));
> > + DEFINE(KEXEC_KRELOC_TRANS_TTBR0, offsetof(struct kern_reloc_arg, trans_ttbr0));
> > + DEFINE(KEXEC_KRELOC_TRANS_TTBR1, offsetof(struct kern_reloc_arg, trans_ttbr1));
> > + DEFINE(KEXEC_KRELOC_TRANS_T0SZ, offsetof(struct kern_reloc_arg, trans_t0sz));
> > + DEFINE(KEXEC_KRELOC_SRC_ADDR, offsetof(struct kern_reloc_arg, src_addr));
> > + DEFINE(KEXEC_KRELOC_DST_ADDR, offsetof(struct kern_reloc_arg, dst_addr));
> > + DEFINE(KEXEC_KRELOC_COPY_LEN, offsetof(struct kern_reloc_arg, copy_len));
> > #endif
> > return 0;
> > }
> > diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
> > index 41d1e3ca13f8..dc1b7e5a54fb 100644
> > --- a/arch/arm64/kernel/machine_kexec.c
> > +++ b/arch/arm64/kernel/machine_kexec.c
> > @@ -21,6 +21,7 @@
> > #include <asm/mmu_context.h>
> > #include <asm/page.h>
> > #include <asm/sections.h>
> > +#include <asm/trans_pgd.h>
> >
> > #include "cpu-reset.h"
> >
> > @@ -71,11 +72,91 @@ static void *kexec_page_alloc(void *arg)
> > return page_address(page);
> > }
> >
> > +/*
> > + * Map source segments starting from src_va, and map destination
> > + * segments starting from dst_va, and return size of copy in
> > + * *copy_len argument.
> > + * Relocation function essentially needs to do:
> > + * memcpy(dst_va, src_va, copy_len);
> > + */
> > +static int map_segments(struct kimage *kimage, pgd_t *pgdp,
> > + struct trans_pgd_info *info,
> > + unsigned long src_va,
> > + unsigned long dst_va,
> > + unsigned long *copy_len)
> > +{
> > + unsigned long *ptr = 0;
> > + unsigned long dest = 0;
> > + unsigned long len = 0;
> > + unsigned long entry, addr;
> > + int rc;
> > +
> > + for (entry = kimage->head; !(entry & IND_DONE); entry = *ptr++) {
> > + addr = entry & PAGE_MASK;
> > +
> > + switch (entry & IND_FLAGS) {
> > + case IND_DESTINATION:
> > + dest = addr;
> > + break;
> > + case IND_INDIRECTION:
> > + ptr = __va(addr);
> > + if (rc)
>
> We're checking a potentially uninitialized value here. I think this
> whole "if (rc) return rc;" conditional can go away.
Thank you Tyler. This is indeed not needed.
Pasha
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for kexec
@ 2021-01-27 16:47 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-27 16:47 UTC (permalink / raw)
To: Tyler Hicks
Cc: Sasha Levin, Mark Rutland, Vladimir Murzin, Jonathan Corbet,
Catalin Marinas, Selin Dag, steve.capper, kexec mailing list,
LKML, James Morris, linux-mm, James Morse, Eric W. Biederman,
Marc Zyngier, Matthias Brugger, rfontana, Will Deacon,
Thomas Gleixner, Linux ARM
On Wed, Jan 27, 2021 at 11:42 AM Tyler Hicks
<tyhicks@linux.microsoft.com> wrote:
>
> On 2021-01-25 19:21:22, Pavel Tatashin wrote:
> > I forgot to make changes to arch/arm64/Kconfig. The correct patch is
> > below.
> >
> > ---
> >
> > From a2bc374320d7c7efd3c40644ad3d6d59a024b301 Mon Sep 17 00:00:00 2001
> > From: Pavel Tatashin <pasha.tatashin@soleen.com>
> > Date: Mon, 29 Jul 2019 21:24:25 -0400
> > Subject: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for
> > kexec
> >
> > Configure a page table located in kexec-safe memory that has
> > the following mappings:
> >
> > 1. identity mapping for text of relocation function with executable
> > permission.
> > 2. va mappings for all source ranges
> > 3. va mappings for all destination ranges.
> >
> > Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
> > ---
> > arch/arm64/Kconfig | 2 +-
> > arch/arm64/include/asm/kexec.h | 12 ++++
> > arch/arm64/kernel/asm-offsets.c | 6 ++
> > arch/arm64/kernel/machine_kexec.c | 91 ++++++++++++++++++++++++++++++-
> > 4 files changed, 109 insertions(+), 2 deletions(-)
> >
> > diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> > index fc0ed9d6e011..440abd0c0ee1 100644
> > --- a/arch/arm64/Kconfig
> > +++ b/arch/arm64/Kconfig
> > @@ -1134,7 +1134,7 @@ config CRASH_DUMP
> >
> > config TRANS_TABLE
> > def_bool y
> > - depends on HIBERNATION
> > + depends on HIBERNATION || KEXEC_CORE
> >
> > config XEN_DOM0
> > def_bool y
> > diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
> > index b96d8a6aac80..049cde429b1b 100644
> > --- a/arch/arm64/include/asm/kexec.h
> > +++ b/arch/arm64/include/asm/kexec.h
> > @@ -105,6 +105,12 @@ extern const char arm64_kexec_el2_vectors[];
> > * el2_vector If present means that relocation routine will go to EL1
> > * from EL2 to do the copy, and then back to EL2 to do the jump
> > * to new world.
> > + * trans_ttbr0 idmap for relocation function and its argument
> > + * trans_ttbr1 map for source/destination addresses.
> > + * trans_t0sz t0sz for idmap page in trans_ttbr0
> > + * src_addr start address for source pages.
> > + * dst_addr start address for destination pages.
> > + * copy_len Number of bytes that need to be copied
> > */
> > struct kern_reloc_arg {
> > phys_addr_t head;
> > @@ -114,6 +120,12 @@ struct kern_reloc_arg {
> > phys_addr_t kern_arg2;
> > phys_addr_t kern_arg3;
> > phys_addr_t el2_vector;
> > + phys_addr_t trans_ttbr0;
> > + phys_addr_t trans_ttbr1;
> > + unsigned long trans_t0sz;
> > + unsigned long src_addr;
> > + unsigned long dst_addr;
> > + unsigned long copy_len;
> > };
> >
> > #define ARCH_HAS_KIMAGE_ARCH
> > diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
> > index 8a9475be1b62..06278611451d 100644
> > --- a/arch/arm64/kernel/asm-offsets.c
> > +++ b/arch/arm64/kernel/asm-offsets.c
> > @@ -160,6 +160,12 @@ int main(void)
> > DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
> > DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
> > DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector));
> > + DEFINE(KEXEC_KRELOC_TRANS_TTBR0, offsetof(struct kern_reloc_arg, trans_ttbr0));
> > + DEFINE(KEXEC_KRELOC_TRANS_TTBR1, offsetof(struct kern_reloc_arg, trans_ttbr1));
> > + DEFINE(KEXEC_KRELOC_TRANS_T0SZ, offsetof(struct kern_reloc_arg, trans_t0sz));
> > + DEFINE(KEXEC_KRELOC_SRC_ADDR, offsetof(struct kern_reloc_arg, src_addr));
> > + DEFINE(KEXEC_KRELOC_DST_ADDR, offsetof(struct kern_reloc_arg, dst_addr));
> > + DEFINE(KEXEC_KRELOC_COPY_LEN, offsetof(struct kern_reloc_arg, copy_len));
> > #endif
> > return 0;
> > }
> > diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
> > index 41d1e3ca13f8..dc1b7e5a54fb 100644
> > --- a/arch/arm64/kernel/machine_kexec.c
> > +++ b/arch/arm64/kernel/machine_kexec.c
> > @@ -21,6 +21,7 @@
> > #include <asm/mmu_context.h>
> > #include <asm/page.h>
> > #include <asm/sections.h>
> > +#include <asm/trans_pgd.h>
> >
> > #include "cpu-reset.h"
> >
> > @@ -71,11 +72,91 @@ static void *kexec_page_alloc(void *arg)
> > return page_address(page);
> > }
> >
> > +/*
> > + * Map source segments starting from src_va, and map destination
> > + * segments starting from dst_va, and return size of copy in
> > + * *copy_len argument.
> > + * Relocation function essentially needs to do:
> > + * memcpy(dst_va, src_va, copy_len);
> > + */
> > +static int map_segments(struct kimage *kimage, pgd_t *pgdp,
> > + struct trans_pgd_info *info,
> > + unsigned long src_va,
> > + unsigned long dst_va,
> > + unsigned long *copy_len)
> > +{
> > + unsigned long *ptr = 0;
> > + unsigned long dest = 0;
> > + unsigned long len = 0;
> > + unsigned long entry, addr;
> > + int rc;
> > +
> > + for (entry = kimage->head; !(entry & IND_DONE); entry = *ptr++) {
> > + addr = entry & PAGE_MASK;
> > +
> > + switch (entry & IND_FLAGS) {
> > + case IND_DESTINATION:
> > + dest = addr;
> > + break;
> > + case IND_INDIRECTION:
> > + ptr = __va(addr);
> > + if (rc)
>
> We're checking a potentially uninitialized value here. I think this
> whole "if (rc) return rc;" conditional can go away.
Thank you Tyler. This is indeed not needed.
Pasha
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for kexec
@ 2021-01-27 16:47 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-27 16:47 UTC (permalink / raw)
To: Tyler Hicks
Cc: James Morris, Sasha Levin, Eric W. Biederman, kexec mailing list,
LKML, Jonathan Corbet, Catalin Marinas, Will Deacon, Linux ARM,
Marc Zyngier, James Morse, Vladimir Murzin, Matthias Brugger,
linux-mm, Mark Rutland, steve.capper, rfontana, Thomas Gleixner,
Selin Dag
On Wed, Jan 27, 2021 at 11:42 AM Tyler Hicks
<tyhicks@linux.microsoft.com> wrote:
>
> On 2021-01-25 19:21:22, Pavel Tatashin wrote:
> > I forgot to make changes to arch/arm64/Kconfig. The correct patch is
> > below.
> >
> > ---
> >
> > From a2bc374320d7c7efd3c40644ad3d6d59a024b301 Mon Sep 17 00:00:00 2001
> > From: Pavel Tatashin <pasha.tatashin@soleen.com>
> > Date: Mon, 29 Jul 2019 21:24:25 -0400
> > Subject: [PATCH v10 16/18] arm64: kexec: configure trans_pgd page table for
> > kexec
> >
> > Configure a page table located in kexec-safe memory that has
> > the following mappings:
> >
> > 1. identity mapping for text of relocation function with executable
> > permission.
> > 2. va mappings for all source ranges
> > 3. va mappings for all destination ranges.
> >
> > Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
> > ---
> > arch/arm64/Kconfig | 2 +-
> > arch/arm64/include/asm/kexec.h | 12 ++++
> > arch/arm64/kernel/asm-offsets.c | 6 ++
> > arch/arm64/kernel/machine_kexec.c | 91 ++++++++++++++++++++++++++++++-
> > 4 files changed, 109 insertions(+), 2 deletions(-)
> >
> > diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> > index fc0ed9d6e011..440abd0c0ee1 100644
> > --- a/arch/arm64/Kconfig
> > +++ b/arch/arm64/Kconfig
> > @@ -1134,7 +1134,7 @@ config CRASH_DUMP
> >
> > config TRANS_TABLE
> > def_bool y
> > - depends on HIBERNATION
> > + depends on HIBERNATION || KEXEC_CORE
> >
> > config XEN_DOM0
> > def_bool y
> > diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
> > index b96d8a6aac80..049cde429b1b 100644
> > --- a/arch/arm64/include/asm/kexec.h
> > +++ b/arch/arm64/include/asm/kexec.h
> > @@ -105,6 +105,12 @@ extern const char arm64_kexec_el2_vectors[];
> > * el2_vector If present means that relocation routine will go to EL1
> > * from EL2 to do the copy, and then back to EL2 to do the jump
> > * to new world.
> > + * trans_ttbr0 idmap for relocation function and its argument
> > + * trans_ttbr1 map for source/destination addresses.
> > + * trans_t0sz t0sz for idmap page in trans_ttbr0
> > + * src_addr start address for source pages.
> > + * dst_addr start address for destination pages.
> > + * copy_len Number of bytes that need to be copied
> > */
> > struct kern_reloc_arg {
> > phys_addr_t head;
> > @@ -114,6 +120,12 @@ struct kern_reloc_arg {
> > phys_addr_t kern_arg2;
> > phys_addr_t kern_arg3;
> > phys_addr_t el2_vector;
> > + phys_addr_t trans_ttbr0;
> > + phys_addr_t trans_ttbr1;
> > + unsigned long trans_t0sz;
> > + unsigned long src_addr;
> > + unsigned long dst_addr;
> > + unsigned long copy_len;
> > };
> >
> > #define ARCH_HAS_KIMAGE_ARCH
> > diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
> > index 8a9475be1b62..06278611451d 100644
> > --- a/arch/arm64/kernel/asm-offsets.c
> > +++ b/arch/arm64/kernel/asm-offsets.c
> > @@ -160,6 +160,12 @@ int main(void)
> > DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2));
> > DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3));
> > DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector));
> > + DEFINE(KEXEC_KRELOC_TRANS_TTBR0, offsetof(struct kern_reloc_arg, trans_ttbr0));
> > + DEFINE(KEXEC_KRELOC_TRANS_TTBR1, offsetof(struct kern_reloc_arg, trans_ttbr1));
> > + DEFINE(KEXEC_KRELOC_TRANS_T0SZ, offsetof(struct kern_reloc_arg, trans_t0sz));
> > + DEFINE(KEXEC_KRELOC_SRC_ADDR, offsetof(struct kern_reloc_arg, src_addr));
> > + DEFINE(KEXEC_KRELOC_DST_ADDR, offsetof(struct kern_reloc_arg, dst_addr));
> > + DEFINE(KEXEC_KRELOC_COPY_LEN, offsetof(struct kern_reloc_arg, copy_len));
> > #endif
> > return 0;
> > }
> > diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
> > index 41d1e3ca13f8..dc1b7e5a54fb 100644
> > --- a/arch/arm64/kernel/machine_kexec.c
> > +++ b/arch/arm64/kernel/machine_kexec.c
> > @@ -21,6 +21,7 @@
> > #include <asm/mmu_context.h>
> > #include <asm/page.h>
> > #include <asm/sections.h>
> > +#include <asm/trans_pgd.h>
> >
> > #include "cpu-reset.h"
> >
> > @@ -71,11 +72,91 @@ static void *kexec_page_alloc(void *arg)
> > return page_address(page);
> > }
> >
> > +/*
> > + * Map source segments starting from src_va, and map destination
> > + * segments starting from dst_va, and return size of copy in
> > + * *copy_len argument.
> > + * Relocation function essentially needs to do:
> > + * memcpy(dst_va, src_va, copy_len);
> > + */
> > +static int map_segments(struct kimage *kimage, pgd_t *pgdp,
> > + struct trans_pgd_info *info,
> > + unsigned long src_va,
> > + unsigned long dst_va,
> > + unsigned long *copy_len)
> > +{
> > + unsigned long *ptr = 0;
> > + unsigned long dest = 0;
> > + unsigned long len = 0;
> > + unsigned long entry, addr;
> > + int rc;
> > +
> > + for (entry = kimage->head; !(entry & IND_DONE); entry = *ptr++) {
> > + addr = entry & PAGE_MASK;
> > +
> > + switch (entry & IND_FLAGS) {
> > + case IND_DESTINATION:
> > + dest = addr;
> > + break;
> > + case IND_INDIRECTION:
> > + ptr = __va(addr);
> > + if (rc)
>
> We're checking a potentially uninitialized value here. I think this
> whole "if (rc) return rc;" conditional can go away.
Thank you Tyler. This is indeed not needed.
Pasha
^ permalink raw reply [flat|nested] 84+ messages in thread
* [PATCH v10 17/18] arm64: kexec: enable MMU during kexec relocation
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-25 19:19 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Now, that we have transitional page tables configured, temporarily enable
MMU to allow faster relocation of segments to final destination.
The performance data: for a moderate size kernel + initramfs: 25M the
relocation was taking 0.382s, with enabled MMU it now takes
0.019s only or x20 improvement.
The time is proportional to the size of relocation, therefore if initramfs
is larger, 100M it could take over a second.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/kernel/relocate_kernel.S | 131 ++++++++++++++++++----------
1 file changed, 87 insertions(+), 44 deletions(-)
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index c6178b1a4e60..9c60981a6911 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -4,6 +4,8 @@
*
* Copyright (C) Linaro.
* Copyright (C) Huawei Futurewei Technologies.
+ * Copyright (C) 2020, Microsoft Corporation.
+ * Pavel Tatashin <pasha.tatashin@soleen.com>
*/
#include <linux/kexec.h>
@@ -14,6 +16,54 @@
#include <asm/page.h>
#include <asm/sysreg.h>
+.macro tlb_invalidate
+ dsb sy
+ dsb ish
+ tlbi vmalle1
+ dsb ish
+ isb
+.endm
+
+.macro turn_off_mmu tmp1, tmp2
+ mrs \tmp1, sctlr_el1
+ mov_q \tmp2, SCTLR_ELx_FLAGS
+ bic \tmp1, \tmp1, \tmp2
+ pre_disable_mmu_workaround
+ msr sctlr_el1, \tmp1
+ isb
+.endm
+
+.macro turn_on_mmu tmp1, tmp2
+ mrs \tmp1, sctlr_el1
+ mov_q \tmp2, SCTLR_ELx_FLAGS
+ orr \tmp1, \tmp1, \tmp2
+ msr sctlr_el1, \tmp1
+ ic iallu
+ dsb nsh
+ isb
+.endm
+
+/*
+ * Set ttbr0 and ttbr1, called while MMU is disabled, so no need to temporarily
+ * set zero_page table. Invalidate TLB after new tables are set.
+ */
+.macro set_ttbr arg, tmp1, tmp2
+ ldr \tmp1, [\arg, #KEXEC_KRELOC_TRANS_TTBR0]
+ msr ttbr0_el1, \tmp1
+ ldr \tmp1, [\arg, #KEXEC_KRELOC_TRANS_TTBR1]
+ offset_ttbr1 \tmp1, \tmp2
+ msr ttbr1_el1, \tmp1
+ isb
+.endm
+
+/* Set T0SZ to match the requirements of idmap page */
+.macro set_tcr_t0sz arg, tmp1, tmp2
+ ldr \tmp2, [\arg, #KEXEC_KRELOC_TRANS_T0SZ]
+ mrs \tmp1, tcr_el1
+ bfi \tmp1, \tmp2, TCR_T0SZ_OFFSET, TCR_TxSZ_WIDTH
+ msr tcr_el1, \tmp1
+.endm
+
.macro el1_sync_64
.align 7
br x4 /* Jump to new world from el2 */
@@ -36,56 +86,49 @@
* symbols arm64_relocate_new_kernel and arm64_relocate_new_kernel_end. The
* machine_kexec() routine will copy arm64_relocate_new_kernel to the kexec
* safe memory that has been set up to be preserved during the copy operation.
+ *
+ * This function temporarily enables MMU if kernel relocation is needed.
+ * Also, if we enter this function at EL2 on non-VHE kernel, we temporarily go
+ * to EL1 to enable MMU, and escalate back to EL2 at the end to do the jump to
+ * the new kernel. This is determined by presence of el2_vector.
*/
SYM_CODE_START(arm64_relocate_new_kernel)
- /* Check if the new image needs relocation. */
- ldr x16, [x0, #KEXEC_KRELOC_HEAD] /* x16 = kimage_head */
- tbnz x16, IND_DONE_BIT, .Ldone
- raw_dcache_line_size x15, x1 /* x15 = dcache line size */
-.Lloop:
- and x12, x16, PAGE_MASK /* x12 = addr */
-
- /* Test the entry flags. */
-.Ltest_source:
- tbz x16, IND_SOURCE_BIT, .Ltest_indirection
-
- /* Invalidate dest page to PoC. */
- mov x2, x13
- add x20, x2, #PAGE_SIZE
- sub x1, x15, #1
- bic x2, x2, x1
-2: dc ivac, x2
- add x2, x2, x15
- cmp x2, x20
- b.lo 2b
- dsb sy
-
- copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8
- b .Lnext
-.Ltest_indirection:
- tbz x16, IND_INDIRECTION_BIT, .Ltest_destination
- mov x14, x12 /* ptr = addr */
- b .Lnext
-.Ltest_destination:
- tbz x16, IND_DESTINATION_BIT, .Lnext
- mov x13, x12 /* dest = addr */
-.Lnext:
- ldr x16, [x14], #8 /* entry = *ptr++ */
- tbz x16, IND_DONE_BIT, .Lloop /* while (!(entry & DONE)) */
-.Ldone:
- /* wait for writes from copy_page to finish */
- dsb nsh
- ic iallu
- dsb nsh
- isb
-
- /* Start new image. */
- ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 = kimage_start */
+ mov x20, xzr /* x20 will hold vector value */
+ ldr x11, [x0, #KEXEC_KRELOC_COPY_LEN]
+ cbz x11, 5f /* Check if need to relocate */
+ ldr x20, [x0, #KEXEC_KRELOC_EL2_VECTOR]
+ cbz x20, 2f /* need to reduce to EL1? */
+ msr vbar_el2, x20 /* el2_vector present, means */
+ adr x1, 2f /* we will do copy in el1 but */
+ msr elr_el2, x1 /* do final jump from el2 */
+ eret /* Reduce to EL1 */
+2: set_tcr_t0sz x0, x1, x2 /* Set t0sz for idmaped page */
+ set_ttbr x0, x1, x2 /* Set our page tables */
+ tlb_invalidate
+ ldr x1, [x0, #KEXEC_KRELOC_DST_ADDR]; /* arg is not idmapped so */
+ ldr x2, [x0, #KEXEC_KRELOC_SRC_ADDR]; /* read before MMU is on */
+ turn_on_mmu x3, x4 /* Turn MMU back on */
+ mov x12, x1 /* x12 dst backup */
+3: copy_page x1, x2, x3, x4, x5, x6, x7, x8, x9, x10
+ sub x11, x11, #PAGE_SIZE
+ cbnz x11, 3b /* page copy loop */
+ raw_dcache_line_size x2, x3 /* x2 = dcache line size */
+ sub x3, x2, #1 /* x3 = dcache_size - 1 */
+ bic x12, x12, x3
+4: dc cvau, x12 /* Flush D-cache */
+ add x12, x12, x2
+ cmp x12, x1 /* Compare to dst + len */
+ b.ne 4b /* D-cache flush loop */
+ turn_off_mmu x1, x2 /* Turn off MMU */
+ tlb_invalidate /* Invalidate TLB */
+5: ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 = kimage_start */
ldr x3, [x0, #KEXEC_KRELOC_KERN_ARG3]
ldr x2, [x0, #KEXEC_KRELOC_KERN_ARG2]
ldr x1, [x0, #KEXEC_KRELOC_KERN_ARG1]
ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */
- br x4
+ cbnz x20, 6f /* need to escalate to el2? */
+ br x4 /* Jump to new world */
+6: hvc #0 /* enters kexec_el1_sync */
SYM_CODE_END(arm64_relocate_new_kernel)
/* el2 vectors - switch el2 here while we restore the memory image. */
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 17/18] arm64: kexec: enable MMU during kexec relocation
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Now, that we have transitional page tables configured, temporarily enable
MMU to allow faster relocation of segments to final destination.
The performance data: for a moderate size kernel + initramfs: 25M the
relocation was taking 0.382s, with enabled MMU it now takes
0.019s only or x20 improvement.
The time is proportional to the size of relocation, therefore if initramfs
is larger, 100M it could take over a second.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/kernel/relocate_kernel.S | 131 ++++++++++++++++++----------
1 file changed, 87 insertions(+), 44 deletions(-)
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index c6178b1a4e60..9c60981a6911 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -4,6 +4,8 @@
*
* Copyright (C) Linaro.
* Copyright (C) Huawei Futurewei Technologies.
+ * Copyright (C) 2020, Microsoft Corporation.
+ * Pavel Tatashin <pasha.tatashin@soleen.com>
*/
#include <linux/kexec.h>
@@ -14,6 +16,54 @@
#include <asm/page.h>
#include <asm/sysreg.h>
+.macro tlb_invalidate
+ dsb sy
+ dsb ish
+ tlbi vmalle1
+ dsb ish
+ isb
+.endm
+
+.macro turn_off_mmu tmp1, tmp2
+ mrs \tmp1, sctlr_el1
+ mov_q \tmp2, SCTLR_ELx_FLAGS
+ bic \tmp1, \tmp1, \tmp2
+ pre_disable_mmu_workaround
+ msr sctlr_el1, \tmp1
+ isb
+.endm
+
+.macro turn_on_mmu tmp1, tmp2
+ mrs \tmp1, sctlr_el1
+ mov_q \tmp2, SCTLR_ELx_FLAGS
+ orr \tmp1, \tmp1, \tmp2
+ msr sctlr_el1, \tmp1
+ ic iallu
+ dsb nsh
+ isb
+.endm
+
+/*
+ * Set ttbr0 and ttbr1, called while MMU is disabled, so no need to temporarily
+ * set zero_page table. Invalidate TLB after new tables are set.
+ */
+.macro set_ttbr arg, tmp1, tmp2
+ ldr \tmp1, [\arg, #KEXEC_KRELOC_TRANS_TTBR0]
+ msr ttbr0_el1, \tmp1
+ ldr \tmp1, [\arg, #KEXEC_KRELOC_TRANS_TTBR1]
+ offset_ttbr1 \tmp1, \tmp2
+ msr ttbr1_el1, \tmp1
+ isb
+.endm
+
+/* Set T0SZ to match the requirements of idmap page */
+.macro set_tcr_t0sz arg, tmp1, tmp2
+ ldr \tmp2, [\arg, #KEXEC_KRELOC_TRANS_T0SZ]
+ mrs \tmp1, tcr_el1
+ bfi \tmp1, \tmp2, TCR_T0SZ_OFFSET, TCR_TxSZ_WIDTH
+ msr tcr_el1, \tmp1
+.endm
+
.macro el1_sync_64
.align 7
br x4 /* Jump to new world from el2 */
@@ -36,56 +86,49 @@
* symbols arm64_relocate_new_kernel and arm64_relocate_new_kernel_end. The
* machine_kexec() routine will copy arm64_relocate_new_kernel to the kexec
* safe memory that has been set up to be preserved during the copy operation.
+ *
+ * This function temporarily enables MMU if kernel relocation is needed.
+ * Also, if we enter this function at EL2 on non-VHE kernel, we temporarily go
+ * to EL1 to enable MMU, and escalate back to EL2 at the end to do the jump to
+ * the new kernel. This is determined by presence of el2_vector.
*/
SYM_CODE_START(arm64_relocate_new_kernel)
- /* Check if the new image needs relocation. */
- ldr x16, [x0, #KEXEC_KRELOC_HEAD] /* x16 = kimage_head */
- tbnz x16, IND_DONE_BIT, .Ldone
- raw_dcache_line_size x15, x1 /* x15 = dcache line size */
-.Lloop:
- and x12, x16, PAGE_MASK /* x12 = addr */
-
- /* Test the entry flags. */
-.Ltest_source:
- tbz x16, IND_SOURCE_BIT, .Ltest_indirection
-
- /* Invalidate dest page to PoC. */
- mov x2, x13
- add x20, x2, #PAGE_SIZE
- sub x1, x15, #1
- bic x2, x2, x1
-2: dc ivac, x2
- add x2, x2, x15
- cmp x2, x20
- b.lo 2b
- dsb sy
-
- copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8
- b .Lnext
-.Ltest_indirection:
- tbz x16, IND_INDIRECTION_BIT, .Ltest_destination
- mov x14, x12 /* ptr = addr */
- b .Lnext
-.Ltest_destination:
- tbz x16, IND_DESTINATION_BIT, .Lnext
- mov x13, x12 /* dest = addr */
-.Lnext:
- ldr x16, [x14], #8 /* entry = *ptr++ */
- tbz x16, IND_DONE_BIT, .Lloop /* while (!(entry & DONE)) */
-.Ldone:
- /* wait for writes from copy_page to finish */
- dsb nsh
- ic iallu
- dsb nsh
- isb
-
- /* Start new image. */
- ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 = kimage_start */
+ mov x20, xzr /* x20 will hold vector value */
+ ldr x11, [x0, #KEXEC_KRELOC_COPY_LEN]
+ cbz x11, 5f /* Check if need to relocate */
+ ldr x20, [x0, #KEXEC_KRELOC_EL2_VECTOR]
+ cbz x20, 2f /* need to reduce to EL1? */
+ msr vbar_el2, x20 /* el2_vector present, means */
+ adr x1, 2f /* we will do copy in el1 but */
+ msr elr_el2, x1 /* do final jump from el2 */
+ eret /* Reduce to EL1 */
+2: set_tcr_t0sz x0, x1, x2 /* Set t0sz for idmaped page */
+ set_ttbr x0, x1, x2 /* Set our page tables */
+ tlb_invalidate
+ ldr x1, [x0, #KEXEC_KRELOC_DST_ADDR]; /* arg is not idmapped so */
+ ldr x2, [x0, #KEXEC_KRELOC_SRC_ADDR]; /* read before MMU is on */
+ turn_on_mmu x3, x4 /* Turn MMU back on */
+ mov x12, x1 /* x12 dst backup */
+3: copy_page x1, x2, x3, x4, x5, x6, x7, x8, x9, x10
+ sub x11, x11, #PAGE_SIZE
+ cbnz x11, 3b /* page copy loop */
+ raw_dcache_line_size x2, x3 /* x2 = dcache line size */
+ sub x3, x2, #1 /* x3 = dcache_size - 1 */
+ bic x12, x12, x3
+4: dc cvau, x12 /* Flush D-cache */
+ add x12, x12, x2
+ cmp x12, x1 /* Compare to dst + len */
+ b.ne 4b /* D-cache flush loop */
+ turn_off_mmu x1, x2 /* Turn off MMU */
+ tlb_invalidate /* Invalidate TLB */
+5: ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 = kimage_start */
ldr x3, [x0, #KEXEC_KRELOC_KERN_ARG3]
ldr x2, [x0, #KEXEC_KRELOC_KERN_ARG2]
ldr x1, [x0, #KEXEC_KRELOC_KERN_ARG1]
ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */
- br x4
+ cbnz x20, 6f /* need to escalate to el2? */
+ br x4 /* Jump to new world */
+6: hvc #0 /* enters kexec_el1_sync */
SYM_CODE_END(arm64_relocate_new_kernel)
/* el2 vectors - switch el2 here while we restore the memory image. */
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 17/18] arm64: kexec: enable MMU during kexec relocation
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Now, that we have transitional page tables configured, temporarily enable
MMU to allow faster relocation of segments to final destination.
The performance data: for a moderate size kernel + initramfs: 25M the
relocation was taking 0.382s, with enabled MMU it now takes
0.019s only or x20 improvement.
The time is proportional to the size of relocation, therefore if initramfs
is larger, 100M it could take over a second.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/kernel/relocate_kernel.S | 131 ++++++++++++++++++----------
1 file changed, 87 insertions(+), 44 deletions(-)
diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S
index c6178b1a4e60..9c60981a6911 100644
--- a/arch/arm64/kernel/relocate_kernel.S
+++ b/arch/arm64/kernel/relocate_kernel.S
@@ -4,6 +4,8 @@
*
* Copyright (C) Linaro.
* Copyright (C) Huawei Futurewei Technologies.
+ * Copyright (C) 2020, Microsoft Corporation.
+ * Pavel Tatashin <pasha.tatashin@soleen.com>
*/
#include <linux/kexec.h>
@@ -14,6 +16,54 @@
#include <asm/page.h>
#include <asm/sysreg.h>
+.macro tlb_invalidate
+ dsb sy
+ dsb ish
+ tlbi vmalle1
+ dsb ish
+ isb
+.endm
+
+.macro turn_off_mmu tmp1, tmp2
+ mrs \tmp1, sctlr_el1
+ mov_q \tmp2, SCTLR_ELx_FLAGS
+ bic \tmp1, \tmp1, \tmp2
+ pre_disable_mmu_workaround
+ msr sctlr_el1, \tmp1
+ isb
+.endm
+
+.macro turn_on_mmu tmp1, tmp2
+ mrs \tmp1, sctlr_el1
+ mov_q \tmp2, SCTLR_ELx_FLAGS
+ orr \tmp1, \tmp1, \tmp2
+ msr sctlr_el1, \tmp1
+ ic iallu
+ dsb nsh
+ isb
+.endm
+
+/*
+ * Set ttbr0 and ttbr1, called while MMU is disabled, so no need to temporarily
+ * set zero_page table. Invalidate TLB after new tables are set.
+ */
+.macro set_ttbr arg, tmp1, tmp2
+ ldr \tmp1, [\arg, #KEXEC_KRELOC_TRANS_TTBR0]
+ msr ttbr0_el1, \tmp1
+ ldr \tmp1, [\arg, #KEXEC_KRELOC_TRANS_TTBR1]
+ offset_ttbr1 \tmp1, \tmp2
+ msr ttbr1_el1, \tmp1
+ isb
+.endm
+
+/* Set T0SZ to match the requirements of idmap page */
+.macro set_tcr_t0sz arg, tmp1, tmp2
+ ldr \tmp2, [\arg, #KEXEC_KRELOC_TRANS_T0SZ]
+ mrs \tmp1, tcr_el1
+ bfi \tmp1, \tmp2, TCR_T0SZ_OFFSET, TCR_TxSZ_WIDTH
+ msr tcr_el1, \tmp1
+.endm
+
.macro el1_sync_64
.align 7
br x4 /* Jump to new world from el2 */
@@ -36,56 +86,49 @@
* symbols arm64_relocate_new_kernel and arm64_relocate_new_kernel_end. The
* machine_kexec() routine will copy arm64_relocate_new_kernel to the kexec
* safe memory that has been set up to be preserved during the copy operation.
+ *
+ * This function temporarily enables MMU if kernel relocation is needed.
+ * Also, if we enter this function at EL2 on non-VHE kernel, we temporarily go
+ * to EL1 to enable MMU, and escalate back to EL2 at the end to do the jump to
+ * the new kernel. This is determined by presence of el2_vector.
*/
SYM_CODE_START(arm64_relocate_new_kernel)
- /* Check if the new image needs relocation. */
- ldr x16, [x0, #KEXEC_KRELOC_HEAD] /* x16 = kimage_head */
- tbnz x16, IND_DONE_BIT, .Ldone
- raw_dcache_line_size x15, x1 /* x15 = dcache line size */
-.Lloop:
- and x12, x16, PAGE_MASK /* x12 = addr */
-
- /* Test the entry flags. */
-.Ltest_source:
- tbz x16, IND_SOURCE_BIT, .Ltest_indirection
-
- /* Invalidate dest page to PoC. */
- mov x2, x13
- add x20, x2, #PAGE_SIZE
- sub x1, x15, #1
- bic x2, x2, x1
-2: dc ivac, x2
- add x2, x2, x15
- cmp x2, x20
- b.lo 2b
- dsb sy
-
- copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8
- b .Lnext
-.Ltest_indirection:
- tbz x16, IND_INDIRECTION_BIT, .Ltest_destination
- mov x14, x12 /* ptr = addr */
- b .Lnext
-.Ltest_destination:
- tbz x16, IND_DESTINATION_BIT, .Lnext
- mov x13, x12 /* dest = addr */
-.Lnext:
- ldr x16, [x14], #8 /* entry = *ptr++ */
- tbz x16, IND_DONE_BIT, .Lloop /* while (!(entry & DONE)) */
-.Ldone:
- /* wait for writes from copy_page to finish */
- dsb nsh
- ic iallu
- dsb nsh
- isb
-
- /* Start new image. */
- ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 = kimage_start */
+ mov x20, xzr /* x20 will hold vector value */
+ ldr x11, [x0, #KEXEC_KRELOC_COPY_LEN]
+ cbz x11, 5f /* Check if need to relocate */
+ ldr x20, [x0, #KEXEC_KRELOC_EL2_VECTOR]
+ cbz x20, 2f /* need to reduce to EL1? */
+ msr vbar_el2, x20 /* el2_vector present, means */
+ adr x1, 2f /* we will do copy in el1 but */
+ msr elr_el2, x1 /* do final jump from el2 */
+ eret /* Reduce to EL1 */
+2: set_tcr_t0sz x0, x1, x2 /* Set t0sz for idmaped page */
+ set_ttbr x0, x1, x2 /* Set our page tables */
+ tlb_invalidate
+ ldr x1, [x0, #KEXEC_KRELOC_DST_ADDR]; /* arg is not idmapped so */
+ ldr x2, [x0, #KEXEC_KRELOC_SRC_ADDR]; /* read before MMU is on */
+ turn_on_mmu x3, x4 /* Turn MMU back on */
+ mov x12, x1 /* x12 dst backup */
+3: copy_page x1, x2, x3, x4, x5, x6, x7, x8, x9, x10
+ sub x11, x11, #PAGE_SIZE
+ cbnz x11, 3b /* page copy loop */
+ raw_dcache_line_size x2, x3 /* x2 = dcache line size */
+ sub x3, x2, #1 /* x3 = dcache_size - 1 */
+ bic x12, x12, x3
+4: dc cvau, x12 /* Flush D-cache */
+ add x12, x12, x2
+ cmp x12, x1 /* Compare to dst + len */
+ b.ne 4b /* D-cache flush loop */
+ turn_off_mmu x1, x2 /* Turn off MMU */
+ tlb_invalidate /* Invalidate TLB */
+5: ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 = kimage_start */
ldr x3, [x0, #KEXEC_KRELOC_KERN_ARG3]
ldr x2, [x0, #KEXEC_KRELOC_KERN_ARG2]
ldr x1, [x0, #KEXEC_KRELOC_KERN_ARG1]
ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */
- br x4
+ cbnz x20, 6f /* need to escalate to el2? */
+ br x4 /* Jump to new world */
+6: hvc #0 /* enters kexec_el1_sync */
SYM_CODE_END(arm64_relocate_new_kernel)
/* el2 vectors - switch el2 here while we restore the memory image. */
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 18/18] arm64: kexec: remove head from relocation argument
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-25 19:19 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Now, that relocation is done using virtual addresses, reloc_arg->head
is not needed anymore.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/include/asm/kexec.h | 2 --
arch/arm64/kernel/asm-offsets.c | 1 -
arch/arm64/kernel/machine_kexec.c | 1 -
3 files changed, 4 deletions(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index 049cde429b1b..2fa4109bd582 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -97,7 +97,6 @@ extern const char arm64_kexec_el2_vectors[];
/*
* kern_reloc_arg is passed to kernel relocation function as an argument.
- * head kimage->head, allows to traverse through relocation segments.
* entry_addr kimage->start, where to jump from relocation function (new
* kernel, or purgatory entry address).
* kern_arg0 first argument to kernel is its dtb address. The other
@@ -113,7 +112,6 @@ extern const char arm64_kexec_el2_vectors[];
* copy_len Number of bytes that need to be copied
*/
struct kern_reloc_arg {
- phys_addr_t head;
phys_addr_t entry_addr;
phys_addr_t kern_arg0;
phys_addr_t kern_arg1;
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 06278611451d..94f050ad6471 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -153,7 +153,6 @@ int main(void)
BLANK();
#endif
#ifdef CONFIG_KEXEC_CORE
- DEFINE(KEXEC_KRELOC_HEAD, offsetof(struct kern_reloc_arg, head));
DEFINE(KEXEC_KRELOC_ENTRY_ADDR, offsetof(struct kern_reloc_arg, entry_addr));
DEFINE(KEXEC_KRELOC_KERN_ARG0, offsetof(struct kern_reloc_arg, kern_arg0));
DEFINE(KEXEC_KRELOC_KERN_ARG1, offsetof(struct kern_reloc_arg, kern_arg1));
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index dc1b7e5a54fb..c2dff232a85b 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -168,7 +168,6 @@ int machine_kexec_post_load(struct kimage *kimage)
memcpy(reloc_code, __relocate_new_kernel_start, reloc_size);
kimage->arch.kern_reloc = __pa(reloc_code) + func_offset;
kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg);
- kern_reloc_arg->head = kimage->head;
kern_reloc_arg->entry_addr = kimage->start;
kern_reloc_arg->kern_arg0 = kimage->arch.dtb_mem;
--
2.25.1
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 18/18] arm64: kexec: remove head from relocation argument
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Now, that relocation is done using virtual addresses, reloc_arg->head
is not needed anymore.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/include/asm/kexec.h | 2 --
arch/arm64/kernel/asm-offsets.c | 1 -
arch/arm64/kernel/machine_kexec.c | 1 -
3 files changed, 4 deletions(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index 049cde429b1b..2fa4109bd582 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -97,7 +97,6 @@ extern const char arm64_kexec_el2_vectors[];
/*
* kern_reloc_arg is passed to kernel relocation function as an argument.
- * head kimage->head, allows to traverse through relocation segments.
* entry_addr kimage->start, where to jump from relocation function (new
* kernel, or purgatory entry address).
* kern_arg0 first argument to kernel is its dtb address. The other
@@ -113,7 +112,6 @@ extern const char arm64_kexec_el2_vectors[];
* copy_len Number of bytes that need to be copied
*/
struct kern_reloc_arg {
- phys_addr_t head;
phys_addr_t entry_addr;
phys_addr_t kern_arg0;
phys_addr_t kern_arg1;
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 06278611451d..94f050ad6471 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -153,7 +153,6 @@ int main(void)
BLANK();
#endif
#ifdef CONFIG_KEXEC_CORE
- DEFINE(KEXEC_KRELOC_HEAD, offsetof(struct kern_reloc_arg, head));
DEFINE(KEXEC_KRELOC_ENTRY_ADDR, offsetof(struct kern_reloc_arg, entry_addr));
DEFINE(KEXEC_KRELOC_KERN_ARG0, offsetof(struct kern_reloc_arg, kern_arg0));
DEFINE(KEXEC_KRELOC_KERN_ARG1, offsetof(struct kern_reloc_arg, kern_arg1));
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index dc1b7e5a54fb..c2dff232a85b 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -168,7 +168,6 @@ int machine_kexec_post_load(struct kimage *kimage)
memcpy(reloc_code, __relocate_new_kernel_start, reloc_size);
kimage->arch.kern_reloc = __pa(reloc_code) + func_offset;
kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg);
- kern_reloc_arg->head = kimage->head;
kern_reloc_arg->entry_addr = kimage->start;
kern_reloc_arg->kern_arg0 = kimage->arch.dtb_mem;
--
2.25.1
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply related [flat|nested] 84+ messages in thread
* [PATCH v10 18/18] arm64: kexec: remove head from relocation argument
@ 2021-01-25 19:19 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-25 19:19 UTC (permalink / raw)
To: pasha.tatashin, jmorris, sashal, ebiederm, kexec, linux-kernel,
corbet, catalin.marinas, will, linux-arm-kernel, maz,
james.morse, vladimir.murzin, matthias.bgg, linux-mm,
mark.rutland, steve.capper, rfontana, tglx, selindag, tyhicks
Now, that relocation is done using virtual addresses, reloc_arg->head
is not needed anymore.
Signed-off-by: Pavel Tatashin <pasha.tatashin@soleen.com>
---
arch/arm64/include/asm/kexec.h | 2 --
arch/arm64/kernel/asm-offsets.c | 1 -
arch/arm64/kernel/machine_kexec.c | 1 -
3 files changed, 4 deletions(-)
diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h
index 049cde429b1b..2fa4109bd582 100644
--- a/arch/arm64/include/asm/kexec.h
+++ b/arch/arm64/include/asm/kexec.h
@@ -97,7 +97,6 @@ extern const char arm64_kexec_el2_vectors[];
/*
* kern_reloc_arg is passed to kernel relocation function as an argument.
- * head kimage->head, allows to traverse through relocation segments.
* entry_addr kimage->start, where to jump from relocation function (new
* kernel, or purgatory entry address).
* kern_arg0 first argument to kernel is its dtb address. The other
@@ -113,7 +112,6 @@ extern const char arm64_kexec_el2_vectors[];
* copy_len Number of bytes that need to be copied
*/
struct kern_reloc_arg {
- phys_addr_t head;
phys_addr_t entry_addr;
phys_addr_t kern_arg0;
phys_addr_t kern_arg1;
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 06278611451d..94f050ad6471 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -153,7 +153,6 @@ int main(void)
BLANK();
#endif
#ifdef CONFIG_KEXEC_CORE
- DEFINE(KEXEC_KRELOC_HEAD, offsetof(struct kern_reloc_arg, head));
DEFINE(KEXEC_KRELOC_ENTRY_ADDR, offsetof(struct kern_reloc_arg, entry_addr));
DEFINE(KEXEC_KRELOC_KERN_ARG0, offsetof(struct kern_reloc_arg, kern_arg0));
DEFINE(KEXEC_KRELOC_KERN_ARG1, offsetof(struct kern_reloc_arg, kern_arg1));
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index dc1b7e5a54fb..c2dff232a85b 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -168,7 +168,6 @@ int machine_kexec_post_load(struct kimage *kimage)
memcpy(reloc_code, __relocate_new_kernel_start, reloc_size);
kimage->arch.kern_reloc = __pa(reloc_code) + func_offset;
kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg);
- kern_reloc_arg->head = kimage->head;
kern_reloc_arg->entry_addr = kimage->start;
kern_reloc_arg->kern_arg0 = kimage->arch.dtb_mem;
--
2.25.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 84+ messages in thread
* Re: [PATCH v10 00/18] arm64: MMU enabled kexec relocation
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-26 22:58 ` Will Deacon
-1 siblings, 0 replies; 84+ messages in thread
From: Will Deacon @ 2021-01-26 22:58 UTC (permalink / raw)
To: Pavel Tatashin
Cc: jmorris, sashal, ebiederm, kexec, linux-kernel, corbet,
catalin.marinas, linux-arm-kernel, maz, james.morse,
vladimir.murzin, matthias.bgg, linux-mm, mark.rutland,
steve.capper, rfontana, tglx, selindag, tyhicks
Hi Pavel,
On Mon, Jan 25, 2021 at 02:19:05PM -0500, Pavel Tatashin wrote:
> Changelog:
> v10:
> - Addressed a lot of comments form James Morse and from Marc Zyngier
> - Added review-by's
> - Synchronized with mainline
This series has been doing the rounds for a while, but still isn't fully
reviewed. Would it help if I merged, e.g. the first 12 patches to reduce the
amount you have to repost?
Will
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [PATCH v10 00/18] arm64: MMU enabled kexec relocation
@ 2021-01-26 22:58 ` Will Deacon
0 siblings, 0 replies; 84+ messages in thread
From: Will Deacon @ 2021-01-26 22:58 UTC (permalink / raw)
To: Pavel Tatashin
Cc: sashal, mark.rutland, vladimir.murzin, corbet, catalin.marinas,
selindag, steve.capper, kexec, linux-kernel, jmorris, linux-mm,
tyhicks, james.morse, ebiederm, maz, matthias.bgg, rfontana,
tglx, linux-arm-kernel
Hi Pavel,
On Mon, Jan 25, 2021 at 02:19:05PM -0500, Pavel Tatashin wrote:
> Changelog:
> v10:
> - Addressed a lot of comments form James Morse and from Marc Zyngier
> - Added review-by's
> - Synchronized with mainline
This series has been doing the rounds for a while, but still isn't fully
reviewed. Would it help if I merged, e.g. the first 12 patches to reduce the
amount you have to repost?
Will
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [PATCH v10 00/18] arm64: MMU enabled kexec relocation
@ 2021-01-26 22:58 ` Will Deacon
0 siblings, 0 replies; 84+ messages in thread
From: Will Deacon @ 2021-01-26 22:58 UTC (permalink / raw)
To: Pavel Tatashin
Cc: sashal, mark.rutland, vladimir.murzin, corbet, catalin.marinas,
selindag, steve.capper, kexec, linux-kernel, jmorris, linux-mm,
tyhicks, james.morse, ebiederm, maz, matthias.bgg, rfontana,
tglx, linux-arm-kernel
Hi Pavel,
On Mon, Jan 25, 2021 at 02:19:05PM -0500, Pavel Tatashin wrote:
> Changelog:
> v10:
> - Addressed a lot of comments form James Morse and from Marc Zyngier
> - Added review-by's
> - Synchronized with mainline
This series has been doing the rounds for a while, but still isn't fully
reviewed. Would it help if I merged, e.g. the first 12 patches to reduce the
amount you have to repost?
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [PATCH v10 00/18] arm64: MMU enabled kexec relocation
2021-01-26 22:58 ` Will Deacon
(?)
(?)
@ 2021-01-26 23:05 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-26 23:05 UTC (permalink / raw)
To: Will Deacon
Cc: James Morris, Sasha Levin, Eric W. Biederman, kexec mailing list,
LKML, Jonathan Corbet, Catalin Marinas, Linux ARM, Marc Zyngier,
James Morse, Vladimir Murzin, Matthias Brugger, linux-mm,
Mark Rutland, steve.capper, rfontana, Thomas Gleixner, Selin Dag,
Tyler Hicks
On Tue, Jan 26, 2021 at 5:58 PM Will Deacon <will@kernel.org> wrote:
>
> Hi Pavel,
>
> On Mon, Jan 25, 2021 at 02:19:05PM -0500, Pavel Tatashin wrote:
> > Changelog:
> > v10:
> > - Addressed a lot of comments form James Morse and from Marc Zyngier
> > - Added review-by's
> > - Synchronized with mainline
>
> This series has been doing the rounds for a while, but still isn't fully
> reviewed. Would it help if I merged, e.g. the first 12 patches to reduce the
> amount you have to repost?
Hi Will,
Yes, it would. The first 10 patches of this series were already merged
sometime ago:
a2c2e67923ec arm64: hibernate: add trans_pgd public functions
7ea4088938b7 arm64: hibernate: add PUD_SECT_RDONLY
13373f0e6580 arm64: hibernate: rename dst to page in create_safe_exec_page
a89d7ff933b0 arm64: hibernate: remove gotos as they are not needed
051a7a94aaa9 arm64: hibernate: use get_safe_page directly
d234332c2815 arm64: hibernate: pass the allocated pgdp to ttbr0
621516789ee6 arm64: kexec: make dtb_mem always enabled
3b54b743397e arm64: kexec: remove unnecessary debug prints
de68e4daea90 kexec: add machine_kexec_post_load()
d42cc530b18d kexec: quiet down kexec reboot
So, merging another ~12, would substantially help with getting
everything reviewed and merged.
Thank you,
Pavel
>
> Will
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [PATCH v10 00/18] arm64: MMU enabled kexec relocation
@ 2021-01-26 23:05 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-26 23:05 UTC (permalink / raw)
To: Will Deacon
Cc: Sasha Levin, Mark Rutland, Vladimir Murzin, Jonathan Corbet,
Catalin Marinas, Selin Dag, steve.capper, kexec mailing list,
LKML, James Morris, linux-mm, Tyler Hicks, James Morse,
Eric W. Biederman, Marc Zyngier, Matthias Brugger, rfontana,
Thomas Gleixner, Linux ARM
On Tue, Jan 26, 2021 at 5:58 PM Will Deacon <will@kernel.org> wrote:
>
> Hi Pavel,
>
> On Mon, Jan 25, 2021 at 02:19:05PM -0500, Pavel Tatashin wrote:
> > Changelog:
> > v10:
> > - Addressed a lot of comments form James Morse and from Marc Zyngier
> > - Added review-by's
> > - Synchronized with mainline
>
> This series has been doing the rounds for a while, but still isn't fully
> reviewed. Would it help if I merged, e.g. the first 12 patches to reduce the
> amount you have to repost?
Hi Will,
Yes, it would. The first 10 patches of this series were already merged
sometime ago:
a2c2e67923ec arm64: hibernate: add trans_pgd public functions
7ea4088938b7 arm64: hibernate: add PUD_SECT_RDONLY
13373f0e6580 arm64: hibernate: rename dst to page in create_safe_exec_page
a89d7ff933b0 arm64: hibernate: remove gotos as they are not needed
051a7a94aaa9 arm64: hibernate: use get_safe_page directly
d234332c2815 arm64: hibernate: pass the allocated pgdp to ttbr0
621516789ee6 arm64: kexec: make dtb_mem always enabled
3b54b743397e arm64: kexec: remove unnecessary debug prints
de68e4daea90 kexec: add machine_kexec_post_load()
d42cc530b18d kexec: quiet down kexec reboot
So, merging another ~12, would substantially help with getting
everything reviewed and merged.
Thank you,
Pavel
>
> Will
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [PATCH v10 00/18] arm64: MMU enabled kexec relocation
@ 2021-01-26 23:05 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-26 23:05 UTC (permalink / raw)
To: Will Deacon
Cc: Sasha Levin, Mark Rutland, Vladimir Murzin, Jonathan Corbet,
Catalin Marinas, Selin Dag, steve.capper, kexec mailing list,
LKML, James Morris, linux-mm, Tyler Hicks, James Morse,
Eric W. Biederman, Marc Zyngier, Matthias Brugger, rfontana,
Thomas Gleixner, Linux ARM
On Tue, Jan 26, 2021 at 5:58 PM Will Deacon <will@kernel.org> wrote:
>
> Hi Pavel,
>
> On Mon, Jan 25, 2021 at 02:19:05PM -0500, Pavel Tatashin wrote:
> > Changelog:
> > v10:
> > - Addressed a lot of comments form James Morse and from Marc Zyngier
> > - Added review-by's
> > - Synchronized with mainline
>
> This series has been doing the rounds for a while, but still isn't fully
> reviewed. Would it help if I merged, e.g. the first 12 patches to reduce the
> amount you have to repost?
Hi Will,
Yes, it would. The first 10 patches of this series were already merged
sometime ago:
a2c2e67923ec arm64: hibernate: add trans_pgd public functions
7ea4088938b7 arm64: hibernate: add PUD_SECT_RDONLY
13373f0e6580 arm64: hibernate: rename dst to page in create_safe_exec_page
a89d7ff933b0 arm64: hibernate: remove gotos as they are not needed
051a7a94aaa9 arm64: hibernate: use get_safe_page directly
d234332c2815 arm64: hibernate: pass the allocated pgdp to ttbr0
621516789ee6 arm64: kexec: make dtb_mem always enabled
3b54b743397e arm64: kexec: remove unnecessary debug prints
de68e4daea90 kexec: add machine_kexec_post_load()
d42cc530b18d kexec: quiet down kexec reboot
So, merging another ~12, would substantially help with getting
everything reviewed and merged.
Thank you,
Pavel
>
> Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [PATCH v10 00/18] arm64: MMU enabled kexec relocation
@ 2021-01-26 23:05 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-26 23:05 UTC (permalink / raw)
To: Will Deacon
Cc: James Morris, Sasha Levin, Eric W. Biederman, kexec mailing list,
LKML, Jonathan Corbet, Catalin Marinas, Linux ARM, Marc Zyngier,
James Morse, Vladimir Murzin, Matthias Brugger, linux-mm,
Mark Rutland, steve.capper, rfontana, Thomas Gleixner, Selin Dag,
Tyler Hicks
On Tue, Jan 26, 2021 at 5:58 PM Will Deacon <will@kernel.org> wrote:
>
> Hi Pavel,
>
> On Mon, Jan 25, 2021 at 02:19:05PM -0500, Pavel Tatashin wrote:
> > Changelog:
> > v10:
> > - Addressed a lot of comments form James Morse and from Marc Zyngier
> > - Added review-by's
> > - Synchronized with mainline
>
> This series has been doing the rounds for a while, but still isn't fully
> reviewed. Would it help if I merged, e.g. the first 12 patches to reduce the
> amount you have to repost?
Hi Will,
Yes, it would. The first 10 patches of this series were already merged
sometime ago:
a2c2e67923ec arm64: hibernate: add trans_pgd public functions
7ea4088938b7 arm64: hibernate: add PUD_SECT_RDONLY
13373f0e6580 arm64: hibernate: rename dst to page in create_safe_exec_page
a89d7ff933b0 arm64: hibernate: remove gotos as they are not needed
051a7a94aaa9 arm64: hibernate: use get_safe_page directly
d234332c2815 arm64: hibernate: pass the allocated pgdp to ttbr0
621516789ee6 arm64: kexec: make dtb_mem always enabled
3b54b743397e arm64: kexec: remove unnecessary debug prints
de68e4daea90 kexec: add machine_kexec_post_load()
d42cc530b18d kexec: quiet down kexec reboot
So, merging another ~12, would substantially help with getting
everything reviewed and merged.
Thank you,
Pavel
>
> Will
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [PATCH v10 00/18] arm64: MMU enabled kexec relocation
2021-01-26 23:05 ` Pavel Tatashin
(?)
@ 2021-01-26 23:11 ` Will Deacon
-1 siblings, 0 replies; 84+ messages in thread
From: Will Deacon @ 2021-01-26 23:11 UTC (permalink / raw)
To: Pavel Tatashin
Cc: James Morris, Sasha Levin, Eric W. Biederman, kexec mailing list,
LKML, Jonathan Corbet, Catalin Marinas, Linux ARM, Marc Zyngier,
James Morse, Vladimir Murzin, Matthias Brugger, linux-mm,
Mark Rutland, steve.capper, rfontana, Thomas Gleixner, Selin Dag,
Tyler Hicks
On Tue, Jan 26, 2021 at 06:05:10PM -0500, Pavel Tatashin wrote:
> On Tue, Jan 26, 2021 at 5:58 PM Will Deacon <will@kernel.org> wrote:
> >
> > Hi Pavel,
> >
> > On Mon, Jan 25, 2021 at 02:19:05PM -0500, Pavel Tatashin wrote:
> > > Changelog:
> > > v10:
> > > - Addressed a lot of comments form James Morse and from Marc Zyngier
> > > - Added review-by's
> > > - Synchronized with mainline
> >
> > This series has been doing the rounds for a while, but still isn't fully
> > reviewed. Would it help if I merged, e.g. the first 12 patches to reduce the
> > amount you have to repost?
>
> Hi Will,
>
> Yes, it would. The first 10 patches of this series were already merged
> sometime ago:
>
> a2c2e67923ec arm64: hibernate: add trans_pgd public functions
> 7ea4088938b7 arm64: hibernate: add PUD_SECT_RDONLY
> 13373f0e6580 arm64: hibernate: rename dst to page in create_safe_exec_page
> a89d7ff933b0 arm64: hibernate: remove gotos as they are not needed
> 051a7a94aaa9 arm64: hibernate: use get_safe_page directly
> d234332c2815 arm64: hibernate: pass the allocated pgdp to ttbr0
> 621516789ee6 arm64: kexec: make dtb_mem always enabled
> 3b54b743397e arm64: kexec: remove unnecessary debug prints
> de68e4daea90 kexec: add machine_kexec_post_load()
> d42cc530b18d kexec: quiet down kexec reboot
>
> So, merging another ~12, would substantially help with getting
> everything reviewed and merged.
Ok, great; I'll have a crack at that tomorrow, and thanks for persevering.
Will
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [PATCH v10 00/18] arm64: MMU enabled kexec relocation
@ 2021-01-26 23:11 ` Will Deacon
0 siblings, 0 replies; 84+ messages in thread
From: Will Deacon @ 2021-01-26 23:11 UTC (permalink / raw)
To: Pavel Tatashin
Cc: Sasha Levin, Mark Rutland, Vladimir Murzin, Jonathan Corbet,
Catalin Marinas, Selin Dag, steve.capper, kexec mailing list,
LKML, James Morris, linux-mm, Tyler Hicks, James Morse,
Eric W. Biederman, Marc Zyngier, Matthias Brugger, rfontana,
Thomas Gleixner, Linux ARM
On Tue, Jan 26, 2021 at 06:05:10PM -0500, Pavel Tatashin wrote:
> On Tue, Jan 26, 2021 at 5:58 PM Will Deacon <will@kernel.org> wrote:
> >
> > Hi Pavel,
> >
> > On Mon, Jan 25, 2021 at 02:19:05PM -0500, Pavel Tatashin wrote:
> > > Changelog:
> > > v10:
> > > - Addressed a lot of comments form James Morse and from Marc Zyngier
> > > - Added review-by's
> > > - Synchronized with mainline
> >
> > This series has been doing the rounds for a while, but still isn't fully
> > reviewed. Would it help if I merged, e.g. the first 12 patches to reduce the
> > amount you have to repost?
>
> Hi Will,
>
> Yes, it would. The first 10 patches of this series were already merged
> sometime ago:
>
> a2c2e67923ec arm64: hibernate: add trans_pgd public functions
> 7ea4088938b7 arm64: hibernate: add PUD_SECT_RDONLY
> 13373f0e6580 arm64: hibernate: rename dst to page in create_safe_exec_page
> a89d7ff933b0 arm64: hibernate: remove gotos as they are not needed
> 051a7a94aaa9 arm64: hibernate: use get_safe_page directly
> d234332c2815 arm64: hibernate: pass the allocated pgdp to ttbr0
> 621516789ee6 arm64: kexec: make dtb_mem always enabled
> 3b54b743397e arm64: kexec: remove unnecessary debug prints
> de68e4daea90 kexec: add machine_kexec_post_load()
> d42cc530b18d kexec: quiet down kexec reboot
>
> So, merging another ~12, would substantially help with getting
> everything reviewed and merged.
Ok, great; I'll have a crack at that tomorrow, and thanks for persevering.
Will
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [PATCH v10 00/18] arm64: MMU enabled kexec relocation
@ 2021-01-26 23:11 ` Will Deacon
0 siblings, 0 replies; 84+ messages in thread
From: Will Deacon @ 2021-01-26 23:11 UTC (permalink / raw)
To: Pavel Tatashin
Cc: Sasha Levin, Mark Rutland, Vladimir Murzin, Jonathan Corbet,
Catalin Marinas, Selin Dag, steve.capper, kexec mailing list,
LKML, James Morris, linux-mm, Tyler Hicks, James Morse,
Eric W. Biederman, Marc Zyngier, Matthias Brugger, rfontana,
Thomas Gleixner, Linux ARM
On Tue, Jan 26, 2021 at 06:05:10PM -0500, Pavel Tatashin wrote:
> On Tue, Jan 26, 2021 at 5:58 PM Will Deacon <will@kernel.org> wrote:
> >
> > Hi Pavel,
> >
> > On Mon, Jan 25, 2021 at 02:19:05PM -0500, Pavel Tatashin wrote:
> > > Changelog:
> > > v10:
> > > - Addressed a lot of comments form James Morse and from Marc Zyngier
> > > - Added review-by's
> > > - Synchronized with mainline
> >
> > This series has been doing the rounds for a while, but still isn't fully
> > reviewed. Would it help if I merged, e.g. the first 12 patches to reduce the
> > amount you have to repost?
>
> Hi Will,
>
> Yes, it would. The first 10 patches of this series were already merged
> sometime ago:
>
> a2c2e67923ec arm64: hibernate: add trans_pgd public functions
> 7ea4088938b7 arm64: hibernate: add PUD_SECT_RDONLY
> 13373f0e6580 arm64: hibernate: rename dst to page in create_safe_exec_page
> a89d7ff933b0 arm64: hibernate: remove gotos as they are not needed
> 051a7a94aaa9 arm64: hibernate: use get_safe_page directly
> d234332c2815 arm64: hibernate: pass the allocated pgdp to ttbr0
> 621516789ee6 arm64: kexec: make dtb_mem always enabled
> 3b54b743397e arm64: kexec: remove unnecessary debug prints
> de68e4daea90 kexec: add machine_kexec_post_load()
> d42cc530b18d kexec: quiet down kexec reboot
>
> So, merging another ~12, would substantially help with getting
> everything reviewed and merged.
Ok, great; I'll have a crack at that tomorrow, and thanks for persevering.
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: (subset) [PATCH v10 00/18] arm64: MMU enabled kexec relocation
2021-01-25 19:19 ` Pavel Tatashin
(?)
@ 2021-01-27 15:59 ` Will Deacon
-1 siblings, 0 replies; 84+ messages in thread
From: Will Deacon @ 2021-01-27 15:59 UTC (permalink / raw)
To: jmorris, maz, linux-arm-kernel, tglx, matthias.bgg, james.morse,
selindag, mark.rutland, Pavel Tatashin, rfontana, sashal,
steve.capper, tyhicks, kexec, ebiederm, vladimir.murzin,
catalin.marinas, linux-kernel, corbet, linux-mm
Cc: kernel-team, Will Deacon
On Mon, 25 Jan 2021 14:19:05 -0500, Pavel Tatashin wrote:
> Changelog:
> v10:
> - Addressed a lot of comments form James Morse and from Marc Zyngier
> - Added review-by's
> - Synchronized with mainline
>
> [...]
Applied the first 12 patches to arm64 (for-next/kexec), thanks!
[01/18] arm64: kexec: make dtb_mem always enabled
https://git.kernel.org/arm64/c/117cda9a7847
[02/18] arm64: hibernate: variable pudp is used instead of pd4dp
https://git.kernel.org/arm64/c/41f67d40a31d
[03/18] arm64: hibernate: move page handling function to new trans_pgd.c
https://git.kernel.org/arm64/c/072e3d96a79a
[04/18] arm64: trans_pgd: make trans_pgd_map_page generic
https://git.kernel.org/arm64/c/50f53fb72181
[05/18] arm64: trans_pgd: pass allocator trans_pgd_create_copy
https://git.kernel.org/arm64/c/89d1410f4af5
[06/18] arm64: trans_pgd: pass NULL instead of init_mm to *_populate functions
https://git.kernel.org/arm64/c/5de59884ac0e
[07/18] arm64: mm: Always update TCR_EL1 from __cpu_set_tcr_t0sz()
https://git.kernel.org/arm64/c/1401bef703a4
[08/18] arm64: trans_pgd: hibernate: idmap the single page that holds the copy page routines
https://git.kernel.org/arm64/c/7018d467ff2d
[09/18] arm64: kexec: move relocation function setup
https://git.kernel.org/arm64/c/4c3c31230c91
[10/18] arm64: kexec: call kexec_image_info only once
https://git.kernel.org/arm64/c/77a43be11648
[11/18] arm64: kexec: arm64_relocate_new_kernel clean-ups and optimizations
https://git.kernel.org/arm64/c/dbd82fee0f25
[12/18] arm64: kexec: arm64_relocate_new_kernel don't use x0 as temp
https://git.kernel.org/arm64/c/a360190e8a42
Cheers,
--
Will
https://fixes.arm64.dev
https://next.arm64.dev
https://will.arm64.dev
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: (subset) [PATCH v10 00/18] arm64: MMU enabled kexec relocation
@ 2021-01-27 15:59 ` Will Deacon
0 siblings, 0 replies; 84+ messages in thread
From: Will Deacon @ 2021-01-27 15:59 UTC (permalink / raw)
To: jmorris, maz, linux-arm-kernel, tglx, matthias.bgg, james.morse,
selindag, mark.rutland, Pavel Tatashin, rfontana, sashal,
steve.capper, tyhicks, kexec, ebiederm, vladimir.murzin,
catalin.marinas, linux-kernel, corbet, linux-mm
Cc: Will Deacon, kernel-team
On Mon, 25 Jan 2021 14:19:05 -0500, Pavel Tatashin wrote:
> Changelog:
> v10:
> - Addressed a lot of comments form James Morse and from Marc Zyngier
> - Added review-by's
> - Synchronized with mainline
>
> [...]
Applied the first 12 patches to arm64 (for-next/kexec), thanks!
[01/18] arm64: kexec: make dtb_mem always enabled
https://git.kernel.org/arm64/c/117cda9a7847
[02/18] arm64: hibernate: variable pudp is used instead of pd4dp
https://git.kernel.org/arm64/c/41f67d40a31d
[03/18] arm64: hibernate: move page handling function to new trans_pgd.c
https://git.kernel.org/arm64/c/072e3d96a79a
[04/18] arm64: trans_pgd: make trans_pgd_map_page generic
https://git.kernel.org/arm64/c/50f53fb72181
[05/18] arm64: trans_pgd: pass allocator trans_pgd_create_copy
https://git.kernel.org/arm64/c/89d1410f4af5
[06/18] arm64: trans_pgd: pass NULL instead of init_mm to *_populate functions
https://git.kernel.org/arm64/c/5de59884ac0e
[07/18] arm64: mm: Always update TCR_EL1 from __cpu_set_tcr_t0sz()
https://git.kernel.org/arm64/c/1401bef703a4
[08/18] arm64: trans_pgd: hibernate: idmap the single page that holds the copy page routines
https://git.kernel.org/arm64/c/7018d467ff2d
[09/18] arm64: kexec: move relocation function setup
https://git.kernel.org/arm64/c/4c3c31230c91
[10/18] arm64: kexec: call kexec_image_info only once
https://git.kernel.org/arm64/c/77a43be11648
[11/18] arm64: kexec: arm64_relocate_new_kernel clean-ups and optimizations
https://git.kernel.org/arm64/c/dbd82fee0f25
[12/18] arm64: kexec: arm64_relocate_new_kernel don't use x0 as temp
https://git.kernel.org/arm64/c/a360190e8a42
Cheers,
--
Will
https://fixes.arm64.dev
https://next.arm64.dev
https://will.arm64.dev
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: (subset) [PATCH v10 00/18] arm64: MMU enabled kexec relocation
@ 2021-01-27 15:59 ` Will Deacon
0 siblings, 0 replies; 84+ messages in thread
From: Will Deacon @ 2021-01-27 15:59 UTC (permalink / raw)
To: jmorris, maz, linux-arm-kernel, tglx, matthias.bgg, james.morse,
selindag, mark.rutland, Pavel Tatashin, rfontana, sashal,
steve.capper, tyhicks, kexec, ebiederm, vladimir.murzin,
catalin.marinas, linux-kernel, corbet, linux-mm
Cc: Will Deacon, kernel-team
On Mon, 25 Jan 2021 14:19:05 -0500, Pavel Tatashin wrote:
> Changelog:
> v10:
> - Addressed a lot of comments form James Morse and from Marc Zyngier
> - Added review-by's
> - Synchronized with mainline
>
> [...]
Applied the first 12 patches to arm64 (for-next/kexec), thanks!
[01/18] arm64: kexec: make dtb_mem always enabled
https://git.kernel.org/arm64/c/117cda9a7847
[02/18] arm64: hibernate: variable pudp is used instead of pd4dp
https://git.kernel.org/arm64/c/41f67d40a31d
[03/18] arm64: hibernate: move page handling function to new trans_pgd.c
https://git.kernel.org/arm64/c/072e3d96a79a
[04/18] arm64: trans_pgd: make trans_pgd_map_page generic
https://git.kernel.org/arm64/c/50f53fb72181
[05/18] arm64: trans_pgd: pass allocator trans_pgd_create_copy
https://git.kernel.org/arm64/c/89d1410f4af5
[06/18] arm64: trans_pgd: pass NULL instead of init_mm to *_populate functions
https://git.kernel.org/arm64/c/5de59884ac0e
[07/18] arm64: mm: Always update TCR_EL1 from __cpu_set_tcr_t0sz()
https://git.kernel.org/arm64/c/1401bef703a4
[08/18] arm64: trans_pgd: hibernate: idmap the single page that holds the copy page routines
https://git.kernel.org/arm64/c/7018d467ff2d
[09/18] arm64: kexec: move relocation function setup
https://git.kernel.org/arm64/c/4c3c31230c91
[10/18] arm64: kexec: call kexec_image_info only once
https://git.kernel.org/arm64/c/77a43be11648
[11/18] arm64: kexec: arm64_relocate_new_kernel clean-ups and optimizations
https://git.kernel.org/arm64/c/dbd82fee0f25
[12/18] arm64: kexec: arm64_relocate_new_kernel don't use x0 as temp
https://git.kernel.org/arm64/c/a360190e8a42
Cheers,
--
Will
https://fixes.arm64.dev
https://next.arm64.dev
https://will.arm64.dev
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: (subset) [PATCH v10 00/18] arm64: MMU enabled kexec relocation
2021-01-27 15:59 ` Will Deacon
(?)
(?)
@ 2021-01-27 16:12 ` Pavel Tatashin
-1 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-27 16:12 UTC (permalink / raw)
To: Will Deacon
Cc: James Morris, Marc Zyngier, Linux ARM, Thomas Gleixner,
Matthias Brugger, James Morse, Selin Dag, Mark Rutland, rfontana,
Sasha Levin, steve.capper, Tyler Hicks, kexec mailing list,
Eric W. Biederman, Vladimir Murzin, Catalin Marinas, LKML,
Jonathan Corbet, linux-mm, kernel-team
On Wed, Jan 27, 2021 at 10:59 AM Will Deacon <will@kernel.org> wrote:
>
> On Mon, 25 Jan 2021 14:19:05 -0500, Pavel Tatashin wrote:
> > Changelog:
> > v10:
> > - Addressed a lot of comments form James Morse and from Marc Zyngier
> > - Added review-by's
> > - Synchronized with mainline
> >
> > [...]
>
> Applied the first 12 patches to arm64 (for-next/kexec), thanks!
Great, thank you. I will resend the rest of the series based on your gate.
Pasha
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: (subset) [PATCH v10 00/18] arm64: MMU enabled kexec relocation
@ 2021-01-27 16:12 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-27 16:12 UTC (permalink / raw)
To: Will Deacon
Cc: Mark Rutland, Sasha Levin, Vladimir Murzin, steve.capper,
Marc Zyngier, Selin Dag, Jonathan Corbet, kexec mailing list,
James Morris, LKML, linux-mm, rfontana, Tyler Hicks, James Morse,
Eric W. Biederman, Catalin Marinas, Matthias Brugger,
Thomas Gleixner, kernel-team, Linux ARM
On Wed, Jan 27, 2021 at 10:59 AM Will Deacon <will@kernel.org> wrote:
>
> On Mon, 25 Jan 2021 14:19:05 -0500, Pavel Tatashin wrote:
> > Changelog:
> > v10:
> > - Addressed a lot of comments form James Morse and from Marc Zyngier
> > - Added review-by's
> > - Synchronized with mainline
> >
> > [...]
>
> Applied the first 12 patches to arm64 (for-next/kexec), thanks!
Great, thank you. I will resend the rest of the series based on your gate.
Pasha
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: (subset) [PATCH v10 00/18] arm64: MMU enabled kexec relocation
@ 2021-01-27 16:12 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-27 16:12 UTC (permalink / raw)
To: Will Deacon
Cc: Mark Rutland, Sasha Levin, Vladimir Murzin, steve.capper,
Marc Zyngier, Selin Dag, Jonathan Corbet, kexec mailing list,
James Morris, LKML, linux-mm, rfontana, Tyler Hicks, James Morse,
Eric W. Biederman, Catalin Marinas, Matthias Brugger,
Thomas Gleixner, kernel-team, Linux ARM
On Wed, Jan 27, 2021 at 10:59 AM Will Deacon <will@kernel.org> wrote:
>
> On Mon, 25 Jan 2021 14:19:05 -0500, Pavel Tatashin wrote:
> > Changelog:
> > v10:
> > - Addressed a lot of comments form James Morse and from Marc Zyngier
> > - Added review-by's
> > - Synchronized with mainline
> >
> > [...]
>
> Applied the first 12 patches to arm64 (for-next/kexec), thanks!
Great, thank you. I will resend the rest of the series based on your gate.
Pasha
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: (subset) [PATCH v10 00/18] arm64: MMU enabled kexec relocation
@ 2021-01-27 16:12 ` Pavel Tatashin
0 siblings, 0 replies; 84+ messages in thread
From: Pavel Tatashin @ 2021-01-27 16:12 UTC (permalink / raw)
To: Will Deacon
Cc: James Morris, Marc Zyngier, Linux ARM, Thomas Gleixner,
Matthias Brugger, James Morse, Selin Dag, Mark Rutland, rfontana,
Sasha Levin, steve.capper, Tyler Hicks, kexec mailing list,
Eric W. Biederman, Vladimir Murzin, Catalin Marinas, LKML,
Jonathan Corbet, linux-mm, kernel-team
On Wed, Jan 27, 2021 at 10:59 AM Will Deacon <will@kernel.org> wrote:
>
> On Mon, 25 Jan 2021 14:19:05 -0500, Pavel Tatashin wrote:
> > Changelog:
> > v10:
> > - Addressed a lot of comments form James Morse and from Marc Zyngier
> > - Added review-by's
> > - Synchronized with mainline
> >
> > [...]
>
> Applied the first 12 patches to arm64 (for-next/kexec), thanks!
Great, thank you. I will resend the rest of the series based on your gate.
Pasha
^ permalink raw reply [flat|nested] 84+ messages in thread