* [PATCH] x86/mm: Pre-allocate p4d/pud pages for vmalloc area
@ 2020-06-26 9:34 Joerg Roedel
2020-06-26 11:07 ` Peter Zijlstra
0 siblings, 1 reply; 6+ messages in thread
From: Joerg Roedel @ 2020-06-26 9:34 UTC (permalink / raw)
To: x86
Cc: hpa, Dave Hansen, Andy Lutomirski, Peter Zijlstra, Andrew Morton,
Steven Rostedt, joro, linux-kernel, linux-mm, Joerg Roedel
From: Joerg Roedel <jroedel@suse.de>
Pre-allocate the page-table pages for the vmalloc area at the level
which needs synchronization on x86. This is P4D for 5-level and PUD
for 4-level paging.
Doing this at boot makes sure all page-tables in the system have these
pages already and do not need to be synchronized at runtime. The
runtime synchronizatin takes the pgd_lock and iterates over all
page-tables in the system, so it can take quite long and is better
avoided.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/mm/init_64.c | 55 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 55 insertions(+)
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index dbae185511cd..475a4008445b 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -1238,6 +1238,59 @@ static void __init register_page_bootmem_info(void)
#endif
}
+/*
+ * Pre-allocates page-table pages for the vmalloc area in the kernel page-table.
+ * Only the level which needs to be synchronized between all page-tables is
+ * allocated because the synchronization can be expensive.
+ */
+static void __init preallocate_vmalloc_pages(void)
+{
+ unsigned long addr;
+ const char *lvl;
+ int count = 0;
+
+ for (addr = VMALLOC_START; addr <= VMALLOC_END; addr = ALIGN(addr + 1, PGDIR_SIZE)) {
+ pgd_t *pgd = pgd_offset_k(addr);
+ p4d_t *p4d;
+ pud_t *pud;
+
+ p4d = p4d_offset(pgd, addr);
+ if (p4d_none(*p4d)) {
+ /* Can only happen with 5-level paging */
+ p4d = p4d_alloc(&init_mm, pgd, addr);
+ if (!p4d) {
+ lvl = "p4d";
+ goto failed;
+ }
+ count += 1;
+ }
+
+ if (pgtable_l5_enabled())
+ continue;
+
+ pud = pud_offset(p4d, addr);
+ if (pud_none(*pud)) {
+ /* Ends up here only with 4-level paging */
+ pud = pud_alloc(&init_mm, p4d, addr);
+ if (!pud) {
+ lvl = "pud";
+ goto failed;
+ }
+ count += 1;
+ }
+ }
+
+ return;
+
+failed:
+
+ /*
+ * A failure here is not fatal - If the pages can be allocated later it
+ * will be synchronized to other page-tables.
+ */
+ pr_err("Failed to pre-allocate %s pages for vmalloc area\n", lvl);
+}
+
void __init mem_init(void)
{
pci_iommu_alloc();
@@ -1261,6 +1314,8 @@ void __init mem_init(void)
if (get_gate_vma(&init_mm))
kclist_add(&kcore_vsyscall, (void *)VSYSCALL_ADDR, PAGE_SIZE, KCORE_USER);
+ preallocate_vmalloc_pages();
+
mem_init_print_info(NULL);
}
--
2.27.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] x86/mm: Pre-allocate p4d/pud pages for vmalloc area
2020-06-26 9:34 [PATCH] x86/mm: Pre-allocate p4d/pud pages for vmalloc area Joerg Roedel
@ 2020-06-26 11:07 ` Peter Zijlstra
2020-06-26 11:17 ` Joerg Roedel
0 siblings, 1 reply; 6+ messages in thread
From: Peter Zijlstra @ 2020-06-26 11:07 UTC (permalink / raw)
To: Joerg Roedel
Cc: x86, hpa, Dave Hansen, Andy Lutomirski, Andrew Morton,
Steven Rostedt, linux-kernel, linux-mm, Joerg Roedel
On Fri, Jun 26, 2020 at 11:34:50AM +0200, Joerg Roedel wrote:
> From: Joerg Roedel <jroedel@suse.de>
>
> Pre-allocate the page-table pages for the vmalloc area at the level
> which needs synchronization on x86. This is P4D for 5-level and PUD
> for 4-level paging.
>
> Doing this at boot makes sure all page-tables in the system have these
> pages already and do not need to be synchronized at runtime. The
> runtime synchronizatin takes the pgd_lock and iterates over all
> page-tables in the system, so it can take quite long and is better
> avoided.
>
> Signed-off-by: Joerg Roedel <jroedel@suse.de>
> ---
Can't we now remove arch_sync_kernel_mappings() from this same file?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] x86/mm: Pre-allocate p4d/pud pages for vmalloc area
2020-06-26 11:07 ` Peter Zijlstra
@ 2020-06-26 11:17 ` Joerg Roedel
2020-06-26 11:31 ` Peter Zijlstra
0 siblings, 1 reply; 6+ messages in thread
From: Joerg Roedel @ 2020-06-26 11:17 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Joerg Roedel, x86, hpa, Dave Hansen, Andy Lutomirski,
Andrew Morton, Steven Rostedt, linux-kernel, linux-mm
On Fri, Jun 26, 2020 at 01:07:31PM +0200, Peter Zijlstra wrote:
> Can't we now remove arch_sync_kernel_mappings() from this same file?
Only if we panic on allocation failure.
Joerg
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] x86/mm: Pre-allocate p4d/pud pages for vmalloc area
2020-06-26 11:17 ` Joerg Roedel
@ 2020-06-26 11:31 ` Peter Zijlstra
2020-06-26 11:32 ` Peter Zijlstra
0 siblings, 1 reply; 6+ messages in thread
From: Peter Zijlstra @ 2020-06-26 11:31 UTC (permalink / raw)
To: Joerg Roedel
Cc: Joerg Roedel, x86, hpa, Dave Hansen, Andy Lutomirski,
Andrew Morton, Steven Rostedt, linux-kernel, linux-mm
On Fri, Jun 26, 2020 at 01:17:11PM +0200, Joerg Roedel wrote:
> On Fri, Jun 26, 2020 at 01:07:31PM +0200, Peter Zijlstra wrote:
> > Can't we now remove arch_sync_kernel_mappings() from this same file?
>
> Only if we panic on allocation failure.
I think we do that in plenty places already, so sure ;-)
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] x86/mm: Pre-allocate p4d/pud pages for vmalloc area
2020-06-26 11:31 ` Peter Zijlstra
@ 2020-06-26 11:32 ` Peter Zijlstra
2020-06-26 11:44 ` Joerg Roedel
0 siblings, 1 reply; 6+ messages in thread
From: Peter Zijlstra @ 2020-06-26 11:32 UTC (permalink / raw)
To: Joerg Roedel
Cc: Joerg Roedel, x86, hpa, Dave Hansen, Andy Lutomirski,
Andrew Morton, Steven Rostedt, linux-kernel, linux-mm
On Fri, Jun 26, 2020 at 01:31:01PM +0200, Peter Zijlstra wrote:
> On Fri, Jun 26, 2020 at 01:17:11PM +0200, Joerg Roedel wrote:
> > On Fri, Jun 26, 2020 at 01:07:31PM +0200, Peter Zijlstra wrote:
> > > Can't we now remove arch_sync_kernel_mappings() from this same file?
> >
> > Only if we panic on allocation failure.
>
> I think we do that in plenty places already, so sure ;-)
That is, this is boot time only, right? clone() would return -ENOMEM, as
it's part of the normal page-table copy.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] x86/mm: Pre-allocate p4d/pud pages for vmalloc area
2020-06-26 11:32 ` Peter Zijlstra
@ 2020-06-26 11:44 ` Joerg Roedel
0 siblings, 0 replies; 6+ messages in thread
From: Joerg Roedel @ 2020-06-26 11:44 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Joerg Roedel, x86, hpa, Dave Hansen, Andy Lutomirski,
Andrew Morton, Steven Rostedt, linux-kernel, linux-mm
On Fri, Jun 26, 2020 at 01:32:15PM +0200, Peter Zijlstra wrote:
> That is, this is boot time only, right? clone() would return -ENOMEM, as
> it's part of the normal page-table copy.
Yes, the pre-allocation happens shortly after the buddy allocator took
over from bootmem. I don't quite get what clone() has to do with it.
Joerg
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-06-26 11:44 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-26 9:34 [PATCH] x86/mm: Pre-allocate p4d/pud pages for vmalloc area Joerg Roedel
2020-06-26 11:07 ` Peter Zijlstra
2020-06-26 11:17 ` Joerg Roedel
2020-06-26 11:31 ` Peter Zijlstra
2020-06-26 11:32 ` Peter Zijlstra
2020-06-26 11:44 ` Joerg Roedel
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).