From: Nadav Amit <namit@vmware.com> To: Peter Zijlstra <peterz@infradead.org>, Borislav Petkov <bp@alien8.de>, Andy Lutomirski <luto@kernel.org>, Ingo Molnar <mingo@redhat.com> Cc: <linux-kernel@vger.kernel.org>, <x86@kernel.org>, <hpa@zytor.com>, Thomas Gleixner <tglx@linutronix.de>, Nadav Amit <nadav.amit@gmail.com>, Dave Hansen <dave.hansen@linux.intel.com>, <linux_dti@icloud.com>, <linux-integrity@vger.kernel.org>, <linux-security-module@vger.kernel.org>, <akpm@linux-foundation.org>, <kernel-hardening@lists.openwall.com>, <linux-mm@kvack.org>, <will.deacon@arm.com>, <ard.biesheuvel@linaro.org>, <kristen@linux.intel.com>, <deneen.t.dock@intel.com>, Rick Edgecombe <rick.p.edgecombe@intel.com>, "Rafael J. Wysocki" <rjw@rjwysocki.net>, Pavel Machek <pavel@ucw.cz> Subject: [PATCH v5 15/23] mm: Make hibernate handle unmapped pages Date: Thu, 25 Apr 2019 17:11:35 -0700 [thread overview] Message-ID: <20190426001143.4983-16-namit@vmware.com> (raw) In-Reply-To: <20190426001143.4983-1-namit@vmware.com> From: Rick Edgecombe <rick.p.edgecombe@intel.com> Make hibernate handle unmapped pages on the direct map when CONFIG_ARCH_HAS_SET_ALIAS is set. These functions allow for setting pages to invalid configurations, so now hibernate should check if the pages have valid mappings and handle if they are unmapped when doing a hibernate save operation. Previously this checking was already done when CONFIG_DEBUG_PAGEALLOC was configured. It does not appear to have a big hibernating performance impact. The speed of the saving operation before this change was measured as 819.02 MB/s, and after was measured at 813.32 MB/s. Before: [ 4.670938] PM: Wrote 171996 kbytes in 0.21 seconds (819.02 MB/s) After: [ 4.504714] PM: Wrote 178932 kbytes in 0.22 seconds (813.32 MB/s) Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net> Cc: Pavel Machek <pavel@ucw.cz> Cc: Borislav Petkov <bp@alien8.de> Acked-by: Pavel Machek <pavel@ucw.cz> Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com> --- arch/x86/mm/pageattr.c | 4 ---- include/linux/mm.h | 18 ++++++------------ kernel/power/snapshot.c | 5 +++-- mm/page_alloc.c | 7 +++++-- 4 files changed, 14 insertions(+), 20 deletions(-) diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c index 3574550192c6..daf4d645e537 100644 --- a/arch/x86/mm/pageattr.c +++ b/arch/x86/mm/pageattr.c @@ -2257,7 +2257,6 @@ int set_direct_map_default_noflush(struct page *page) return __set_pages_p(page, 1); } -#ifdef CONFIG_DEBUG_PAGEALLOC void __kernel_map_pages(struct page *page, int numpages, int enable) { if (PageHighMem(page)) @@ -2302,11 +2301,8 @@ bool kernel_page_present(struct page *page) pte = lookup_address((unsigned long)page_address(page), &level); return (pte_val(*pte) & _PAGE_PRESENT); } - #endif /* CONFIG_HIBERNATION */ -#endif /* CONFIG_DEBUG_PAGEALLOC */ - int __init kernel_map_pages_in_pgd(pgd_t *pgd, u64 pfn, unsigned long address, unsigned numpages, unsigned long page_flags) { diff --git a/include/linux/mm.h b/include/linux/mm.h index 6b10c21630f5..083d7b4863ed 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2610,37 +2610,31 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { } #endif -#ifdef CONFIG_DEBUG_PAGEALLOC extern bool _debug_pagealloc_enabled; -extern void __kernel_map_pages(struct page *page, int numpages, int enable); static inline bool debug_pagealloc_enabled(void) { - return _debug_pagealloc_enabled; + return IS_ENABLED(CONFIG_DEBUG_PAGEALLOC) && _debug_pagealloc_enabled; } +#if defined(CONFIG_DEBUG_PAGEALLOC) || defined(CONFIG_ARCH_HAS_SET_DIRECT_MAP) +extern void __kernel_map_pages(struct page *page, int numpages, int enable); + static inline void kernel_map_pages(struct page *page, int numpages, int enable) { - if (!debug_pagealloc_enabled()) - return; - __kernel_map_pages(page, numpages, enable); } #ifdef CONFIG_HIBERNATION extern bool kernel_page_present(struct page *page); #endif /* CONFIG_HIBERNATION */ -#else /* CONFIG_DEBUG_PAGEALLOC */ +#else /* CONFIG_DEBUG_PAGEALLOC || CONFIG_ARCH_HAS_SET_DIRECT_MAP */ static inline void kernel_map_pages(struct page *page, int numpages, int enable) {} #ifdef CONFIG_HIBERNATION static inline bool kernel_page_present(struct page *page) { return true; } #endif /* CONFIG_HIBERNATION */ -static inline bool debug_pagealloc_enabled(void) -{ - return false; -} -#endif /* CONFIG_DEBUG_PAGEALLOC */ +#endif /* CONFIG_DEBUG_PAGEALLOC || CONFIG_ARCH_HAS_SET_DIRECT_MAP */ #ifdef __HAVE_ARCH_GATE_AREA extern struct vm_area_struct *get_gate_vma(struct mm_struct *mm); diff --git a/kernel/power/snapshot.c b/kernel/power/snapshot.c index f08a1e4ee1d4..bc9558ab1e5b 100644 --- a/kernel/power/snapshot.c +++ b/kernel/power/snapshot.c @@ -1342,8 +1342,9 @@ static inline void do_copy_page(long *dst, long *src) * safe_copy_page - Copy a page in a safe way. * * Check if the page we are going to copy is marked as present in the kernel - * page tables (this always is the case if CONFIG_DEBUG_PAGEALLOC is not set - * and in that case kernel_page_present() always returns 'true'). + * page tables. This always is the case if CONFIG_DEBUG_PAGEALLOC or + * CONFIG_ARCH_HAS_SET_DIRECT_MAP is not set. In that case kernel_page_present() + * always returns 'true'. */ static void safe_copy_page(void *dst, struct page *s_page) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d96ca5bc555b..34a70681a4af 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1131,7 +1131,9 @@ static __always_inline bool free_pages_prepare(struct page *page, } arch_free_page(page, order); kernel_poison_pages(page, 1 << order, 0); - kernel_map_pages(page, 1 << order, 0); + if (debug_pagealloc_enabled()) + kernel_map_pages(page, 1 << order, 0); + kasan_free_nondeferred_pages(page, order); return true; @@ -2001,7 +2003,8 @@ inline void post_alloc_hook(struct page *page, unsigned int order, set_page_refcounted(page); arch_alloc_page(page, order); - kernel_map_pages(page, 1 << order, 1); + if (debug_pagealloc_enabled()) + kernel_map_pages(page, 1 << order, 1); kasan_alloc_pages(page, order); kernel_poison_pages(page, 1 << order, 1); set_page_owner(page, order, gfp_flags); -- 2.17.1
WARNING: multiple messages have this Message-ID (diff)
From: Nadav Amit <namit@vmware.com> To: Peter Zijlstra <peterz@infradead.org>, Borislav Petkov <bp@alien8.de>, Andy Lutomirski <luto@kernel.org>, Ingo Molnar <mingo@redhat.com> Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner <tglx@linutronix.de>, Nadav Amit <nadav.amit@gmail.com>, Dave Hansen <dave.hansen@linux.intel.com>, linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe <rick.p.edgecombe@intel.com>, "Rafael J. Wysocki" <rjw@rjwysocki.net>, Pavel Machek <pavel@ucw.cz> Subject: [PATCH v5 15/23] mm: Make hibernate handle unmapped pages Date: Thu, 25 Apr 2019 17:11:35 -0700 [thread overview] Message-ID: <20190426001143.4983-16-namit@vmware.com> (raw) In-Reply-To: <20190426001143.4983-1-namit@vmware.com> From: Rick Edgecombe <rick.p.edgecombe@intel.com> Make hibernate handle unmapped pages on the direct map when CONFIG_ARCH_HAS_SET_ALIAS is set. These functions allow for setting pages to invalid configurations, so now hibernate should check if the pages have valid mappings and handle if they are unmapped when doing a hibernate save operation. Previously this checking was already done when CONFIG_DEBUG_PAGEALLOC was configured. It does not appear to have a big hibernating performance impact. The speed of the saving operation before this change was measured as 819.02 MB/s, and after was measured at 813.32 MB/s. Before: [ 4.670938] PM: Wrote 171996 kbytes in 0.21 seconds (819.02 MB/s) After: [ 4.504714] PM: Wrote 178932 kbytes in 0.22 seconds (813.32 MB/s) Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net> Cc: Pavel Machek <pavel@ucw.cz> Cc: Borislav Petkov <bp@alien8.de> Acked-by: Pavel Machek <pavel@ucw.cz> Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com> --- arch/x86/mm/pageattr.c | 4 ---- include/linux/mm.h | 18 ++++++------------ kernel/power/snapshot.c | 5 +++-- mm/page_alloc.c | 7 +++++-- 4 files changed, 14 insertions(+), 20 deletions(-) diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c index 3574550192c6..daf4d645e537 100644 --- a/arch/x86/mm/pageattr.c +++ b/arch/x86/mm/pageattr.c @@ -2257,7 +2257,6 @@ int set_direct_map_default_noflush(struct page *page) return __set_pages_p(page, 1); } -#ifdef CONFIG_DEBUG_PAGEALLOC void __kernel_map_pages(struct page *page, int numpages, int enable) { if (PageHighMem(page)) @@ -2302,11 +2301,8 @@ bool kernel_page_present(struct page *page) pte = lookup_address((unsigned long)page_address(page), &level); return (pte_val(*pte) & _PAGE_PRESENT); } - #endif /* CONFIG_HIBERNATION */ -#endif /* CONFIG_DEBUG_PAGEALLOC */ - int __init kernel_map_pages_in_pgd(pgd_t *pgd, u64 pfn, unsigned long address, unsigned numpages, unsigned long page_flags) { diff --git a/include/linux/mm.h b/include/linux/mm.h index 6b10c21630f5..083d7b4863ed 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2610,37 +2610,31 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { } #endif -#ifdef CONFIG_DEBUG_PAGEALLOC extern bool _debug_pagealloc_enabled; -extern void __kernel_map_pages(struct page *page, int numpages, int enable); static inline bool debug_pagealloc_enabled(void) { - return _debug_pagealloc_enabled; + return IS_ENABLED(CONFIG_DEBUG_PAGEALLOC) && _debug_pagealloc_enabled; } +#if defined(CONFIG_DEBUG_PAGEALLOC) || defined(CONFIG_ARCH_HAS_SET_DIRECT_MAP) +extern void __kernel_map_pages(struct page *page, int numpages, int enable); + static inline void kernel_map_pages(struct page *page, int numpages, int enable) { - if (!debug_pagealloc_enabled()) - return; - __kernel_map_pages(page, numpages, enable); } #ifdef CONFIG_HIBERNATION extern bool kernel_page_present(struct page *page); #endif /* CONFIG_HIBERNATION */ -#else /* CONFIG_DEBUG_PAGEALLOC */ +#else /* CONFIG_DEBUG_PAGEALLOC || CONFIG_ARCH_HAS_SET_DIRECT_MAP */ static inline void kernel_map_pages(struct page *page, int numpages, int enable) {} #ifdef CONFIG_HIBERNATION static inline bool kernel_page_present(struct page *page) { return true; } #endif /* CONFIG_HIBERNATION */ -static inline bool debug_pagealloc_enabled(void) -{ - return false; -} -#endif /* CONFIG_DEBUG_PAGEALLOC */ +#endif /* CONFIG_DEBUG_PAGEALLOC || CONFIG_ARCH_HAS_SET_DIRECT_MAP */ #ifdef __HAVE_ARCH_GATE_AREA extern struct vm_area_struct *get_gate_vma(struct mm_struct *mm); diff --git a/kernel/power/snapshot.c b/kernel/power/snapshot.c index f08a1e4ee1d4..bc9558ab1e5b 100644 --- a/kernel/power/snapshot.c +++ b/kernel/power/snapshot.c @@ -1342,8 +1342,9 @@ static inline void do_copy_page(long *dst, long *src) * safe_copy_page - Copy a page in a safe way. * * Check if the page we are going to copy is marked as present in the kernel - * page tables (this always is the case if CONFIG_DEBUG_PAGEALLOC is not set - * and in that case kernel_page_present() always returns 'true'). + * page tables. This always is the case if CONFIG_DEBUG_PAGEALLOC or + * CONFIG_ARCH_HAS_SET_DIRECT_MAP is not set. In that case kernel_page_present() + * always returns 'true'. */ static void safe_copy_page(void *dst, struct page *s_page) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d96ca5bc555b..34a70681a4af 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1131,7 +1131,9 @@ static __always_inline bool free_pages_prepare(struct page *page, } arch_free_page(page, order); kernel_poison_pages(page, 1 << order, 0); - kernel_map_pages(page, 1 << order, 0); + if (debug_pagealloc_enabled()) + kernel_map_pages(page, 1 << order, 0); + kasan_free_nondeferred_pages(page, order); return true; @@ -2001,7 +2003,8 @@ inline void post_alloc_hook(struct page *page, unsigned int order, set_page_refcounted(page); arch_alloc_page(page, order); - kernel_map_pages(page, 1 << order, 1); + if (debug_pagealloc_enabled()) + kernel_map_pages(page, 1 << order, 1); kasan_alloc_pages(page, order); kernel_poison_pages(page, 1 << order, 1); set_page_owner(page, order, gfp_flags); -- 2.17.1
next prev parent reply other threads:[~2019-04-26 7:33 UTC|newest] Thread overview: 74+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-04-26 0:11 [PATCH v5 00/23] x86: text_poke() fixes and executable lockdowns Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-26 0:11 ` [PATCH v5 01/23] Fix "x86/alternatives: Lockdep-enforce text_mutex in text_poke*()" Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:13 ` [tip:x86/mm] x86/alternatives: Add text_poke_kgdb() to not assert the lock when debugging tip-bot for Nadav Amit 2019-04-26 0:11 ` [PATCH v5 02/23] x86/jump_label: Use text_poke_early() during early init Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:15 ` [tip:x86/mm] " tip-bot for Nadav Amit 2019-04-26 0:11 ` [PATCH v5 03/23] x86/mm: Introduce temporary mm structs Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:16 ` [tip:x86/mm] " tip-bot for Andy Lutomirski 2019-04-26 0:11 ` [PATCH v5 04/23] x86/mm: Save debug registers when loading a temporary mm Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:17 ` [tip:x86/mm] " tip-bot for Nadav Amit 2019-04-26 0:11 ` [PATCH v5 05/23] fork: Provide a function for copying init_mm Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:18 ` [tip:x86/mm] " tip-bot for Nadav Amit 2019-04-26 0:11 ` [PATCH v5 06/23] x86/alternative: Initialize temporary mm for patching Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-26 0:11 ` [PATCH v5 07/23] x86/alternative: Use temporary mm for text poking Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:20 ` [tip:x86/mm] x86/alternatives: " tip-bot for Nadav Amit 2019-04-26 0:11 ` [PATCH v5 08/23] x86/kgdb: Avoid redundant comparison of patched code Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:20 ` [tip:x86/mm] " tip-bot for Nadav Amit 2019-04-26 0:11 ` [PATCH v5 09/23] x86/ftrace: Set trampoline pages as executable Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:21 ` [tip:x86/mm] " tip-bot for Nadav Amit 2019-04-26 0:11 ` [PATCH v5 10/23] x86/kprobes: Set instruction page " Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:22 ` [tip:x86/mm] " tip-bot for Nadav Amit 2019-04-26 0:11 ` [PATCH v5 11/23] x86/module: Avoid breaking W^X while loading modules Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:22 ` [tip:x86/mm] x86/modules: " tip-bot for Nadav Amit 2019-04-26 0:11 ` [PATCH v5 12/23] x86/jump-label: Remove support for custom poker Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:23 ` [tip:x86/mm] x86/jump-label: Remove support for custom text poker tip-bot for Nadav Amit 2019-04-26 0:11 ` [PATCH v5 13/23] x86/alternative: Remove the return value of text_poke_*() Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:24 ` [tip:x86/mm] x86/alternatives: " tip-bot for Nadav Amit 2019-04-26 0:11 ` [PATCH v5 14/23] x86/mm/cpa: Add set_direct_map_ functions Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-26 16:40 ` Linus Torvalds 2019-04-26 16:40 ` Linus Torvalds 2019-04-26 16:43 ` Nadav Amit 2019-04-30 11:24 ` [tip:x86/mm] x86/mm/cpa: Add set_direct_map_*() functions tip-bot for Rick Edgecombe 2019-04-26 0:11 ` Nadav Amit [this message] 2019-04-26 0:11 ` [PATCH v5 15/23] mm: Make hibernate handle unmapped pages Nadav Amit 2019-04-30 11:25 ` [tip:x86/mm] mm/hibernation: Make hibernation " tip-bot for Rick Edgecombe 2019-04-26 0:11 ` [PATCH v5 16/23] vmalloc: Add flag for free of special permsissions Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:26 ` [tip:x86/mm] mm/vmalloc: Add flag for freeing " tip-bot for Rick Edgecombe 2019-04-26 0:11 ` [PATCH v5 17/23] modules: Use vmalloc special flag Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:26 ` [tip:x86/mm] " tip-bot for Rick Edgecombe 2019-04-26 0:11 ` [PATCH v5 18/23] bpf: " Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:27 ` [tip:x86/mm] " tip-bot for Rick Edgecombe 2019-04-26 0:11 ` [PATCH v5 19/23] x86/ftrace: " Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:28 ` [tip:x86/mm] " tip-bot for Rick Edgecombe 2019-04-26 0:11 ` [PATCH v5 20/23] x86/kprobes: " Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:28 ` [tip:x86/mm] " tip-bot for Rick Edgecombe 2019-04-26 0:11 ` [PATCH v5 21/23] x86/alternative: Comment about module removal races Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:29 ` [tip:x86/mm] x86/alternatives: Add comment " tip-bot for Nadav Amit 2019-04-26 0:11 ` [PATCH v5 22/23] mm/tlb: Provide default nmi_uaccess_okay() Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:14 ` [tip:x86/mm] " tip-bot for Nadav Amit 2019-04-26 0:11 ` [PATCH v5 23/23] bpf: Fail bpf_probe_write_user() while mm is switched Nadav Amit 2019-04-26 0:11 ` Nadav Amit 2019-04-30 11:15 ` [tip:x86/mm] " tip-bot for Nadav Amit 2019-04-26 12:36 ` [PATCH v5 00/23] x86: text_poke() fixes and executable lockdowns Peter Zijlstra
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20190426001143.4983-16-namit@vmware.com \ --to=namit@vmware.com \ --cc=akpm@linux-foundation.org \ --cc=ard.biesheuvel@linaro.org \ --cc=bp@alien8.de \ --cc=dave.hansen@linux.intel.com \ --cc=deneen.t.dock@intel.com \ --cc=hpa@zytor.com \ --cc=kernel-hardening@lists.openwall.com \ --cc=kristen@linux.intel.com \ --cc=linux-integrity@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=linux-security-module@vger.kernel.org \ --cc=linux_dti@icloud.com \ --cc=luto@kernel.org \ --cc=mingo@redhat.com \ --cc=nadav.amit@gmail.com \ --cc=pavel@ucw.cz \ --cc=peterz@infradead.org \ --cc=rick.p.edgecombe@intel.com \ --cc=rjw@rjwysocki.net \ --cc=tglx@linutronix.de \ --cc=will.deacon@arm.com \ --cc=x86@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.