From: Yu-cheng Yu <yu-cheng.yu@intel.com> To: x86@kernel.org, "H. Peter Anvin" <hpa@zytor.com>, Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@redhat.com>, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann <arnd@arndb.de>, Andy Lutomirski <luto@amacapital.net>, Balbir Singh <bsingharora@gmail.com>, Borislav Petkov <bp@alien8.de>, Cyrill Gorcunov <gorcunov@gmail.com>, Dave Hansen <dave.hansen@linux.intel.com>, Eugene Syromiatnikov <esyr@redhat.com>, Florian Weimer <fweimer@redhat.com>, "H.J. Lu" <hjl.tools@gmail.com>, Jann Horn <jannh@google.com>, Jonathan Corbet <corbet@lwn.net>, Kees Cook <keescook@chromium.org>, Mike Kravetz <mike.kravetz@oracle.com>, Nadav Amit <nadav.amit@gmail.com>, Oleg Nesterov <oleg@redhat.com>, Pavel Machek <pavel@ucw.cz>, Peter Zijlstra <peterz@infradead.org>, Randy Dunlap <rdunlap@infradead.org>, "Ravi V. Shankar" <ravi.v.shankar@intel.com>, Vedvyas Shanbhogue <vedvyas.shanbhogue@intel.com>, Dave Martin <Dave.Martin@arm.com> Cc: Yu-cheng Yu <yu-cheng.yu@intel.com> Subject: [PATCH v8 16/27] mm: Handle THP/HugeTLB shadow stack page fault Date: Tue, 13 Aug 2019 13:52:14 -0700 [thread overview] Message-ID: <20190813205225.12032-17-yu-cheng.yu@intel.com> (raw) In-Reply-To: <20190813205225.12032-1-yu-cheng.yu@intel.com> This patch implements THP shadow stack (SHSTK) copying in the same way as in the previous patch for regular PTE. In copy_huge_pmd(), clear the dirty bit from the PMD to cause a page fault upon the next SHSTK access to the PMD. At that time, fix the PMD and copy/re-use the page. Signed-off-by: Yu-cheng Yu <yu-cheng.yu@intel.com> --- arch/x86/mm/pgtable.c | 8 ++++++++ include/asm-generic/pgtable.h | 6 ++++++ mm/huge_memory.c | 4 ++++ 3 files changed, 18 insertions(+) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 0c10d0c5e329..6f3959ca2a08 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -886,6 +886,14 @@ inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma) return pte; } +inline pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pmd_mkdirty_shstk(pmd); + else + return pmd; +} + inline bool arch_copy_pte_mapping(vm_flags_t vm_flags) { return (vm_flags & VM_SHSTK); diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index 89b0fa132f1f..438ce73b57ea 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1194,12 +1194,18 @@ static inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma) return pte; } +static inline pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma) +{ + return pmd; +} + static inline bool arch_copy_pte_mapping(vm_flags_t vm_flags) { return false; } #else pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma); +pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma); bool arch_copy_pte_mapping(vm_flags_t vm_flags); #endif diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 1334ede667a8..39d66c628121 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -611,6 +611,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, entry = mk_huge_pmd(page, vma->vm_page_prot); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); page_add_new_anon_rmap(page, vma, haddr, true); mem_cgroup_commit_charge(page, memcg, false, true); lru_cache_add_active_or_unevictable(page, vma); @@ -1253,6 +1254,7 @@ static vm_fault_t do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, pte_t entry; entry = mk_pte(pages[i], vma->vm_page_prot); entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = pte_set_vma_features(entry, vma); memcg = (void *)page_private(pages[i]); set_page_private(pages[i], 0); page_add_new_anon_rmap(pages[i], vmf->vma, haddr, false); @@ -1335,6 +1337,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) pmd_t entry; entry = pmd_mkyoung(orig_pmd); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); if (pmdp_set_access_flags(vma, haddr, vmf->pmd, entry, 1)) update_mmu_cache_pmd(vma, vmf->address, vmf->pmd); ret |= VM_FAULT_WRITE; @@ -1407,6 +1410,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) pmd_t entry; entry = mk_huge_pmd(new_page, vma->vm_page_prot); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); pmdp_huge_clear_flush_notify(vma, haddr, vmf->pmd); page_add_new_anon_rmap(new_page, vma, haddr, true); mem_cgroup_commit_charge(new_page, memcg, false, true); -- 2.17.1
WARNING: multiple messages have this Message-ID (diff)
From: Yu-cheng Yu <yu-cheng.yu@intel.com> To: x86@kernel.org, "H. Peter Anvin" <hpa@zytor.com>, Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@redhat.com>, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann <arnd@arndb.de>, Andy Lutomirski <luto@amacapital.net>, Balbir Singh <bsingharora@gmail.com>, Borislav Petkov <bp@alien8.de>, Cyrill Gorcunov <gorcunov@gmail.com>, Dave Hansen <dave.hansen@linux.intel.com>, Eugene Syromiatnikov <esyr@redhat.com>, Florian Weimer <fweimer@redhat.com>, "H.J. Lu" <hjl.tools@gmail.com>, Jann Horn <jannh@google.com>, Jonathan Corbet <corbet@lwn.net>, Kees Cook <keescook@chromium.org>, Mike Kravetz <mike.kravetz@oracle.com>, Nadav Amit <nadav.amit@gmail.com> Cc: Yu-cheng Yu <yu-cheng.yu@intel.com> Subject: [PATCH v8 16/27] mm: Handle THP/HugeTLB shadow stack page fault Date: Tue, 13 Aug 2019 13:52:14 -0700 [thread overview] Message-ID: <20190813205225.12032-17-yu-cheng.yu@intel.com> (raw) In-Reply-To: <20190813205225.12032-1-yu-cheng.yu@intel.com> This patch implements THP shadow stack (SHSTK) copying in the same way as in the previous patch for regular PTE. In copy_huge_pmd(), clear the dirty bit from the PMD to cause a page fault upon the next SHSTK access to the PMD. At that time, fix the PMD and copy/re-use the page. Signed-off-by: Yu-cheng Yu <yu-cheng.yu@intel.com> --- arch/x86/mm/pgtable.c | 8 ++++++++ include/asm-generic/pgtable.h | 6 ++++++ mm/huge_memory.c | 4 ++++ 3 files changed, 18 insertions(+) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 0c10d0c5e329..6f3959ca2a08 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -886,6 +886,14 @@ inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma) return pte; } +inline pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pmd_mkdirty_shstk(pmd); + else + return pmd; +} + inline bool arch_copy_pte_mapping(vm_flags_t vm_flags) { return (vm_flags & VM_SHSTK); diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index 89b0fa132f1f..438ce73b57ea 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1194,12 +1194,18 @@ static inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma) return pte; } +static inline pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma) +{ + return pmd; +} + static inline bool arch_copy_pte_mapping(vm_flags_t vm_flags) { return false; } #else pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma); +pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma); bool arch_copy_pte_mapping(vm_flags_t vm_flags); #endif diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 1334ede667a8..39d66c628121 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -611,6 +611,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, entry = mk_huge_pmd(page, vma->vm_page_prot); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); page_add_new_anon_rmap(page, vma, haddr, true); mem_cgroup_commit_charge(page, memcg, false, true); lru_cache_add_active_or_unevictable(page, vma); @@ -1253,6 +1254,7 @@ static vm_fault_t do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, pte_t entry; entry = mk_pte(pages[i], vma->vm_page_prot); entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = pte_set_vma_features(entry, vma); memcg = (void *)page_private(pages[i]); set_page_private(pages[i], 0); page_add_new_anon_rmap(pages[i], vmf->vma, haddr, false); @@ -1335,6 +1337,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) pmd_t entry; entry = pmd_mkyoung(orig_pmd); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); if (pmdp_set_access_flags(vma, haddr, vmf->pmd, entry, 1)) update_mmu_cache_pmd(vma, vmf->address, vmf->pmd); ret |= VM_FAULT_WRITE; @@ -1407,6 +1410,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) pmd_t entry; entry = mk_huge_pmd(new_page, vma->vm_page_prot); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); pmdp_huge_clear_flush_notify(vma, haddr, vmf->pmd); page_add_new_anon_rmap(new_page, vma, haddr, true); mem_cgroup_commit_charge(new_page, memcg, false, true); -- 2.17.1
next prev parent reply other threads:[~2019-08-13 21:05 UTC|newest] Thread overview: 124+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-08-13 20:51 [PATCH v8 00/27] Control-flow Enforcement: Shadow Stack Yu-cheng Yu 2019-08-13 20:51 ` Yu-cheng Yu 2019-08-13 20:51 ` [PATCH v8 01/27] Documentation/x86: Add CET description Yu-cheng Yu 2019-08-13 20:51 ` Yu-cheng Yu 2019-08-14 8:07 ` Florian Weimer 2019-08-14 8:07 ` Florian Weimer 2019-08-14 8:07 ` Florian Weimer 2019-08-14 15:57 ` Yu-cheng Yu 2019-08-14 15:57 ` Yu-cheng Yu 2019-08-14 15:57 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 02/27] x86/cpufeatures: Add CET CPU feature flags for Control-flow Enforcement Technology (CET) Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-21 10:20 ` Borislav Petkov 2019-08-21 10:20 ` Borislav Petkov 2019-08-21 14:46 ` Yu-cheng Yu 2019-08-21 14:46 ` Yu-cheng Yu 2019-08-21 14:46 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 03/27] x86/fpu/xstate: Change names to separate XSAVES system and user states Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-21 14:45 ` Borislav Petkov 2019-08-21 14:45 ` Borislav Petkov 2019-08-13 20:52 ` [PATCH v8 04/27] x86/fpu/xstate: Introduce XSAVES system states Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-16 19:56 ` Thomas Gleixner 2019-08-16 19:56 ` Thomas Gleixner 2019-08-16 19:56 ` Thomas Gleixner 2019-08-16 19:59 ` Yu-cheng Yu 2019-08-16 19:59 ` Yu-cheng Yu 2019-08-16 19:59 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 05/27] x86/fpu/xstate: Introduce CET MSR " Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 06/27] x86/cet: Add control protection exception handler Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 07/27] x86/cet/shstk: Add Kconfig option for user-mode shadow stack Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 08/27] mm: Introduce VM_SHSTK for shadow stack memory Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 09/27] mm/mmap: Prevent Shadow Stack VMA merges Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 22:34 ` Dave Hansen 2019-08-13 22:34 ` Dave Hansen 2019-08-14 16:20 ` Yu-cheng Yu 2019-08-14 16:20 ` Yu-cheng Yu 2019-08-14 16:20 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 10/27] x86/mm: Change _PAGE_DIRTY to _PAGE_DIRTY_HW Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 11/27] x86/mm: Introduce _PAGE_DIRTY_SW Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 23:02 ` Dave Hansen 2019-08-13 23:02 ` Dave Hansen 2019-08-13 23:49 ` Andy Lutomirski 2019-08-13 23:49 ` Andy Lutomirski 2019-08-14 16:42 ` Yu-cheng Yu 2019-08-14 16:42 ` Yu-cheng Yu 2019-08-14 16:42 ` Yu-cheng Yu 2019-08-14 16:58 ` Dave Hansen 2019-08-14 16:58 ` Dave Hansen 2019-08-23 14:02 ` Peter Zijlstra 2019-08-23 14:02 ` Peter Zijlstra 2019-08-27 22:37 ` Yu-cheng Yu 2019-08-27 22:37 ` Yu-cheng Yu 2019-08-27 22:37 ` Yu-cheng Yu 2019-08-28 7:03 ` Peter Zijlstra 2019-08-28 7:03 ` Peter Zijlstra 2019-08-28 14:57 ` Yu-cheng Yu 2019-08-28 14:57 ` Yu-cheng Yu 2019-08-28 14:57 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 12/27] drm/i915/gvt: Update _PAGE_DIRTY to _PAGE_DIRTY_BITS Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 13/27] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 14/27] x86/mm: Shadow stack page fault error checking Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 15/27] mm: Handle shadow stack page fault Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 22:55 ` Andy Lutomirski 2019-08-13 22:55 ` Andy Lutomirski 2019-08-13 22:55 ` Andy Lutomirski 2019-08-14 16:27 ` Yu-cheng Yu 2019-08-14 16:27 ` Yu-cheng Yu 2019-08-14 16:27 ` Yu-cheng Yu 2019-08-14 16:48 ` Dave Hansen 2019-08-14 16:48 ` Dave Hansen 2019-08-14 17:00 ` Yu-cheng Yu 2019-08-14 17:00 ` Yu-cheng Yu 2019-08-14 17:00 ` Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu [this message] 2019-08-13 20:52 ` [PATCH v8 16/27] mm: Handle THP/HugeTLB " Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 17/27] mm: Update can_follow_write_pte/pmd for shadow stack Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 18/27] mm: Introduce do_mmap_locked() Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-20 1:02 ` Sean Christopherson 2019-08-20 1:02 ` Sean Christopherson 2019-08-20 16:08 ` Yu-cheng Yu 2019-08-20 16:08 ` Yu-cheng Yu 2019-08-20 16:08 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 19/27] x86/cet/shstk: User-mode shadow stack support Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 20/27] x86/cet/shstk: Introduce WRUSS instruction Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 21/27] x86/cet/shstk: Handle signals for shadow stack Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 22/27] binfmt_elf: Extract .note.gnu.property from an ELF file Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-16 13:55 ` Dave Martin 2019-08-16 13:55 ` Dave Martin 2019-08-20 10:02 ` Dave Martin 2019-08-20 10:02 ` Dave Martin 2019-08-13 20:52 ` [PATCH v8 23/27] x86/cet/shstk: ELF header parsing of Shadow Stack Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 24/27] x86/cet/shstk: Handle thread shadow stack Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 25/27] mm/mmap: Add Shadow stack pages to memory accounting Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 26/27] x86/cet/shstk: Add arch_prctl functions for Shadow Stack Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-08-13 20:52 ` [PATCH v8 27/27] x86/cet/shstk: Add Shadow Stack instructions to opcode map Yu-cheng Yu 2019-08-13 20:52 ` Yu-cheng Yu 2019-11-08 13:27 ` Adrian Hunter 2019-11-08 13:27 ` Adrian Hunter 2019-11-08 18:09 ` Yu-cheng Yu 2019-11-08 18:09 ` Yu-cheng Yu 2019-11-08 18:09 ` Yu-cheng Yu
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20190813205225.12032-17-yu-cheng.yu@intel.com \ --to=yu-cheng.yu@intel.com \ --cc=Dave.Martin@arm.com \ --cc=arnd@arndb.de \ --cc=bp@alien8.de \ --cc=bsingharora@gmail.com \ --cc=corbet@lwn.net \ --cc=dave.hansen@linux.intel.com \ --cc=esyr@redhat.com \ --cc=fweimer@redhat.com \ --cc=gorcunov@gmail.com \ --cc=hjl.tools@gmail.com \ --cc=hpa@zytor.com \ --cc=jannh@google.com \ --cc=keescook@chromium.org \ --cc=linux-api@vger.kernel.org \ --cc=linux-arch@vger.kernel.org \ --cc=linux-doc@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=luto@amacapital.net \ --cc=mike.kravetz@oracle.com \ --cc=mingo@redhat.com \ --cc=nadav.amit@gmail.com \ --cc=oleg@redhat.com \ --cc=pavel@ucw.cz \ --cc=peterz@infradead.org \ --cc=ravi.v.shankar@intel.com \ --cc=rdunlap@infradead.org \ --cc=tglx@linutronix.de \ --cc=vedvyas.shanbhogue@intel.com \ --cc=x86@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.