From: Khalid Aziz <khalid.aziz@oracle.com> To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, akpm@linux-foundation.org, mhocko@suse.com, catalin.marinas@arm.com, will.deacon@arm.com, jmorris@namei.org, konrad.wilk@oracle.com Cc: deepa.srinivasan@oracle.com, chris.hyser@oracle.com, tyhicks@canonical.com, dwmw@amazon.co.uk, andrew.cooper3@citrix.com, jcm@redhat.com, boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com, oao.m.martins@oracle.com, jmattson@google.com, pradeep.vincent@oracle.com, john.haxby@oracle.com, tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de, steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org, dave.hansen@intel.com, peterz@infradead.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Khalid Aziz <khalid.aziz@oracle.com>, "Vasileios P . Kemerlis" <vpk@cs.columbia.edu>, Juerg Haefliger <juerg.haefliger@canonical.com>, Tycho Andersen <tycho@docker.com>, Marco Benatto <marco.antonio.780@gmail.com>, David Woodhouse <dwmw2@infradead.org> Subject: [RFC PATCH v8 12/14] xpfo, mm: optimize spinlock usage in xpfo_kunmap Date: Wed, 13 Feb 2019 17:01:35 -0700 [thread overview] Message-ID: <fb183dba421ad5cc1eda9acaf0678970bdf2159b.1550088114.git.khalid.aziz@oracle.com> (raw) In-Reply-To: <cover.1550088114.git.khalid.aziz@oracle.com> In-Reply-To: <cover.1550088114.git.khalid.aziz@oracle.com> From: Julian Stecklina <jsteckli@amazon.de> Only the xpfo_kunmap call that needs to actually unmap the page needs to be serialized. We need to be careful to handle the case, where after the atomic decrement of the mapcount, a xpfo_kmap increased the mapcount again. In this case, we can safely skip modifying the page table. Model-checked with up to 4 concurrent callers with Spin. Signed-off-by: Julian Stecklina <jsteckli@amazon.de> Signed-off-by: Khalid Aziz <khalid.aziz@oracle.com> Cc: x86@kernel.org Cc: kernel-hardening@lists.openwall.com Cc: Vasileios P. Kemerlis <vpk@cs.columbia.edu> Cc: Juerg Haefliger <juerg.haefliger@canonical.com> Cc: Tycho Andersen <tycho@docker.com> Cc: Marco Benatto <marco.antonio.780@gmail.com> Cc: David Woodhouse <dwmw2@infradead.org> --- mm/xpfo.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/mm/xpfo.c b/mm/xpfo.c index dc03c423c52f..5157cbebce4b 100644 --- a/mm/xpfo.c +++ b/mm/xpfo.c @@ -124,28 +124,35 @@ EXPORT_SYMBOL(xpfo_kmap); void xpfo_kunmap(void *kaddr, struct page *page) { + bool flush_tlb = false; + if (!static_branch_unlikely(&xpfo_inited)) return; if (!PageXpfoUser(page)) return; - spin_lock(&page->xpfo_lock); - /* * The page is to be allocated back to user space, so unmap it from the * kernel, flush the TLB and tag it as a user page. */ if (atomic_dec_return(&page->xpfo_mapcount) == 0) { -#ifdef CONFIG_XPFO_DEBUG - BUG_ON(PageXpfoUnmapped(page)); -#endif - SetPageXpfoUnmapped(page); - set_kpte(kaddr, page, __pgprot(0)); - xpfo_flush_kernel_tlb(page, 0); + spin_lock(&page->xpfo_lock); + + /* + * In the case, where we raced with kmap after the + * atomic_dec_return, we must not nuke the mapping. + */ + if (atomic_read(&page->xpfo_mapcount) == 0) { + SetPageXpfoUnmapped(page); + set_kpte(kaddr, page, __pgprot(0)); + flush_tlb = true; + } + spin_unlock(&page->xpfo_lock); } - spin_unlock(&page->xpfo_lock); + if (flush_tlb) + xpfo_flush_kernel_tlb(page, 0); } EXPORT_SYMBOL(xpfo_kunmap); -- 2.17.1
WARNING: multiple messages have this Message-ID (diff)
From: Khalid Aziz <khalid.aziz@oracle.com> To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de, ak@linux.intel.com, torvalds@linux-foundation.org, liran.alon@oracle.com, keescook@google.com, akpm@linux-foundation.org, mhocko@suse.com, catalin.marinas@arm.com, will.deacon@arm.com, jmorris@namei.org, konrad.wilk@oracle.com Cc: Tycho Andersen <tycho@docker.com>, kernel-hardening@lists.openwall.com, peterz@infradead.org, dave.hansen@intel.com, Khalid Aziz <khalid.aziz@oracle.com>, deepa.srinivasan@oracle.com, steven.sistare@oracle.com, hch@lst.de, "Vasileios P . Kemerlis" <vpk@cs.columbia.edu>, x86@kernel.org, kanth.ghatraju@oracle.com, Juerg Haefliger <juerg.haefliger@canonical.com>, labbott@redhat.com, pradeep.vincent@oracle.com, jcm@redhat.com, luto@kernel.org, boris.ostrovsky@oracle.com, chris.hyser@oracle.com, linux-arm-kernel@lists.infradead.org, jmattson@google.com, Marco Benatto <marco.antonio.780@gmail.com>, linux-mm@kvack.org, andrew.cooper3@citrix.com, linux-kernel@vger.kernel.org, tyhicks@canonical.com, john.haxby@oracle.com, tglx@linutronix.de, oao.m.martins@oracle.com, David Woodhouse <dwmw2@infradead.org>, dwmw@amazon.co.uk, kirill.shutemov@linux.intel.com Subject: [RFC PATCH v8 12/14] xpfo, mm: optimize spinlock usage in xpfo_kunmap Date: Wed, 13 Feb 2019 17:01:35 -0700 [thread overview] Message-ID: <fb183dba421ad5cc1eda9acaf0678970bdf2159b.1550088114.git.khalid.aziz@oracle.com> (raw) In-Reply-To: <cover.1550088114.git.khalid.aziz@oracle.com> In-Reply-To: <cover.1550088114.git.khalid.aziz@oracle.com> From: Julian Stecklina <jsteckli@amazon.de> Only the xpfo_kunmap call that needs to actually unmap the page needs to be serialized. We need to be careful to handle the case, where after the atomic decrement of the mapcount, a xpfo_kmap increased the mapcount again. In this case, we can safely skip modifying the page table. Model-checked with up to 4 concurrent callers with Spin. Signed-off-by: Julian Stecklina <jsteckli@amazon.de> Signed-off-by: Khalid Aziz <khalid.aziz@oracle.com> Cc: x86@kernel.org Cc: kernel-hardening@lists.openwall.com Cc: Vasileios P. Kemerlis <vpk@cs.columbia.edu> Cc: Juerg Haefliger <juerg.haefliger@canonical.com> Cc: Tycho Andersen <tycho@docker.com> Cc: Marco Benatto <marco.antonio.780@gmail.com> Cc: David Woodhouse <dwmw2@infradead.org> --- mm/xpfo.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/mm/xpfo.c b/mm/xpfo.c index dc03c423c52f..5157cbebce4b 100644 --- a/mm/xpfo.c +++ b/mm/xpfo.c @@ -124,28 +124,35 @@ EXPORT_SYMBOL(xpfo_kmap); void xpfo_kunmap(void *kaddr, struct page *page) { + bool flush_tlb = false; + if (!static_branch_unlikely(&xpfo_inited)) return; if (!PageXpfoUser(page)) return; - spin_lock(&page->xpfo_lock); - /* * The page is to be allocated back to user space, so unmap it from the * kernel, flush the TLB and tag it as a user page. */ if (atomic_dec_return(&page->xpfo_mapcount) == 0) { -#ifdef CONFIG_XPFO_DEBUG - BUG_ON(PageXpfoUnmapped(page)); -#endif - SetPageXpfoUnmapped(page); - set_kpte(kaddr, page, __pgprot(0)); - xpfo_flush_kernel_tlb(page, 0); + spin_lock(&page->xpfo_lock); + + /* + * In the case, where we raced with kmap after the + * atomic_dec_return, we must not nuke the mapping. + */ + if (atomic_read(&page->xpfo_mapcount) == 0) { + SetPageXpfoUnmapped(page); + set_kpte(kaddr, page, __pgprot(0)); + flush_tlb = true; + } + spin_unlock(&page->xpfo_lock); } - spin_unlock(&page->xpfo_lock); + if (flush_tlb) + xpfo_flush_kernel_tlb(page, 0); } EXPORT_SYMBOL(xpfo_kunmap); -- 2.17.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2019-02-14 0:06 UTC|newest] Thread overview: 64+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-02-14 0:01 [RFC PATCH v8 00/14] Add support for eXclusive Page Frame Ownership Khalid Aziz 2019-02-14 0:01 ` Khalid Aziz 2019-02-14 0:01 ` [RFC PATCH v8 01/14] mm: add MAP_HUGETLB support to vm_mmap Khalid Aziz 2019-02-14 0:01 ` Khalid Aziz 2019-02-14 0:01 ` [RFC PATCH v8 02/14] x86: always set IF before oopsing from page fault Khalid Aziz 2019-02-14 0:01 ` Khalid Aziz 2019-02-14 0:01 ` [RFC PATCH v8 03/14] mm, x86: Add support for eXclusive Page Frame Ownership (XPFO) Khalid Aziz 2019-02-14 0:01 ` Khalid Aziz 2019-02-14 10:56 ` Peter Zijlstra 2019-02-14 10:56 ` Peter Zijlstra 2019-02-14 16:15 ` Borislav Petkov 2019-02-14 16:15 ` Borislav Petkov 2019-02-14 17:19 ` Khalid Aziz 2019-02-14 17:19 ` Khalid Aziz 2019-02-14 17:13 ` Khalid Aziz 2019-02-14 17:13 ` Khalid Aziz 2019-02-14 19:08 ` Peter Zijlstra 2019-02-14 19:08 ` Peter Zijlstra 2019-02-14 19:58 ` Khalid Aziz 2019-02-14 19:58 ` Khalid Aziz 2019-02-14 0:01 ` [RFC PATCH v8 04/14] swiotlb: Map the buffer if it was unmapped by XPFO Khalid Aziz 2019-02-14 0:01 ` Khalid Aziz 2019-02-14 7:47 ` Christoph Hellwig 2019-02-14 7:47 ` Christoph Hellwig 2019-02-14 16:56 ` Khalid Aziz 2019-02-14 16:56 ` Khalid Aziz 2019-02-14 17:44 ` Christoph Hellwig 2019-02-14 17:44 ` Christoph Hellwig 2019-02-14 19:48 ` Khalid Aziz 2019-02-14 19:48 ` Khalid Aziz 2019-02-14 0:01 ` [RFC PATCH v8 05/14] arm64/mm: Add support for XPFO Khalid Aziz 2019-02-14 0:01 ` Khalid Aziz 2019-02-14 0:01 ` [RFC PATCH v8 06/14] xpfo: add primitives for mapping underlying memory Khalid Aziz 2019-02-14 0:01 ` Khalid Aziz 2019-02-14 0:01 ` [RFC PATCH v8 07/14] arm64/mm, xpfo: temporarily map dcache regions Khalid Aziz 2019-02-14 0:01 ` Khalid Aziz 2019-02-14 15:54 ` Tycho Andersen 2019-02-14 15:54 ` Tycho Andersen 2019-02-14 17:29 ` Khalid Aziz 2019-02-14 17:29 ` Khalid Aziz 2019-02-14 23:49 ` Tycho Andersen 2019-02-14 23:49 ` Tycho Andersen 2019-02-14 0:01 ` [RFC PATCH v8 08/14] arm64/mm: disable section/contiguous mappings if XPFO is enabled Khalid Aziz 2019-02-14 0:01 ` Khalid Aziz 2019-02-15 13:09 ` Mark Rutland 2019-02-15 13:09 ` Mark Rutland 2019-02-15 14:47 ` Khalid Aziz 2019-02-15 14:47 ` Khalid Aziz 2019-02-14 0:01 ` [RFC PATCH v8 09/14] mm: add a user_virt_to_phys symbol Khalid Aziz 2019-02-14 0:01 ` Khalid Aziz 2019-02-14 0:01 ` [RFC PATCH v8 10/14] lkdtm: Add test for XPFO Khalid Aziz 2019-02-14 0:01 ` Khalid Aziz 2019-02-14 0:01 ` [RFC PATCH v8 11/14] xpfo, mm: remove dependency on CONFIG_PAGE_EXTENSION Khalid Aziz 2019-02-14 0:01 ` Khalid Aziz 2019-02-14 0:01 ` Khalid Aziz [this message] 2019-02-14 0:01 ` [RFC PATCH v8 12/14] xpfo, mm: optimize spinlock usage in xpfo_kunmap Khalid Aziz 2019-02-14 0:01 ` [RFC PATCH v8 13/14] xpfo, mm: Defer TLB flushes for non-current CPUs (x86 only) Khalid Aziz 2019-02-14 0:01 ` Khalid Aziz 2019-02-14 17:42 ` Dave Hansen 2019-02-14 17:42 ` Dave Hansen 2019-02-14 19:57 ` Khalid Aziz 2019-02-14 19:57 ` Khalid Aziz 2019-02-14 0:01 ` [RFC PATCH v8 14/14] xpfo, mm: Optimize XPFO TLB flushes by batching them together Khalid Aziz 2019-02-14 0:01 ` Khalid Aziz
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=fb183dba421ad5cc1eda9acaf0678970bdf2159b.1550088114.git.khalid.aziz@oracle.com \ --to=khalid.aziz@oracle.com \ --cc=ak@linux.intel.com \ --cc=akpm@linux-foundation.org \ --cc=andrew.cooper3@citrix.com \ --cc=boris.ostrovsky@oracle.com \ --cc=catalin.marinas@arm.com \ --cc=chris.hyser@oracle.com \ --cc=dave.hansen@intel.com \ --cc=deepa.srinivasan@oracle.com \ --cc=dwmw2@infradead.org \ --cc=dwmw@amazon.co.uk \ --cc=hch@lst.de \ --cc=jcm@redhat.com \ --cc=jmattson@google.com \ --cc=jmorris@namei.org \ --cc=john.haxby@oracle.com \ --cc=jsteckli@amazon.de \ --cc=juerg.haefliger@canonical.com \ --cc=juergh@gmail.com \ --cc=kanth.ghatraju@oracle.com \ --cc=keescook@google.com \ --cc=kernel-hardening@lists.openwall.com \ --cc=kirill.shutemov@linux.intel.com \ --cc=konrad.wilk@oracle.com \ --cc=labbott@redhat.com \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=liran.alon@oracle.com \ --cc=luto@kernel.org \ --cc=marco.antonio.780@gmail.com \ --cc=mhocko@suse.com \ --cc=oao.m.martins@oracle.com \ --cc=peterz@infradead.org \ --cc=pradeep.vincent@oracle.com \ --cc=steven.sistare@oracle.com \ --cc=tglx@linutronix.de \ --cc=torvalds@linux-foundation.org \ --cc=tycho@docker.com \ --cc=tycho@tycho.ws \ --cc=tyhicks@canonical.com \ --cc=vpk@cs.columbia.edu \ --cc=will.deacon@arm.com \ --cc=x86@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.