linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed
From: Khalid Aziz <khalid.aziz@oracle.com>
To: juergh@gmail.com, tycho@tycho.ws, jsteckli@amazon.de,
	ak@linux.intel.com, liran.alon@oracle.com, keescook@google.com,
	akpm@linux-foundation.org, konrad.wilk@oracle.com
Cc: Khalid Aziz <khalid.aziz@oracle.com>,
	deepa.srinivasan@oracle.com, chris.hyser@oracle.com,
	tyhicks@canonical.com, dwmw@amazon.co.uk,
	andrew.cooper3@citrix.com, jcm@redhat.com,
	boris.ostrovsky@oracle.com, kanth.ghatraju@oracle.com,
	joao.m.martins@oracle.com, jmattson@google.com,
	pradeep.vincent@oracle.com, john.haxby@oracle.com,
	tglx@linutronix.de, kirill.shutemov@linux.intel.com, hch@lst.de,
	steven.sistare@oracle.com, labbott@redhat.com, luto@kernel.org,
	dave.hansen@intel.com, peterz@infradead.org, aaron.lu@intel.com,
	alexander.h.duyck@linux.intel.com, amir73il@gmail.com,
	andreyknvl@google.com, aneesh.kumar@linux.ibm.com,
	anthony.yznaga@oracle.com, ard.biesheuvel@linaro.org,
	arnd@arndb.de, arunks@codeaurora.org, ben@decadent.org.uk,
	bigeasy@linutronix.de, bp@alien8.de, brgl@bgdev.pl,
	catalin.marinas@arm.com, corbet@lwn.net, cpandya@codeaurora.org,
	daniel.vetter@ffwll.ch, dan.j.williams@intel.com,
	gregkh@linuxfoundation.org, guro@fb.com, hannes@cmpxchg.org,
	hpa@zytor.com, iamjoonsoo.kim@lge.com, james.morse@arm.com,
	jannh@google.com, jgross@suse.com, jkosina@suse.cz,
	jmorris@namei.org, joe@perches.com, jrdr.linux@gmail.com,
	jroedel@suse.de, keith.busch@intel.com,
	khlebnikov@yandex-team.ru, logang@deltatee.com,
	marco.antonio.780@gmail.com, mark.rutland@arm.com,
	mgorman@techsingularity.net, mhocko@suse.com, mhocko@suse.cz,
	mike.kravetz@oracle.com, mingo@redhat.com, mst@redhat.com,
	m.szyprowski@samsung.com, npiggin@gmail.com, osalvador@suse.de,
	paulmck@linux.vnet.ibm.com, pavel.tatashin@microsoft.com,
	rdunlap@infradead.org, richard.weiyang@gmail.com,
	riel@surriel.com, rientjes@google.com, robin.murphy@arm.com,
	rostedt@goodmis.org, rppt@linux.vnet.ibm.com,
	sai.praneeth.prakhya@intel.com, serge@hallyn.com,
	steve.capper@arm.com, thymovanbeers@gmail.com, vbabka@suse.cz,
	will.deacon@arm.com, willy@infradead.org,
	yang.shi@linux.alibaba.com, yaojun8558363@gmail.com,
	ying.huang@intel.com, zhangshaokun@hisilicon.com,
	khalid@gonehiking.org, iommu@lists.linux-foundation.org,
	x86@kernel.org, linux-arm-kernel@lists.infradead.org,
	linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org,
	linux-mm@kvack.org, linux-security-module@vger.kernel.org
Subject: [RFC PATCH v9 00/13] Add support for eXclusive Page Frame Ownership
Date: Wed,  3 Apr 2019 11:34:01 -0600	[thread overview]
Message-ID: <cover.1554248001.git.khalid.aziz@oracle.com> (raw)

This is another update to the work Juerg, Tycho and Julian have
done on XPFO. After the last round of updates, we were seeing very
significant performance penalties when stale TLB entries were
flushed actively after an XPFO TLB update.  Benchmark for measuring
performance is kernel build using parallel make. To get full
protection from ret2dir attackes, we must flush stale TLB entries.
Performance penalty from flushing stale TLB entries goes up as the
number of cores goes up. On a desktop class machine with only 4
cores, enabling TLB flush for stale entries causes system time for
"make -j4" to go up by a factor of 2.61x but on a larger machine
with 96 cores, system time with "make -j60" goes up by a factor of
26.37x!  I have been working on reducing this performance penalty.

I implemented two solutions to reduce performance penalty and that
has had large impact. XPFO code flushes TLB every time a page is
allocated to userspace. It does so by sending IPIs to all processors
to flush TLB. Back to back allocations of pages to userspace on
multiple processors results in a storm of IPIs.  Each one of these
incoming IPIs is handled by a processor by flushing its TLB. To
reduce this IPI storm, I have added a per CPU flag that can be set
to tell a processor to flush its TLB. A processor checks this flag
on every context switch. If the flag is set, it flushes its TLB and
clears the flag. This allows for multiple TLB flush requests to a
single CPU to be combined into a single request. A kernel TLB entry
for a page that has been allocated to userspace is flushed on all
processors unlike the previous version of this patch. A processor
could hold a stale kernel TLB entry that was removed on another
processor until the next context switch. A local userspace page
allocation by the currently running process could force the TLB
flush earlier for such entries.

The other solution reduces the number of TLB flushes required, by
performing TLB flush for multiple pages at one time when pages are
refilled on the per-cpu freelist. If the pages being addedd to
per-cpu freelist are marked for userspace allocation, TLB entries
for these pages can be flushed upfront and pages tagged as currently
unmapped. When any such page is allocated to userspace, there is no
need to performa a TLB flush at that time any more. This batching of
TLB flushes reduces performance imapct further. Similarly when
these user pages are freed by userspace and added back to per-cpu
free list, they are left unmapped and tagged so. This further
optimization reduced performance impact from 1.32x to 1.28x for
96-core server and from 1.31x to 1.27x for a 4-core desktop.

I measured system time for parallel make with unmodified 4.20
kernel, 4.20 with XPFO patches before these patches and then again
after applying each of these patches. Here are the results:

Hardware: 96-core Intel Xeon Platinum 8160 CPU @ 2.10GHz, 768 GB RAM
make -j60 all

5.0					913.862s
5.0+this patch series			1165.259ss	1.28x


Hardware: 4-core Intel Core i5-3550 CPU @ 3.30GHz, 8G RAM
make -j4 all

5.0					610.642s
5.0+this patch series			773.075s	1.27x

Performance with this patch set is good enough to use these as
starting point for further refinement before we merge it into main
kernel, hence RFC.

I have restructurerd the patches in this version to separate out
architecture independent code. I folded much of the code
improvement by Julian to not use page extension into patch 3. 

What remains to be done beyond this patch series:

1. Performance improvements: Ideas to explore - (1) kernel mappings
   private to an mm, (2) Any others??
2. Re-evaluate the patch "arm64/mm: Add support for XPFO to swiotlb"
   from Juerg. I dropped it for now since swiotlb code for ARM has
   changed a lot since this patch was written. I could use help
   from ARM experts on this.
3. Extend the patch "xpfo, mm: Defer TLB flushes for non-current
   CPUs" to other architectures besides x86.
4. Change kmap to not map the page back to physmap, instead map it
   to a new va similar to what kmap_high does. Mapping page back
   into physmap re-opens the ret2dir security for the duration of
   kmap. All of the kmap_high and related code can be reused for this
   but that will require restructuring that code so it can be built for
   64-bits as well. Any objections to that?

---------------------------------------------------------

Juerg Haefliger (6):
  mm: Add support for eXclusive Page Frame Ownership (XPFO)
  xpfo, x86: Add support for XPFO for x86-64
  lkdtm: Add test for XPFO
  arm64/mm: Add support for XPFO
  swiotlb: Map the buffer if it was unmapped by XPFO
  arm64/mm, xpfo: temporarily map dcache regions

Julian Stecklina (1):
  xpfo, mm: optimize spinlock usage in xpfo_kunmap

Khalid Aziz (2):
  xpfo, mm: Defer TLB flushes for non-current CPUs (x86 only)
  xpfo, mm: Optimize XPFO TLB flushes by batching them together

Tycho Andersen (4):
  mm: add MAP_HUGETLB support to vm_mmap
  x86: always set IF before oopsing from page fault
  mm: add a user_virt_to_phys symbol
  xpfo: add primitives for mapping underlying memory

 .../admin-guide/kernel-parameters.txt         |   6 +
 arch/arm64/Kconfig                            |   1 +
 arch/arm64/mm/Makefile                        |   2 +
 arch/arm64/mm/flush.c                         |   7 +
 arch/arm64/mm/mmu.c                           |   2 +-
 arch/arm64/mm/xpfo.c                          |  66 ++++++
 arch/x86/Kconfig                              |   1 +
 arch/x86/include/asm/pgtable.h                |  26 +++
 arch/x86/include/asm/tlbflush.h               |   1 +
 arch/x86/mm/Makefile                          |   2 +
 arch/x86/mm/fault.c                           |   6 +
 arch/x86/mm/pageattr.c                        |  32 +--
 arch/x86/mm/tlb.c                             |  39 ++++
 arch/x86/mm/xpfo.c                            | 185 +++++++++++++++++
 drivers/misc/lkdtm/Makefile                   |   1 +
 drivers/misc/lkdtm/core.c                     |   3 +
 drivers/misc/lkdtm/lkdtm.h                    |   5 +
 drivers/misc/lkdtm/xpfo.c                     | 196 ++++++++++++++++++
 include/linux/highmem.h                       |  34 +--
 include/linux/mm.h                            |   2 +
 include/linux/mm_types.h                      |   8 +
 include/linux/page-flags.h                    |  23 +-
 include/linux/xpfo.h                          | 191 +++++++++++++++++
 include/trace/events/mmflags.h                |  10 +-
 kernel/dma/swiotlb.c                          |   3 +-
 mm/Makefile                                   |   1 +
 mm/compaction.c                               |   2 +-
 mm/internal.h                                 |   2 +-
 mm/mmap.c                                     |  19 +-
 mm/page_alloc.c                               |  19 +-
 mm/page_isolation.c                           |   2 +-
 mm/util.c                                     |  32 +++
 mm/xpfo.c                                     | 170 +++++++++++++++
 security/Kconfig                              |  27 +++
 34 files changed, 1047 insertions(+), 79 deletions(-)
 create mode 100644 arch/arm64/mm/xpfo.c
 create mode 100644 arch/x86/mm/xpfo.c
 create mode 100644 drivers/misc/lkdtm/xpfo.c
 create mode 100644 include/linux/xpfo.h
 create mode 100644 mm/xpfo.c

-- 
2.17.1


             reply	other threads:[~2019-04-03 17:37 UTC|newest]

Thread overview: 70+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-04-03 17:34 Khalid Aziz [this message]
2019-04-03 17:34 ` [RFC PATCH v9 01/13] mm: add MAP_HUGETLB support to vm_mmap Khalid Aziz
2019-04-03 17:34 ` [RFC PATCH v9 02/13] x86: always set IF before oopsing from page fault Khalid Aziz
2019-04-04  0:12   ` Andy Lutomirski
2019-04-04  1:42     ` Tycho Andersen
2019-04-04  4:12       ` Andy Lutomirski
2019-04-04 15:47         ` Tycho Andersen
2019-04-04 16:23           ` Sebastian Andrzej Siewior
2019-04-04 16:28           ` Thomas Gleixner
2019-04-04 17:11             ` Andy Lutomirski
2019-04-03 17:34 ` [RFC PATCH v9 03/13] mm: Add support for eXclusive Page Frame Ownership (XPFO) Khalid Aziz
2019-04-04  7:21   ` Peter Zijlstra
2019-04-04  9:25     ` Peter Zijlstra
2019-04-04 14:48     ` Tycho Andersen
2019-04-04  7:43   ` Peter Zijlstra
2019-04-04 15:15     ` Khalid Aziz
2019-04-04 17:01       ` Peter Zijlstra
2019-04-17 16:15   ` Ingo Molnar
2019-04-17 16:49     ` Khalid Aziz
2019-04-17 17:09       ` Ingo Molnar
2019-04-17 17:19         ` Nadav Amit
2019-04-17 17:26           ` Ingo Molnar
2019-04-17 17:44             ` Nadav Amit
2019-04-17 21:19               ` Thomas Gleixner
2019-04-17 23:18                 ` Linus Torvalds
2019-04-17 23:42                   ` Thomas Gleixner
2019-04-17 23:52                     ` Linus Torvalds
2019-04-18  4:41                       ` Andy Lutomirski
2019-04-18  5:41                         ` Kees Cook
2019-04-18 14:34                           ` Khalid Aziz
2019-04-22 19:30                             ` Khalid Aziz
2019-04-22 22:23                             ` Kees Cook
2019-04-18  6:14                       ` Thomas Gleixner
2019-04-17 17:33         ` Khalid Aziz
2019-04-17 19:49           ` Andy Lutomirski
2019-04-17 19:52             ` Tycho Andersen
2019-04-17 20:12             ` Khalid Aziz
2019-05-01 14:49       ` Waiman Long
2019-05-01 15:18         ` Khalid Aziz
2019-04-03 17:34 ` [RFC PATCH v9 04/13] xpfo, x86: Add support for XPFO for x86-64 Khalid Aziz
2019-04-04  7:52   ` Peter Zijlstra
2019-04-04 15:40     ` Khalid Aziz
2019-04-03 17:34 ` [RFC PATCH v9 05/13] mm: add a user_virt_to_phys symbol Khalid Aziz
2019-04-03 17:34 ` [RFC PATCH v9 06/13] lkdtm: Add test for XPFO Khalid Aziz
2019-04-03 17:34 ` [RFC PATCH v9 07/13] arm64/mm: Add support " Khalid Aziz
2019-04-03 17:34 ` [RFC PATCH v9 08/13] swiotlb: Map the buffer if it was unmapped by XPFO Khalid Aziz
2019-04-03 17:34 ` [RFC PATCH v9 09/13] xpfo: add primitives for mapping underlying memory Khalid Aziz
2019-04-03 17:34 ` [RFC PATCH v9 10/13] arm64/mm, xpfo: temporarily map dcache regions Khalid Aziz
2019-04-03 17:34 ` [RFC PATCH v9 11/13] xpfo, mm: optimize spinlock usage in xpfo_kunmap Khalid Aziz
2019-04-04  7:56   ` Peter Zijlstra
2019-04-04 16:06     ` Khalid Aziz
2019-04-03 17:34 ` [RFC PATCH v9 12/13] xpfo, mm: Defer TLB flushes for non-current CPUs (x86 only) Khalid Aziz
2019-04-04  4:10   ` Andy Lutomirski
     [not found]     ` <91f1dbce-332e-25d1-15f6-0e9cfc8b797b@oracle.com>
2019-04-05  7:17       ` Thomas Gleixner
2019-04-05 14:44         ` Dave Hansen
2019-04-05 15:24           ` Andy Lutomirski
2019-04-05 15:56             ` Tycho Andersen
2019-04-05 16:32               ` Andy Lutomirski
2019-04-05 15:56             ` Khalid Aziz
2019-04-05 16:01             ` Dave Hansen
2019-04-05 16:27               ` Andy Lutomirski
2019-04-05 16:41                 ` Peter Zijlstra
2019-04-05 17:35                 ` Khalid Aziz
2019-04-05 15:44           ` Khalid Aziz
2019-04-05 15:24       ` Andy Lutomirski
2019-04-04  8:18   ` Peter Zijlstra
2019-04-03 17:34 ` [RFC PATCH v9 13/13] xpfo, mm: Optimize XPFO TLB flushes by batching them together Khalid Aziz
2019-04-04 16:44 ` [RFC PATCH v9 00/13] Add support for eXclusive Page Frame Ownership Nadav Amit
2019-04-04 17:18   ` Khalid Aziz
2019-04-06  6:40 ` Jon Masters

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=cover.1554248001.git.khalid.aziz@oracle.com \
    --to=khalid.aziz@oracle.com \
    --cc=aaron.lu@intel.com \
    --cc=ak@linux.intel.com \
    --cc=akpm@linux-foundation.org \
    --cc=alexander.h.duyck@linux.intel.com \
    --cc=amir73il@gmail.com \
    --cc=andrew.cooper3@citrix.com \
    --cc=andreyknvl@google.com \
    --cc=aneesh.kumar@linux.ibm.com \
    --cc=anthony.yznaga@oracle.com \
    --cc=ard.biesheuvel@linaro.org \
    --cc=arnd@arndb.de \
    --cc=arunks@codeaurora.org \
    --cc=ben@decadent.org.uk \
    --cc=bigeasy@linutronix.de \
    --cc=boris.ostrovsky@oracle.com \
    --cc=bp@alien8.de \
    --cc=brgl@bgdev.pl \
    --cc=catalin.marinas@arm.com \
    --cc=chris.hyser@oracle.com \
    --cc=corbet@lwn.net \
    --cc=cpandya@codeaurora.org \
    --cc=dan.j.williams@intel.com \
    --cc=daniel.vetter@ffwll.ch \
    --cc=dave.hansen@intel.com \
    --cc=deepa.srinivasan@oracle.com \
    --cc=dwmw@amazon.co.uk \
    --cc=gregkh@linuxfoundation.org \
    --cc=guro@fb.com \
    --cc=hannes@cmpxchg.org \
    --cc=hch@lst.de \
    --cc=hpa@zytor.com \
    --cc=iamjoonsoo.kim@lge.com \
    --cc=iommu@lists.linux-foundation.org \
    --cc=james.morse@arm.com \
    --cc=jannh@google.com \
    --cc=jcm@redhat.com \
    --cc=jgross@suse.com \
    --cc=jkosina@suse.cz \
    --cc=jmattson@google.com \
    --cc=jmorris@namei.org \
    --cc=joao.m.martins@oracle.com \
    --cc=joe@perches.com \
    --cc=john.haxby@oracle.com \
    --cc=jrdr.linux@gmail.com \
    --cc=jroedel@suse.de \
    --cc=jsteckli@amazon.de \
    --cc=juergh@gmail.com \
    --cc=kanth.ghatraju@oracle.com \
    --cc=keescook@google.com \
    --cc=keith.busch@intel.com \
    --cc=khalid@gonehiking.org \
    --cc=khlebnikov@yandex-team.ru \
    --cc=kirill.shutemov@linux.intel.com \
    --cc=konrad.wilk@oracle.com \
    --cc=labbott@redhat.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=linux-security-module@vger.kernel.org \
    --cc=liran.alon@oracle.com \
    --cc=logang@deltatee.com \
    --cc=luto@kernel.org \
    --cc=m.szyprowski@samsung.com \
    --cc=marco.antonio.780@gmail.com \
    --cc=mark.rutland@arm.com \
    --cc=mgorman@techsingularity.net \
    --cc=mhocko@suse.com \
    --cc=mhocko@suse.cz \
    --cc=mike.kravetz@oracle.com \
    --cc=mingo@redhat.com \
    --cc=mst@redhat.com \
    --cc=npiggin@gmail.com \
    --cc=osalvador@suse.de \
    --cc=paulmck@linux.vnet.ibm.com \
    --cc=pavel.tatashin@microsoft.com \
    --cc=peterz@infradead.org \
    --cc=pradeep.vincent@oracle.com \
    --cc=rdunlap@infradead.org \
    --cc=richard.weiyang@gmail.com \
    --cc=riel@surriel.com \
    --cc=rientjes@google.com \
    --cc=robin.murphy@arm.com \
    --cc=rostedt@goodmis.org \
    --cc=rppt@linux.vnet.ibm.com \
    --cc=sai.praneeth.prakhya@intel.com \
    --cc=serge@hallyn.com \
    --cc=steve.capper@arm.com \
    --cc=steven.sistare@oracle.com \
    --cc=tglx@linutronix.de \
    --cc=thymovanbeers@gmail.com \
    --cc=tycho@tycho.ws \
    --cc=tyhicks@canonical.com \
    --cc=vbabka@suse.cz \
    --cc=will.deacon@arm.com \
    --cc=willy@infradead.org \
    --cc=x86@kernel.org \
    --cc=yang.shi@linux.alibaba.com \
    --cc=yaojun8558363@gmail.com \
    --cc=ying.huang@intel.com \
    --cc=zhangshaokun@hisilicon.com \
    --subject='Re: [RFC PATCH v9 00/13] Add support for eXclusive Page Frame Ownership' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).