From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
stable@vger.kernel.org, kernel test robot <oliver.sang@intel.com>,
Shile Zhang <shile.zhang@linux.alibaba.com>,
Joerg Roedel <jroedel@suse.de>,
Andrew Morton <akpm@linux-foundation.org>,
Borislav Petkov <bp@suse.de>,
Dave Hansen <dave.hansen@linux.intel.com>,
Andy Lutomirski <luto@kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@redhat.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
"Rafael J . Wysocki" <rafael.j.wysocki@intel.com>
Subject: [PATCH 4.4 25/91] x86/mm: split vmalloc_sync_all()
Date: Wed, 1 Apr 2020 18:17:21 +0200 [thread overview]
Message-ID: <20200401161521.679075629@linuxfoundation.org> (raw)
In-Reply-To: <20200401161512.917494101@linuxfoundation.org>
From: Joerg Roedel <jroedel@suse.de>
commit 763802b53a427ed3cbd419dbba255c414fdd9e7c upstream.
Commit 3f8fd02b1bf1 ("mm/vmalloc: Sync unmappings in
__purge_vmap_area_lazy()") introduced a call to vmalloc_sync_all() in
the vunmap() code-path. While this change was necessary to maintain
correctness on x86-32-pae kernels, it also adds additional cycles for
architectures that don't need it.
Specifically on x86-64 with CONFIG_VMAP_STACK=y some people reported
severe performance regressions in micro-benchmarks because it now also
calls the x86-64 implementation of vmalloc_sync_all() on vunmap(). But
the vmalloc_sync_all() implementation on x86-64 is only needed for newly
created mappings.
To avoid the unnecessary work on x86-64 and to gain the performance
back, split up vmalloc_sync_all() into two functions:
* vmalloc_sync_mappings(), and
* vmalloc_sync_unmappings()
Most call-sites to vmalloc_sync_all() only care about new mappings being
synchronized. The only exception is the new call-site added in the
above mentioned commit.
Shile Zhang directed us to a report of an 80% regression in reaim
throughput.
Fixes: 3f8fd02b1bf1 ("mm/vmalloc: Sync unmappings in __purge_vmap_area_lazy()")
Reported-by: kernel test robot <oliver.sang@intel.com>
Reported-by: Shile Zhang <shile.zhang@linux.alibaba.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Tested-by: Borislav Petkov <bp@suse.de>
Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> [GHES]
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: <stable@vger.kernel.org>
Link: http://lkml.kernel.org/r/20191009124418.8286-1-joro@8bytes.org
Link: https://lists.01.org/hyperkitty/list/lkp@lists.01.org/thread/4D3JPPHBNOSPFK2KEPC6KGKS6J25AIDB/
Link: http://lkml.kernel.org/r/20191113095530.228959-1-shile.zhang@linux.alibaba.com
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
arch/x86/mm/fault.c | 26 ++++++++++++++++++++++++--
drivers/acpi/apei/ghes.c | 2 +-
include/linux/vmalloc.h | 5 +++--
kernel/notifier.c | 2 +-
mm/nommu.c | 10 +++++++---
mm/vmalloc.c | 11 +++++++----
6 files changed, 43 insertions(+), 13 deletions(-)
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -228,7 +228,7 @@ static inline pmd_t *vmalloc_sync_one(pg
return pmd_k;
}
-void vmalloc_sync_all(void)
+static void vmalloc_sync(void)
{
unsigned long address;
@@ -255,6 +255,16 @@ void vmalloc_sync_all(void)
}
}
+void vmalloc_sync_mappings(void)
+{
+ vmalloc_sync();
+}
+
+void vmalloc_sync_unmappings(void)
+{
+ vmalloc_sync();
+}
+
/*
* 32-bit:
*
@@ -349,11 +359,23 @@ out:
#else /* CONFIG_X86_64: */
-void vmalloc_sync_all(void)
+void vmalloc_sync_mappings(void)
{
+ /*
+ * 64-bit mappings might allocate new p4d/pud pages
+ * that need to be propagated to all tasks' PGDs.
+ */
sync_global_pgds(VMALLOC_START & PGDIR_MASK, VMALLOC_END, 0);
}
+void vmalloc_sync_unmappings(void)
+{
+ /*
+ * Unmappings never allocate or free p4d/pud pages.
+ * No work is required here.
+ */
+}
+
/*
* 64-bit:
*
--- a/drivers/acpi/apei/ghes.c
+++ b/drivers/acpi/apei/ghes.c
@@ -229,7 +229,7 @@ static int ghes_estatus_pool_expand(unsi
* New allocation must be visible in all pgd before it can be found by
* an NMI allocating from the pool.
*/
- vmalloc_sync_all();
+ vmalloc_sync_mappings();
return gen_pool_add(ghes_estatus_pool, addr, PAGE_ALIGN(len), -1);
}
--- a/include/linux/vmalloc.h
+++ b/include/linux/vmalloc.h
@@ -92,8 +92,9 @@ extern int remap_vmalloc_range_partial(s
extern int remap_vmalloc_range(struct vm_area_struct *vma, void *addr,
unsigned long pgoff);
-void vmalloc_sync_all(void);
-
+void vmalloc_sync_mappings(void);
+void vmalloc_sync_unmappings(void);
+
/*
* Lowlevel-APIs (not for driver use!)
*/
--- a/kernel/notifier.c
+++ b/kernel/notifier.c
@@ -552,7 +552,7 @@ NOKPROBE_SYMBOL(notify_die);
int register_die_notifier(struct notifier_block *nb)
{
- vmalloc_sync_all();
+ vmalloc_sync_mappings();
return atomic_notifier_chain_register(&die_chain, nb);
}
EXPORT_SYMBOL_GPL(register_die_notifier);
--- a/mm/nommu.c
+++ b/mm/nommu.c
@@ -472,10 +472,14 @@ void vm_unmap_aliases(void)
EXPORT_SYMBOL_GPL(vm_unmap_aliases);
/*
- * Implement a stub for vmalloc_sync_all() if the architecture chose not to
- * have one.
+ * Implement a stub for vmalloc_sync_[un]mapping() if the architecture
+ * chose not to have one.
*/
-void __weak vmalloc_sync_all(void)
+void __weak vmalloc_sync_mappings(void)
+{
+}
+
+void __weak vmalloc_sync_unmappings(void)
{
}
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -1681,7 +1681,7 @@ void *__vmalloc_node_range(unsigned long
* First make sure the mappings are removed from all page-tables
* before they are freed.
*/
- vmalloc_sync_all();
+ vmalloc_sync_unmappings();
/*
* In this function, newly allocated vm_struct has VM_UNINITIALIZED
@@ -2218,16 +2218,19 @@ int remap_vmalloc_range(struct vm_area_s
EXPORT_SYMBOL(remap_vmalloc_range);
/*
- * Implement a stub for vmalloc_sync_all() if the architecture chose not to
- * have one.
+ * Implement stubs for vmalloc_sync_[un]mappings () if the architecture chose
+ * not to have one.
*
* The purpose of this function is to make sure the vmalloc area
* mappings are identical in all page-tables in the system.
*/
-void __weak vmalloc_sync_all(void)
+void __weak vmalloc_sync_mappings(void)
{
}
+void __weak vmalloc_sync_unmappings(void)
+{
+}
static int f(pte_t *pte, pgtable_t table, unsigned long addr, void *data)
{
next prev parent reply other threads:[~2020-04-01 16:32 UTC|newest]
Thread overview: 97+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-04-01 16:16 [PATCH 4.4 00/91] 4.4.218-rc1 review Greg Kroah-Hartman
2020-04-01 16:16 ` [PATCH 4.4 01/91] spi: qup: call spi_qup_pm_resume_runtime before suspending Greg Kroah-Hartman
2020-04-01 16:16 ` [PATCH 4.4 02/91] powerpc: Include .BTF section Greg Kroah-Hartman
2020-04-01 16:16 ` [PATCH 4.4 03/91] ARM: dts: dra7: Add "dma-ranges" property to PCIe RC DT nodes Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 04/91] spi/zynqmp: remove entry that causes a cs glitch Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 05/91] drm/exynos: dsi: propagate error value and silence meaningless warning Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 06/91] drm/exynos: dsi: fix workaround for the legacy clock name Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 07/91] altera-stapl: altera_get_note: prevent write beyond end of key Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 08/91] USB: Disable LPM on WD19s Realtek Hub Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 09/91] usb: quirks: add NO_LPM quirk for RTL8153 based ethernet adapters Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 10/91] USB: serial: option: add ME910G1 ECM composition 0x110b Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 11/91] usb: host: xhci-plat: add a shutdown Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 12/91] USB: serial: pl2303: add device-id for HP LD381 Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 13/91] ALSA: line6: Fix endless MIDI read loop Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 14/91] ALSA: seq: virmidi: Fix running status after receiving sysex Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 15/91] ALSA: seq: oss: " Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 16/91] ALSA: pcm: oss: Avoid plugin buffer overflow Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 17/91] ALSA: pcm: oss: Remove WARNING from snd_pcm_plug_alloc() checks Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 18/91] staging: rtl8188eu: Add device id for MERCUSYS MW150US v2 Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 19/91] staging/speakup: fix get_word non-space look-ahead Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 20/91] intel_th: Fix user-visible error codes Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 21/91] rtc: max8907: add missing select REGMAP_IRQ Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 22/91] memcg: fix NULL pointer dereference in __mem_cgroup_usage_unregister_event Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 23/91] mm: slub: be more careful about the double cmpxchg of freelist Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 24/91] mm, slub: prevent kmalloc_node crashes and memory leaks Greg Kroah-Hartman
2020-04-01 16:17 ` Greg Kroah-Hartman [this message]
2020-04-01 16:17 ` [PATCH 4.4 26/91] USB: cdc-acm: fix close_delay and closing_wait units in TIOCSSERIAL Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 27/91] USB: cdc-acm: fix rounding error " Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 28/91] kbuild: Disable -Wpointer-to-enum-cast Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 29/91] futex: Fix inode life-time issue Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 30/91] futex: Unbreak futex hashing Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 31/91] ALSA: hda/realtek: Fix pop noise on ALC225 Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 32/91] arm64: smp: fix smp_send_stop() behaviour Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 33/91] Revert "drm/dp_mst: Skip validating ports during destruction, just ref" Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 34/91] hsr: fix general protection fault in hsr_addr_is_self() Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 35/91] net: dsa: Fix duplicate frames flooded by learning Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 36/91] net_sched: cls_route: remove the right filter from hashtable Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 37/91] net_sched: keep alloc_hash updated after hash allocation Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 38/91] NFC: fdp: Fix a signedness bug in fdp_nci_send_patch() Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 39/91] slcan: not call free_netdev before rtnl_unlock in slcan_open Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 40/91] vxlan: check return value of gro_cells_init() Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 41/91] hsr: use rcu_read_lock() in hsr_get_node_{list/status}() Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 42/91] hsr: add restart routine into hsr_get_node_list() Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 43/91] hsr: set .netnsok flag Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 44/91] vhost: Check docket sk_family instead of call getname Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 45/91] IB/ipoib: Do not warn if IPoIB debugfs doesnt exist Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 46/91] uapi glibc compat: fix outer guard of net device flags enum Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 47/91] KVM: VMX: Do not allow reexecute_instruction() when skipping MMIO instr Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 48/91] drivers/hwspinlock: use correct radix tree API Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 49/91] net: ipv4: dont let PMTU updates increase route MTU Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 50/91] cpupower: avoid multiple definition with gcc -fno-common Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 51/91] dt-bindings: net: FMan erratum A050385 Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 52/91] scsi: ipr: Fix softlockup when rescanning devices in petitboot Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 53/91] mac80211: Do not send mesh HWMP PREQ if HWMP is disabled Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 54/91] sxgbe: Fix off by one in samsung driver strncpy size arg Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 55/91] i2c: hix5hd2: add missed clk_disable_unprepare in remove Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 56/91] perf probe: Do not depend on dwfl_module_addrsym() Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 57/91] scripts/dtc: Remove redundant YYLOC global declaration Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 58/91] scsi: sd: Fix optimal I/O size for devices that change reported values Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 59/91] mac80211: mark station unauthorized before key removal Greg Kroah-Hartman
2020-04-02 14:13 ` Ben Hutchings
2020-04-01 16:17 ` [PATCH 4.4 60/91] genirq: Fix reference leaks on irq affinity notifiers Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 61/91] vti[6]: fix packet tx through bpf_redirect() in XinY cases Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 62/91] xfrm: fix uctx len check in verify_sec_ctx_len Greg Kroah-Hartman
2020-04-01 16:17 ` [PATCH 4.4 63/91] xfrm: add the missing verify_sec_ctx_len check in xfrm_add_acquire Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 64/91] xfrm: policy: Fix doulbe free in xfrm_policy_timer Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 65/91] vti6: Fix memory leak of skb if input policy check fails Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 66/91] tools: Let O= makes handle a relative path with -C option Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 67/91] USB: serial: option: add support for ASKEY WWHC050 Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 68/91] USB: serial: option: add BroadMobi BM806U Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 69/91] USB: serial: option: add Wistron Neweb D19Q1 Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 70/91] USB: cdc-acm: restore capability check order Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 71/91] USB: serial: io_edgeport: fix slab-out-of-bounds read in edge_interrupt_callback Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 72/91] usb: musb: fix crash with highmen PIO and usbmon Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 73/91] media: flexcop-usb: fix endpoint sanity check Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 74/91] media: usbtv: fix control-message timeouts Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 75/91] staging: rtl8188eu: Add ASUS USB-N10 Nano B1 to device table Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 76/91] staging: wlan-ng: fix use-after-free Read in hfa384x_usbin_callback Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 77/91] libfs: fix infoleak in simple_attr_read() Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 78/91] media: ov519: add missing endpoint sanity checks Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 79/91] media: dib0700: fix rc endpoint lookup Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 80/91] media: stv06xx: add missing descriptor sanity checks Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 81/91] media: xirlink_cit: " Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 82/91] vt: selection, introduce vc_is_sel Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 83/91] vt: ioctl, switch VT_IS_IN_USE and VT_BUSY to inlines Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 84/91] vt: switch vt_dont_switch to bool Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 85/91] vt: vt_ioctl: remove unnecessary console allocation checks Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 86/91] vt: vt_ioctl: fix VT_DISALLOCATE freeing in-use virtual console Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 87/91] locking/atomic, kref: Add kref_read() Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 88/91] vt: vt_ioctl: fix use-after-free in vt_in_use() Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 89/91] bpf: Explicitly memset the bpf_attr structure Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 90/91] net: ks8851-ml: Fix IO operations, again Greg Kroah-Hartman
2020-04-01 16:18 ` [PATCH 4.4 91/91] perf map: Fix off by one in strncpy() size argument Greg Kroah-Hartman
2020-04-01 20:18 ` [PATCH 4.4 00/91] 4.4.218-rc1 review Chris Paterson
2020-04-02 0:10 ` Guenter Roeck
2020-04-02 7:09 ` Jon Hunter
2020-04-02 7:32 ` Naresh Kamboju
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200401161521.679075629@linuxfoundation.org \
--to=gregkh@linuxfoundation.org \
--cc=akpm@linux-foundation.org \
--cc=bp@suse.de \
--cc=dave.hansen@linux.intel.com \
--cc=jroedel@suse.de \
--cc=linux-kernel@vger.kernel.org \
--cc=luto@kernel.org \
--cc=mingo@redhat.com \
--cc=oliver.sang@intel.com \
--cc=peterz@infradead.org \
--cc=rafael.j.wysocki@intel.com \
--cc=shile.zhang@linux.alibaba.com \
--cc=stable@vger.kernel.org \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).