* [PATCH v10 1/3] mm: Shuffle initial free memory to improve memory-side-cache utilization
2019-02-01 5:15 [PATCH v10 0/3] mm: Randomize free memory Dan Williams
@ 2019-02-01 5:15 ` Dan Williams
2019-02-05 22:04 ` Andrew Morton
2019-05-31 7:33 ` Vlastimil Babka
2019-02-01 5:15 ` [PATCH v10 2/3] mm: Move buddy list manipulations into helpers Dan Williams
2019-02-01 5:15 ` [PATCH v10 3/3] mm: Maintain randomization of page free lists Dan Williams
2 siblings, 2 replies; 12+ messages in thread
From: Dan Williams @ 2019-02-01 5:15 UTC (permalink / raw)
To: akpm
Cc: Dave Hansen, Michal Hocko, Kees Cook, linux-mm, linux-kernel,
keith.busch
Randomization of the page allocator improves the average utilization of
a direct-mapped memory-side-cache. Memory side caching is a platform
capability that Linux has been previously exposed to in HPC
(high-performance computing) environments on specialty platforms. In
that instance it was a smaller pool of high-bandwidth-memory relative to
higher-capacity / lower-bandwidth DRAM. Now, this capability is going to
be found on general purpose server platforms where DRAM is a cache in
front of higher latency persistent memory [1].
Robert offered an explanation of the state of the art of Linux
interactions with memory-side-caches [2], and I copy it here:
It's been a problem in the HPC space:
http://www.nersc.gov/research-and-development/knl-cache-mode-performance-coe/
A kernel module called zonesort is available to try to help:
https://software.intel.com/en-us/articles/xeon-phi-software
and this abandoned patch series proposed that for the kernel:
https://lkml.kernel.org/r/20170823100205.17311-1-lukasz.daniluk@intel.com
Dan's patch series doesn't attempt to ensure buffers won't conflict, but
also reduces the chance that the buffers will. This will make performance
more consistent, albeit slower than "optimal" (which is near impossible
to attain in a general-purpose kernel). That's better than forcing
users to deploy remedies like:
"To eliminate this gradual degradation, we have added a Stream
measurement to the Node Health Check that follows each job;
nodes are rebooted whenever their measured memory bandwidth
falls below 300 GB/s."
A replacement for zonesort was merged upstream in commit cc9aec03e58f
"x86/numa_emulation: Introduce uniform split capability". With this
numa_emulation capability, memory can be split into cache sized
("near-memory" sized) numa nodes. A bind operation to such a node, and
disabling workloads on other nodes, enables full cache performance.
However, once the workload exceeds the cache size then cache conflicts
are unavoidable. While HPC environments might be able to tolerate
time-scheduling of cache sized workloads, for general purpose server
platforms, the oversubscribed cache case will be the common case.
The worst case scenario is that a server system owner benchmarks a
workload at boot with an un-contended cache only to see that performance
degrade over time, even below the average cache performance due to
excessive conflicts. Randomization clips the peaks and fills in the
valleys of cache utilization to yield steady average performance.
Here are some performance impact details of the patches:
1/ An Intel internal synthetic memory bandwidth measurement tool, saw a
3X speedup in a contrived case that tries to force cache conflicts. The
contrived cased used the numa_emulation capability to force an instance
of the benchmark to be run in two of the near-memory sized numa nodes.
If both instances were placed on the same emulated they would fit and
cause zero conflicts. While on separate emulated nodes without
randomization they underutilized the cache and conflicted unnecessarily
due to the in-order allocation per node.
2/ A well known Java server application benchmark was run with a heap
size that exceeded cache size by 3X. The cache conflict rate was 8% for
the first run and degraded to 21% after page allocator aging. With
randomization enabled the rate levelled out at 11%.
3/ A MongoDB workload did not observe measurable difference in
cache-conflict rates, but the overall throughput dropped by 7% with
randomization in one case.
4/ Mel Gorman ran his suite of performance workloads with randomization
enabled on platforms without a memory-side-cache and saw a mix of some
improvements and some losses [3].
While there is potentially significant improvement for applications that
depend on low latency access across a wide working-set, the performance
may be negligible to negative for other workloads. For this reason the
shuffle capability defaults to off unless a direct-mapped
memory-side-cache is detected. Even then, the page_alloc.shuffle=0
parameter can be specified to disable the randomization on those
systems.
Outside of memory-side-cache utilization concerns there is potentially
security benefit from randomization. Some data exfiltration and
return-oriented-programming attacks rely on the ability to infer the
location of sensitive data objects. The kernel page allocator,
especially early in system boot, has predictable first-in-first out
behavior for physical pages. Pages are freed in physical address order
when first onlined.
Quoting Kees:
"While we already have a base-address randomization
(CONFIG_RANDOMIZE_MEMORY), attacks against the same hardware and
memory layouts would certainly be using the predictability of
allocation ordering (i.e. for attacks where the base address isn't
important: only the relative positions between allocated memory).
This is common in lots of heap-style attacks. They try to gain
control over ordering by spraying allocations, etc.
I'd really like to see this because it gives us something similar
to CONFIG_SLAB_FREELIST_RANDOM but for the page allocator."
While SLAB_FREELIST_RANDOM reduces the predictability of some local slab
caches it leaves vast bulk of memory to be predictably in order
allocated. However, it should be noted, the concrete security benefits
are hard to quantify, and no known CVE is mitigated by this
randomization.
Introduce shuffle_free_memory(), and its helper shuffle_zone(), to
perform a Fisher-Yates shuffle of the page allocator 'free_area' lists
when they are initially populated with free memory at boot and at
hotplug time. Do this based on either the presence of a
page_alloc.shuffle=Y command line parameter, or autodetection of a
memory-side-cache (to be added in a follow-on patch).
The shuffling is done in terms of CONFIG_SHUFFLE_PAGE_ORDER sized free
pages where the default CONFIG_SHUFFLE_PAGE_ORDER is MAX_ORDER-1 i.e.
10, 4MB this trades off randomization granularity for time spent
shuffling. MAX_ORDER-1 was chosen to be minimally invasive to the page
allocator while still showing memory-side cache behavior improvements,
and the expectation that the security implications of finer granularity
randomization is mitigated by CONFIG_SLAB_FREELIST_RANDOM. The
performance impact of the shuffling appears to be in the noise compared
to other memory initialization work.
This initial randomization can be undone over time so a follow-on patch
is introduced to inject entropy on page free decisions. It is reasonable
to ask if the page free entropy is sufficient, but it is not enough due
to the in-order initial freeing of pages. At the start of that process
putting page1 in front or behind page0 still keeps them close together,
page2 is still near page1 and has a high chance of being adjacent. As
more pages are added ordering diversity improves, but there is still
high page locality for the low address pages and this leads to no
significant impact to the cache conflict rate.
[1]: https://itpeernetwork.intel.com/intel-optane-dc-persistent-memory-operating-modes/
[2]: https://lkml.kernel.org/r/AT5PR8401MB1169D656C8B5E121752FC0F8AB120@AT5PR8401MB1169.NAMPRD84.PROD.OUTLOOK.COM
[3]: https://lkml.org/lkml/2018/10/12/309
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
Documentation/admin-guide/kernel-parameters.txt | 10 +
include/linux/list.h | 17 ++
include/linux/mmzone.h | 1
init/Kconfig | 23 +++
mm/Makefile | 7 +
mm/memory_hotplug.c | 3
mm/page_alloc.c | 6 +
mm/shuffle.c | 170 +++++++++++++++++++++++
mm/shuffle.h | 52 +++++++
9 files changed, 287 insertions(+), 2 deletions(-)
create mode 100644 mm/shuffle.c
create mode 100644 mm/shuffle.h
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index b799bcf67d7b..abfd4a325354 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -3080,6 +3080,16 @@
This will also cause panics on machine check exceptions.
Useful together with panic=30 to trigger a reboot.
+ page_alloc.shuffle=
+ [KNL] Boolean flag to control whether the page allocator
+ should randomize its free lists. The randomization may
+ be automatically enabled if the kernel detects it is
+ running on a platform with a direct-mapped memory-side
+ cache, and this parameter can be used to
+ override/disable that behavior. The state of the flag
+ can be read from sysfs at:
+ /sys/module/page_alloc/parameters/shuffle.
+
page_owner= [KNL] Boot-time page_owner enabling option.
Storage of the information about who allocated
each page is disabled in default. With this switch,
diff --git a/include/linux/list.h b/include/linux/list.h
index edb7628e46ed..3dfb8953f241 100644
--- a/include/linux/list.h
+++ b/include/linux/list.h
@@ -150,6 +150,23 @@ static inline void list_replace_init(struct list_head *old,
INIT_LIST_HEAD(old);
}
+/**
+ * list_swap - replace entry1 with entry2 and re-add entry1 at entry2's position
+ * @entry1: the location to place entry2
+ * @entry2: the location to place entry1
+ */
+static inline void list_swap(struct list_head *entry1,
+ struct list_head *entry2)
+{
+ struct list_head *pos = entry2->prev;
+
+ list_del(entry2);
+ list_replace(entry1, entry2);
+ if (pos == entry1)
+ pos = entry2;
+ list_add(entry1, pos);
+}
+
/**
* list_del_init - deletes entry from list and reinitialize it.
* @entry: the element to delete from the list.
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index cc4a507d7ca4..c151c87a728a 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -1272,6 +1272,7 @@ void sparse_init(void);
#else
#define sparse_init() do {} while (0)
#define sparse_index_init(_sec, _nid) do {} while (0)
+#define pfn_present pfn_valid
#endif /* CONFIG_SPARSEMEM */
/*
diff --git a/init/Kconfig b/init/Kconfig
index d47cb77a220e..cfa199f3e9be 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1714,6 +1714,29 @@ config SLAB_FREELIST_HARDENED
sacrifies to harden the kernel slab allocator against common
freelist exploit methods.
+config SHUFFLE_PAGE_ALLOCATOR
+ bool "Page allocator randomization"
+ default SLAB_FREELIST_RANDOM && ACPI_NUMA
+ help
+ Randomization of the page allocator improves the average
+ utilization of a direct-mapped memory-side-cache. See section
+ 5.2.27 Heterogeneous Memory Attribute Table (HMAT) in the ACPI
+ 6.2a specification for an example of how a platform advertises
+ the presence of a memory-side-cache. There are also incidental
+ security benefits as it reduces the predictability of page
+ allocations to compliment SLAB_FREELIST_RANDOM, but the
+ default granularity of shuffling on 4MB (MAX_ORDER) pages is
+ selected based on cache utilization benefits.
+
+ While the randomization improves cache utilization it may
+ negatively impact workloads on platforms without a cache. For
+ this reason, by default, the randomization is enabled only
+ after runtime detection of a direct-mapped memory-side-cache.
+ Otherwise, the randomization may be force enabled with the
+ 'page_alloc.shuffle' kernel command line parameter.
+
+ Say Y if unsure.
+
config SLUB_CPU_PARTIAL
default y
depends on SLUB && SMP
diff --git a/mm/Makefile b/mm/Makefile
index d210cc9d6f80..ac5e5ba78874 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -33,7 +33,7 @@ mmu-$(CONFIG_MMU) += process_vm_access.o
endif
obj-y := filemap.o mempool.o oom_kill.o fadvise.o \
- maccess.o page_alloc.o page-writeback.o \
+ maccess.o page-writeback.o \
readahead.o swap.o truncate.o vmscan.o shmem.o \
util.o mmzone.o vmstat.o backing-dev.o \
mm_init.o mmu_context.o percpu.o slab_common.o \
@@ -41,6 +41,11 @@ obj-y := filemap.o mempool.o oom_kill.o fadvise.o \
interval_tree.o list_lru.o workingset.o \
debug.o $(mmu-y)
+# Give 'page_alloc' its own module-parameter namespace
+page-alloc-y := page_alloc.o
+page-alloc-$(CONFIG_SHUFFLE_PAGE_ALLOCATOR) += shuffle.o
+
+obj-y += page-alloc.o
obj-y += init-mm.o
obj-y += memblock.o
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index b9a667d36c55..0bb10e132f95 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -39,6 +39,7 @@
#include <asm/tlbflush.h>
#include "internal.h"
+#include "shuffle.h"
/*
* online_page_callback contains pointer to current page onlining function.
@@ -895,6 +896,8 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ
zone->zone_pgdat->node_present_pages += onlined_pages;
pgdat_resize_unlock(zone->zone_pgdat, &flags);
+ shuffle_zone(zone);
+
if (onlined_pages) {
node_states_set_node(nid, &arg);
if (need_zonelists_rebuild)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index cde5dac6229a..14ef39445544 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -72,6 +72,7 @@
#include <asm/tlbflush.h>
#include <asm/div64.h>
#include "internal.h"
+#include "shuffle.h"
/* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */
static DEFINE_MUTEX(pcp_batch_high_lock);
@@ -1752,9 +1753,9 @@ _deferred_grow_zone(struct zone *zone, unsigned int order)
void __init page_alloc_init_late(void)
{
struct zone *zone;
+ int nid;
#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
- int nid;
/* There will be num_node_state(N_MEMORY) threads */
atomic_set(&pgdat_init_n_undone, num_node_state(N_MEMORY));
@@ -1779,6 +1780,9 @@ void __init page_alloc_init_late(void)
memblock_discard();
#endif
+ for_each_node_state(nid, N_MEMORY)
+ shuffle_free_memory(NODE_DATA(nid));
+
for_each_populated_zone(zone)
set_zone_contiguous(zone);
}
diff --git a/mm/shuffle.c b/mm/shuffle.c
new file mode 100644
index 000000000000..8badf4f0a852
--- /dev/null
+++ b/mm/shuffle.c
@@ -0,0 +1,170 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+
+#include <linux/mm.h>
+#include <linux/init.h>
+#include <linux/mmzone.h>
+#include <linux/random.h>
+#include <linux/moduleparam.h>
+#include "internal.h"
+#include "shuffle.h"
+
+DEFINE_STATIC_KEY_FALSE(page_alloc_shuffle_key);
+static unsigned long shuffle_state __ro_after_init;
+
+/*
+ * Depending on the architecture, module parameter parsing may run
+ * before, or after the cache detection. SHUFFLE_FORCE_DISABLE prevents,
+ * or reverts the enabling of the shuffle implementation. SHUFFLE_ENABLE
+ * attempts to turn on the implementation, but aborts if it finds
+ * SHUFFLE_FORCE_DISABLE already set.
+ */
+__meminit void page_alloc_shuffle(enum mm_shuffle_ctl ctl)
+{
+ if (ctl == SHUFFLE_FORCE_DISABLE)
+ set_bit(SHUFFLE_FORCE_DISABLE, &shuffle_state);
+
+ if (test_bit(SHUFFLE_FORCE_DISABLE, &shuffle_state)) {
+ if (test_and_clear_bit(SHUFFLE_ENABLE, &shuffle_state))
+ static_branch_disable(&page_alloc_shuffle_key);
+ } else if (ctl == SHUFFLE_ENABLE
+ && !test_and_set_bit(SHUFFLE_ENABLE, &shuffle_state))
+ static_branch_enable(&page_alloc_shuffle_key);
+}
+
+static bool shuffle_param;
+extern int shuffle_show(char *buffer, const struct kernel_param *kp)
+{
+ return sprintf(buffer, "%c\n", test_bit(SHUFFLE_ENABLE, &shuffle_state)
+ ? 'Y' : 'N');
+}
+module_param_call(shuffle, NULL, shuffle_show, &shuffle_param, 0400);
+
+/*
+ * For two pages to be swapped in the shuffle, they must be free (on a
+ * 'free_area' lru), have the same order, and have the same migratetype.
+ */
+static struct page * __meminit shuffle_valid_page(unsigned long pfn, int order)
+{
+ struct page *page;
+
+ /*
+ * Given we're dealing with randomly selected pfns in a zone we
+ * need to ask questions like...
+ */
+
+ /* ...is the pfn even in the memmap? */
+ if (!pfn_valid_within(pfn))
+ return NULL;
+
+ /* ...is the pfn in a present section or a hole? */
+ if (!pfn_present(pfn))
+ return NULL;
+
+ /* ...is the page free and currently on a free_area list? */
+ page = pfn_to_page(pfn);
+ if (!PageBuddy(page))
+ return NULL;
+
+ /*
+ * ...is the page on the same list as the page we will
+ * shuffle it with?
+ */
+ if (page_order(page) != order)
+ return NULL;
+
+ return page;
+}
+
+/*
+ * Fisher-Yates shuffle the freelist which prescribes iterating through an
+ * array, pfns in this case, and randomly swapping each entry with another in
+ * the span, end_pfn - start_pfn.
+ *
+ * To keep the implementation simple it does not attempt to correct for sources
+ * of bias in the distribution, like modulo bias or pseudo-random number
+ * generator bias. I.e. the expectation is that this shuffling raises the bar
+ * for attacks that exploit the predictability of page allocations, but need not
+ * be a perfect shuffle.
+ */
+#define SHUFFLE_RETRY 10
+void __meminit __shuffle_zone(struct zone *z)
+{
+ unsigned long i, flags;
+ unsigned long start_pfn = z->zone_start_pfn;
+ unsigned long end_pfn = zone_end_pfn(z);
+ const int order = SHUFFLE_ORDER;
+ const int order_pages = 1 << order;
+
+ spin_lock_irqsave(&z->lock, flags);
+ start_pfn = ALIGN(start_pfn, order_pages);
+ for (i = start_pfn; i < end_pfn; i += order_pages) {
+ unsigned long j;
+ int migratetype, retry;
+ struct page *page_i, *page_j;
+
+ /*
+ * We expect page_i, in the sub-range of a zone being added
+ * (@start_pfn to @end_pfn), to more likely be valid compared to
+ * page_j randomly selected in the span @zone_start_pfn to
+ * @spanned_pages.
+ */
+ page_i = shuffle_valid_page(i, order);
+ if (!page_i)
+ continue;
+
+ for (retry = 0; retry < SHUFFLE_RETRY; retry++) {
+ /*
+ * Pick a random order aligned page in the zone span as
+ * a swap target. If the selected pfn is a hole, retry
+ * up to SHUFFLE_RETRY attempts find a random valid pfn
+ * in the zone.
+ */
+ j = z->zone_start_pfn +
+ ALIGN_DOWN(get_random_long() % z->spanned_pages,
+ order_pages);
+ page_j = shuffle_valid_page(j, order);
+ if (page_j && page_j != page_i)
+ break;
+ }
+ if (retry >= SHUFFLE_RETRY) {
+ pr_debug("%s: failed to swap %#lx\n", __func__, i);
+ continue;
+ }
+
+ /*
+ * Each migratetype corresponds to its own list, make sure the
+ * types match otherwise we're moving pages to lists where they
+ * do not belong.
+ */
+ migratetype = get_pageblock_migratetype(page_i);
+ if (get_pageblock_migratetype(page_j) != migratetype) {
+ pr_debug("%s: migratetype mismatch %#lx\n", __func__, i);
+ continue;
+ }
+
+ list_swap(&page_i->lru, &page_j->lru);
+
+ pr_debug("%s: swap: %#lx -> %#lx\n", __func__, i, j);
+
+ /* take it easy on the zone lock */
+ if ((i % (100 * order_pages)) == 0) {
+ spin_unlock_irqrestore(&z->lock, flags);
+ cond_resched();
+ spin_lock_irqsave(&z->lock, flags);
+ }
+ }
+ spin_unlock_irqrestore(&z->lock, flags);
+}
+
+/**
+ * shuffle_free_memory - reduce the predictability of the page allocator
+ * @pgdat: node page data
+ */
+void __meminit __shuffle_free_memory(pg_data_t *pgdat)
+{
+ struct zone *z;
+
+ for (z = pgdat->node_zones; z < pgdat->node_zones + MAX_NR_ZONES; z++)
+ shuffle_zone(z);
+}
diff --git a/mm/shuffle.h b/mm/shuffle.h
new file mode 100644
index 000000000000..644c8ee97b9e
--- /dev/null
+++ b/mm/shuffle.h
@@ -0,0 +1,52 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+#ifndef _MM_SHUFFLE_H
+#define _MM_SHUFFLE_H
+#include <linux/jump_label.h>
+
+/*
+ * SHUFFLE_ENABLE is called from the command line enabling path, or by
+ * platform-firmware enabling that indicates the presence of a
+ * direct-mapped memory-side-cache. SHUFFLE_FORCE_DISABLE is called from
+ * the command line path and overrides any previous or future
+ * SHUFFLE_ENABLE.
+ */
+enum mm_shuffle_ctl {
+ SHUFFLE_ENABLE,
+ SHUFFLE_FORCE_DISABLE,
+};
+
+#define SHUFFLE_ORDER (MAX_ORDER-1)
+
+#ifdef CONFIG_SHUFFLE_PAGE_ALLOCATOR
+DECLARE_STATIC_KEY_FALSE(page_alloc_shuffle_key);
+extern void page_alloc_shuffle(enum mm_shuffle_ctl ctl);
+extern void __shuffle_free_memory(pg_data_t *pgdat);
+static inline void shuffle_free_memory(pg_data_t *pgdat)
+{
+ if (!static_branch_unlikely(&page_alloc_shuffle_key))
+ return;
+ __shuffle_free_memory(pgdat);
+}
+
+extern void __shuffle_zone(struct zone *z);
+static inline void shuffle_zone(struct zone *z)
+{
+ if (!static_branch_unlikely(&page_alloc_shuffle_key))
+ return;
+ __shuffle_zone(z);
+}
+#else
+static inline void shuffle_free_memory(pg_data_t *pgdat)
+{
+}
+
+static inline void shuffle_zone(struct zone *z)
+{
+}
+
+static inline void page_alloc_shuffle(enum mm_shuffle_ctl ctl)
+{
+}
+#endif
+#endif /* _MM_SHUFFLE_H */
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PATCH v10 1/3] mm: Shuffle initial free memory to improve memory-side-cache utilization
2019-02-01 5:15 ` [PATCH v10 1/3] mm: Shuffle initial free memory to improve memory-side-cache utilization Dan Williams
@ 2019-02-05 22:04 ` Andrew Morton
2019-02-05 23:11 ` Kees Cook
2019-05-31 7:33 ` Vlastimil Babka
1 sibling, 1 reply; 12+ messages in thread
From: Andrew Morton @ 2019-02-05 22:04 UTC (permalink / raw)
To: Dan Williams
Cc: Dave Hansen, Michal Hocko, Kees Cook, linux-mm, linux-kernel,
keith.busch
On Thu, 31 Jan 2019 21:15:17 -0800 Dan Williams <dan.j.williams@intel.com> wrote:
> +config SHUFFLE_PAGE_ALLOCATOR
> + bool "Page allocator randomization"
> + default SLAB_FREELIST_RANDOM && ACPI_NUMA
> + help
SLAB_FREELIST_RANDOM is default n, so this patchset won't get much
runtime testing.
How about you cook up a (-mm only) patch which makes the kernel default
to SLAB_FREELIST_RANDOM=y, SHUFFLE_PAGE_ALLOCATOR=y (or whatever) to
ensure we get a decent amount of runtime testing? Then I can hold that
in -mm (and -next) until we get bored of it?
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v10 1/3] mm: Shuffle initial free memory to improve memory-side-cache utilization
2019-02-05 22:04 ` Andrew Morton
@ 2019-02-05 23:11 ` Kees Cook
2019-02-05 23:29 ` Matthew Wilcox
0 siblings, 1 reply; 12+ messages in thread
From: Kees Cook @ 2019-02-05 23:11 UTC (permalink / raw)
To: Andrew Morton
Cc: Dan Williams, Dave Hansen, Michal Hocko, Linux-MM, LKML, Keith Busch
On Tue, Feb 5, 2019 at 10:04 PM Andrew Morton <akpm@linux-foundation.org> wrote:
>
> On Thu, 31 Jan 2019 21:15:17 -0800 Dan Williams <dan.j.williams@intel.com> wrote:
>
> > +config SHUFFLE_PAGE_ALLOCATOR
> > + bool "Page allocator randomization"
> > + default SLAB_FREELIST_RANDOM && ACPI_NUMA
> > + help
>
> SLAB_FREELIST_RANDOM is default n, so this patchset won't get much
> runtime testing.
>
> How about you cook up a (-mm only) patch which makes the kernel default
> to SLAB_FREELIST_RANDOM=y, SHUFFLE_PAGE_ALLOCATOR=y (or whatever) to
> ensure we get a decent amount of runtime testing? Then I can hold that
> in -mm (and -next) until we get bored of it?
I love this plan. :)
FWIW, distros have enabled it by default for a while. Here's Ubuntu,
for example:
$ grep SLAB_FREELIST /boot/config-4.1*
/boot/config-4.15.0-45-generic:CONFIG_SLAB_FREELIST_RANDOM=y
/boot/config-4.15.0-45-generic:CONFIG_SLAB_FREELIST_HARDENED=y
/boot/config-4.18.0-13-generic:CONFIG_SLAB_FREELIST_RANDOM=y
/boot/config-4.18.0-13-generic:CONFIG_SLAB_FREELIST_HARDENED=y
/boot/config-4.18.0-14-generic:CONFIG_SLAB_FREELIST_RANDOM=y
/boot/config-4.18.0-14-generic:CONFIG_SLAB_FREELIST_HARDENED=y
and Fedora too:
$ curl -s 'https://git.kernel.org/pub/scm/linux/kernel/git/jwboyer/fedora.git/patch/fedora/configs/kernel-4.16.12-x86_64.config?h=f26'
| grep SLAB_FREELIST
+CONFIG_SLAB_FREELIST_RANDOM=y
+CONFIG_SLAB_FREELIST_HARDENED=y
--
Kees Cook
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v10 1/3] mm: Shuffle initial free memory to improve memory-side-cache utilization
2019-02-05 23:11 ` Kees Cook
@ 2019-02-05 23:29 ` Matthew Wilcox
0 siblings, 0 replies; 12+ messages in thread
From: Matthew Wilcox @ 2019-02-05 23:29 UTC (permalink / raw)
To: Kees Cook
Cc: Andrew Morton, Dan Williams, Dave Hansen, Michal Hocko, Linux-MM,
LKML, Keith Busch
On Tue, Feb 05, 2019 at 11:11:06PM +0000, Kees Cook wrote:
> FWIW, distros have enabled it by default for a while. Here's Ubuntu,
> for example:
>
> and Fedora too:
Also Debian:
$ grep SLAB_FREELIST /boot/config-4.19.0-1-amd64
CONFIG_SLAB_FREELIST_RANDOM=y
CONFIG_SLAB_FREELIST_HARDENED=y
linux (4.15.4-1) unstable; urgency=medium
* Switch to SLUB as kernel allocator. (Closes: #862718)
- Enable SLUB_DEBUG, SLAB_FREELIST_HARDENED except on armel/marvell.
(Closes: #883069)
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v10 1/3] mm: Shuffle initial free memory to improve memory-side-cache utilization
2019-02-01 5:15 ` [PATCH v10 1/3] mm: Shuffle initial free memory to improve memory-side-cache utilization Dan Williams
2019-02-05 22:04 ` Andrew Morton
@ 2019-05-31 7:33 ` Vlastimil Babka
2019-05-31 14:59 ` Dan Williams
1 sibling, 1 reply; 12+ messages in thread
From: Vlastimil Babka @ 2019-05-31 7:33 UTC (permalink / raw)
To: Dan Williams, akpm
Cc: Dave Hansen, Michal Hocko, Kees Cook, linux-mm, linux-kernel,
keith.busch
On 2/1/19 6:15 AM, Dan Williams wrote:
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -1714,6 +1714,29 @@ config SLAB_FREELIST_HARDENED
> sacrifies to harden the kernel slab allocator against common
> freelist exploit methods.
>
> +config SHUFFLE_PAGE_ALLOCATOR
> + bool "Page allocator randomization"
> + default SLAB_FREELIST_RANDOM && ACPI_NUMA
> + help
> + Randomization of the page allocator improves the average
> + utilization of a direct-mapped memory-side-cache. See section
> + 5.2.27 Heterogeneous Memory Attribute Table (HMAT) in the ACPI
> + 6.2a specification for an example of how a platform advertises
> + the presence of a memory-side-cache. There are also incidental
> + security benefits as it reduces the predictability of page
> + allocations to compliment SLAB_FREELIST_RANDOM, but the
> + default granularity of shuffling on 4MB (MAX_ORDER) pages is
> + selected based on cache utilization benefits.
> +
> + While the randomization improves cache utilization it may
> + negatively impact workloads on platforms without a cache. For
> + this reason, by default, the randomization is enabled only
> + after runtime detection of a direct-mapped memory-side-cache.
> + Otherwise, the randomization may be force enabled with the
> + 'page_alloc.shuffle' kernel command line parameter.
> +
> + Say Y if unsure.
It says "Say Y if unsure", yet if I run make oldconfig, the default is
N. Does that make sense?
> Page allocator randomization (SHUFFLE_PAGE_ALLOCATOR) [N/y/?] (NEW)
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v10 1/3] mm: Shuffle initial free memory to improve memory-side-cache utilization
2019-05-31 7:33 ` Vlastimil Babka
@ 2019-05-31 14:59 ` Dan Williams
0 siblings, 0 replies; 12+ messages in thread
From: Dan Williams @ 2019-05-31 14:59 UTC (permalink / raw)
To: Vlastimil Babka
Cc: Andrew Morton, Dave Hansen, Michal Hocko, Kees Cook, Linux MM,
Linux Kernel Mailing List, Keith Busch
On Fri, May 31, 2019 at 12:33 AM Vlastimil Babka <vbabka@suse.cz> wrote:
>
> On 2/1/19 6:15 AM, Dan Williams wrote:
> > --- a/init/Kconfig
> > +++ b/init/Kconfig
> > @@ -1714,6 +1714,29 @@ config SLAB_FREELIST_HARDENED
> > sacrifies to harden the kernel slab allocator against common
> > freelist exploit methods.
> >
> > +config SHUFFLE_PAGE_ALLOCATOR
> > + bool "Page allocator randomization"
> > + default SLAB_FREELIST_RANDOM && ACPI_NUMA
> > + help
> > + Randomization of the page allocator improves the average
> > + utilization of a direct-mapped memory-side-cache. See section
> > + 5.2.27 Heterogeneous Memory Attribute Table (HMAT) in the ACPI
> > + 6.2a specification for an example of how a platform advertises
> > + the presence of a memory-side-cache. There are also incidental
> > + security benefits as it reduces the predictability of page
> > + allocations to compliment SLAB_FREELIST_RANDOM, but the
> > + default granularity of shuffling on 4MB (MAX_ORDER) pages is
> > + selected based on cache utilization benefits.
> > +
> > + While the randomization improves cache utilization it may
> > + negatively impact workloads on platforms without a cache. For
> > + this reason, by default, the randomization is enabled only
> > + after runtime detection of a direct-mapped memory-side-cache.
> > + Otherwise, the randomization may be force enabled with the
> > + 'page_alloc.shuffle' kernel command line parameter.
> > +
> > + Say Y if unsure.
>
> It says "Say Y if unsure", yet if I run make oldconfig, the default is
> N. Does that make sense?
The default is due to the general policy of not forcing users into new
kernel functionality (i.e. the common Linus objection when a new
config symbol is default 'y') . However, if someone is actively
considering whether to enable it I think there's no harm in
recommending 'y' because the facility currently needs to be paired
with the page_alloc.shuffle=1 command line option.
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v10 2/3] mm: Move buddy list manipulations into helpers
2019-02-01 5:15 [PATCH v10 0/3] mm: Randomize free memory Dan Williams
2019-02-01 5:15 ` [PATCH v10 1/3] mm: Shuffle initial free memory to improve memory-side-cache utilization Dan Williams
@ 2019-02-01 5:15 ` Dan Williams
2019-02-19 15:24 ` Vlastimil Babka
2019-02-01 5:15 ` [PATCH v10 3/3] mm: Maintain randomization of page free lists Dan Williams
2 siblings, 1 reply; 12+ messages in thread
From: Dan Williams @ 2019-02-01 5:15 UTC (permalink / raw)
To: akpm; +Cc: Michal Hocko, Dave Hansen, linux-mm, linux-kernel, keith.busch
In preparation for runtime randomization of the zone lists, take all
(well, most of) the list_*() functions in the buddy allocator and put
them in helper functions. Provide a common control point for injecting
additional behavior when freeing pages.
Acked-by: Michal Hocko <mhocko@suse.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
include/linux/mm.h | 3 --
include/linux/mm_types.h | 3 ++
include/linux/mmzone.h | 46 ++++++++++++++++++++++++++++++
mm/compaction.c | 4 +--
mm/page_alloc.c | 70 ++++++++++++++++++----------------------------
5 files changed, 79 insertions(+), 47 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 80bb6408fe73..1621acd10f83 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -500,9 +500,6 @@ static inline void vma_set_anonymous(struct vm_area_struct *vma)
struct mmu_gather;
struct inode;
-#define page_private(page) ((page)->private)
-#define set_page_private(page, v) ((page)->private = (v))
-
#if !defined(__HAVE_ARCH_PTE_DEVMAP) || !defined(CONFIG_TRANSPARENT_HUGEPAGE)
static inline int pmd_devmap(pmd_t pmd)
{
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 2c471a2c43fa..1c7dc7ffa288 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -214,6 +214,9 @@ struct page {
#define PAGE_FRAG_CACHE_MAX_SIZE __ALIGN_MASK(32768, ~PAGE_MASK)
#define PAGE_FRAG_CACHE_MAX_ORDER get_order(PAGE_FRAG_CACHE_MAX_SIZE)
+#define page_private(page) ((page)->private)
+#define set_page_private(page, v) ((page)->private = (v))
+
struct page_frag_cache {
void * va;
#if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index c151c87a728a..2274e43933ae 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -18,6 +18,8 @@
#include <linux/pageblock-flags.h>
#include <linux/page-flags-layout.h>
#include <linux/atomic.h>
+#include <linux/mm_types.h>
+#include <linux/page-flags.h>
#include <asm/page.h>
/* Free memory management - zoned buddy allocator. */
@@ -98,6 +100,50 @@ struct free_area {
unsigned long nr_free;
};
+/* Used for pages not on another list */
+static inline void add_to_free_area(struct page *page, struct free_area *area,
+ int migratetype)
+{
+ list_add(&page->lru, &area->free_list[migratetype]);
+ area->nr_free++;
+}
+
+/* Used for pages not on another list */
+static inline void add_to_free_area_tail(struct page *page, struct free_area *area,
+ int migratetype)
+{
+ list_add_tail(&page->lru, &area->free_list[migratetype]);
+ area->nr_free++;
+}
+
+/* Used for pages which are on another list */
+static inline void move_to_free_area(struct page *page, struct free_area *area,
+ int migratetype)
+{
+ list_move(&page->lru, &area->free_list[migratetype]);
+}
+
+static inline struct page *get_page_from_free_area(struct free_area *area,
+ int migratetype)
+{
+ return list_first_entry_or_null(&area->free_list[migratetype],
+ struct page, lru);
+}
+
+static inline void del_page_from_free_area(struct page *page,
+ struct free_area *area, int migratetype)
+{
+ list_del(&page->lru);
+ __ClearPageBuddy(page);
+ set_page_private(page, 0);
+ area->nr_free--;
+}
+
+static inline bool free_area_empty(struct free_area *area, int migratetype)
+{
+ return list_empty(&area->free_list[migratetype]);
+}
+
struct pglist_data;
/*
diff --git a/mm/compaction.c b/mm/compaction.c
index ef29490b0f46..a22ac7ab65c5 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -1359,13 +1359,13 @@ static enum compact_result __compact_finished(struct zone *zone,
bool can_steal;
/* Job done if page is free of the right migratetype */
- if (!list_empty(&area->free_list[migratetype]))
+ if (!free_area_empty(area, migratetype))
return COMPACT_SUCCESS;
#ifdef CONFIG_CMA
/* MIGRATE_MOVABLE can fallback on MIGRATE_CMA */
if (migratetype == MIGRATE_MOVABLE &&
- !list_empty(&area->free_list[MIGRATE_CMA]))
+ !free_area_empty(area, MIGRATE_CMA))
return COMPACT_SUCCESS;
#endif
/*
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 14ef39445544..3fd0df403766 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -743,12 +743,6 @@ static inline void set_page_order(struct page *page, unsigned int order)
__SetPageBuddy(page);
}
-static inline void rmv_page_order(struct page *page)
-{
- __ClearPageBuddy(page);
- set_page_private(page, 0);
-}
-
/*
* This function checks whether a page is free && is the buddy
* we can coalesce a page and its buddy if
@@ -849,13 +843,11 @@ static inline void __free_one_page(struct page *page,
* Our buddy is free or it is CONFIG_DEBUG_PAGEALLOC guard page,
* merge with it and move up one order.
*/
- if (page_is_guard(buddy)) {
+ if (page_is_guard(buddy))
clear_page_guard(zone, buddy, order, migratetype);
- } else {
- list_del(&buddy->lru);
- zone->free_area[order].nr_free--;
- rmv_page_order(buddy);
- }
+ else
+ del_page_from_free_area(buddy, &zone->free_area[order],
+ migratetype);
combined_pfn = buddy_pfn & pfn;
page = page + (combined_pfn - pfn);
pfn = combined_pfn;
@@ -905,15 +897,13 @@ static inline void __free_one_page(struct page *page,
higher_buddy = higher_page + (buddy_pfn - combined_pfn);
if (pfn_valid_within(buddy_pfn) &&
page_is_buddy(higher_page, higher_buddy, order + 1)) {
- list_add_tail(&page->lru,
- &zone->free_area[order].free_list[migratetype]);
- goto out;
+ add_to_free_area_tail(page, &zone->free_area[order],
+ migratetype);
+ return;
}
}
- list_add(&page->lru, &zone->free_area[order].free_list[migratetype]);
-out:
- zone->free_area[order].nr_free++;
+ add_to_free_area(page, &zone->free_area[order], migratetype);
}
/*
@@ -1853,7 +1843,7 @@ static inline void expand(struct zone *zone, struct page *page,
if (set_page_guard(zone, &page[size], high, migratetype))
continue;
- list_add(&page[size].lru, &area->free_list[migratetype]);
+ add_to_free_area(&page[size], area, migratetype);
area->nr_free++;
set_page_order(&page[size], high);
}
@@ -1995,13 +1985,10 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order,
/* Find a page of the appropriate size in the preferred list */
for (current_order = order; current_order < MAX_ORDER; ++current_order) {
area = &(zone->free_area[current_order]);
- page = list_first_entry_or_null(&area->free_list[migratetype],
- struct page, lru);
+ page = get_page_from_free_area(area, migratetype);
if (!page)
continue;
- list_del(&page->lru);
- rmv_page_order(page);
- area->nr_free--;
+ del_page_from_free_area(page, area, migratetype);
expand(zone, page, order, current_order, area, migratetype);
set_pcppage_migratetype(page, migratetype);
return page;
@@ -2087,8 +2074,7 @@ static int move_freepages(struct zone *zone,
}
order = page_order(page);
- list_move(&page->lru,
- &zone->free_area[order].free_list[migratetype]);
+ move_to_free_area(page, &zone->free_area[order], migratetype);
page += 1 << order;
pages_moved += 1 << order;
}
@@ -2264,7 +2250,7 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page,
single_page:
area = &zone->free_area[current_order];
- list_move(&page->lru, &area->free_list[start_type]);
+ move_to_free_area(page, area, start_type);
}
/*
@@ -2288,7 +2274,7 @@ int find_suitable_fallback(struct free_area *area, unsigned int order,
if (fallback_mt == MIGRATE_TYPES)
break;
- if (list_empty(&area->free_list[fallback_mt]))
+ if (free_area_empty(area, fallback_mt))
continue;
if (can_steal_fallback(order, migratetype))
@@ -2375,9 +2361,7 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac,
for (order = 0; order < MAX_ORDER; order++) {
struct free_area *area = &(zone->free_area[order]);
- page = list_first_entry_or_null(
- &area->free_list[MIGRATE_HIGHATOMIC],
- struct page, lru);
+ page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC);
if (!page)
continue;
@@ -2500,8 +2484,7 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype,
VM_BUG_ON(current_order == MAX_ORDER);
do_steal:
- page = list_first_entry(&area->free_list[fallback_mt],
- struct page, lru);
+ page = get_page_from_free_area(area, fallback_mt);
steal_suitable_fallback(zone, page, alloc_flags, start_migratetype,
can_steal);
@@ -2938,6 +2921,7 @@ EXPORT_SYMBOL_GPL(split_page);
int __isolate_free_page(struct page *page, unsigned int order)
{
+ struct free_area *area = &page_zone(page)->free_area[order];
unsigned long watermark;
struct zone *zone;
int mt;
@@ -2962,9 +2946,8 @@ int __isolate_free_page(struct page *page, unsigned int order)
}
/* Remove page from free list */
- list_del(&page->lru);
- zone->free_area[order].nr_free--;
- rmv_page_order(page);
+
+ del_page_from_free_area(page, area, mt);
/*
* Set the pageblock if the isolated page is at least half of a
@@ -3266,13 +3249,13 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
continue;
for (mt = 0; mt < MIGRATE_PCPTYPES; mt++) {
- if (!list_empty(&area->free_list[mt]))
+ if (!free_area_empty(area, mt))
return true;
}
#ifdef CONFIG_CMA
if ((alloc_flags & ALLOC_CMA) &&
- !list_empty(&area->free_list[MIGRATE_CMA])) {
+ !free_area_empty(area, MIGRATE_CMA)) {
return true;
}
#endif
@@ -5174,7 +5157,7 @@ void show_free_areas(unsigned int filter, nodemask_t *nodemask)
types[order] = 0;
for (type = 0; type < MIGRATE_TYPES; type++) {
- if (!list_empty(&area->free_list[type]))
+ if (!free_area_empty(area, type))
types[order] |= 1 << type;
}
}
@@ -8319,6 +8302,9 @@ __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn)
spin_lock_irqsave(&zone->lock, flags);
pfn = start_pfn;
while (pfn < end_pfn) {
+ struct free_area *area;
+ int mt;
+
if (!pfn_valid(pfn)) {
pfn++;
continue;
@@ -8337,13 +8323,13 @@ __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn)
BUG_ON(page_count(page));
BUG_ON(!PageBuddy(page));
order = page_order(page);
+ area = &zone->free_area[order];
#ifdef CONFIG_DEBUG_VM
pr_info("remove from free list %lx %d %lx\n",
pfn, 1 << order, end_pfn);
#endif
- list_del(&page->lru);
- rmv_page_order(page);
- zone->free_area[order].nr_free--;
+ mt = get_pageblock_migratetype(page);
+ del_page_from_free_area(page, area, mt);
for (i = 0; i < (1 << order); i++)
SetPageReserved((page+i));
pfn += (1 << order);
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PATCH v10 2/3] mm: Move buddy list manipulations into helpers
2019-02-01 5:15 ` [PATCH v10 2/3] mm: Move buddy list manipulations into helpers Dan Williams
@ 2019-02-19 15:24 ` Vlastimil Babka
2019-02-19 17:21 ` Dan Williams
0 siblings, 1 reply; 12+ messages in thread
From: Vlastimil Babka @ 2019-02-19 15:24 UTC (permalink / raw)
To: Dan Williams, akpm
Cc: Michal Hocko, Dave Hansen, linux-mm, linux-kernel, keith.busch
On 2/1/19 6:15 AM, Dan Williams wrote:
> In preparation for runtime randomization of the zone lists, take all
> (well, most of) the list_*() functions in the buddy allocator and put
> them in helper functions. Provide a common control point for injecting
> additional behavior when freeing pages.
>
> Acked-by: Michal Hocko <mhocko@suse.com>
> Cc: Dave Hansen <dave.hansen@linux.intel.com>
> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Here's another fixlet to fold into mm-move-buddy-list-manipulations-into-helpers.patch
This time not critical.
----8<----
From 05aaff61f62f86e646c4a2581fe2ff63ff66a199 Mon Sep 17 00:00:00 2001
From: Vlastimil Babka <vbabka@suse.cz>
Date: Tue, 19 Feb 2019 16:20:33 +0100
Subject: [PATCH] mm: Move buddy list manipulations into helpers-fix2
del_page_from_free_area() migratetype parameter is unused, remove it.
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
---
include/linux/mmzone.h | 2 +-
mm/page_alloc.c | 14 ++++----------
2 files changed, 5 insertions(+), 11 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index da5321c747f8..2fd4247262e9 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -143,7 +143,7 @@ static inline struct page *get_page_from_free_area(struct free_area *area,
}
static inline void del_page_from_free_area(struct page *page,
- struct free_area *area, int migratetype)
+ struct free_area *area)
{
list_del(&page->lru);
__ClearPageBuddy(page);
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 37ed14ad0b59..d2b6d5245568 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -901,8 +901,7 @@ static inline void __free_one_page(struct page *page,
if (page_is_guard(buddy))
clear_page_guard(zone, buddy, order, migratetype);
else
- del_page_from_free_area(buddy, &zone->free_area[order],
- migratetype);
+ del_page_from_free_area(buddy, &zone->free_area[order]);
combined_pfn = buddy_pfn & pfn;
page = page + (combined_pfn - pfn);
pfn = combined_pfn;
@@ -2173,7 +2172,7 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order,
page = get_page_from_free_area(area, migratetype);
if (!page)
continue;
- del_page_from_free_area(page, area, migratetype);
+ del_page_from_free_area(page, area);
expand(zone, page, order, current_order, area, migratetype);
set_pcppage_migratetype(page, migratetype);
return page;
@@ -3144,7 +3143,7 @@ int __isolate_free_page(struct page *page, unsigned int order)
/* Remove page from free list */
- del_page_from_free_area(page, area, mt);
+ del_page_from_free_area(page, area);
/*
* Set the pageblock if the isolated page is at least half of a
@@ -8507,9 +8506,6 @@ __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn)
spin_lock_irqsave(&zone->lock, flags);
pfn = start_pfn;
while (pfn < end_pfn) {
- struct free_area *area;
- int mt;
-
if (!pfn_valid(pfn)) {
pfn++;
continue;
@@ -8528,13 +8524,11 @@ __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn)
BUG_ON(page_count(page));
BUG_ON(!PageBuddy(page));
order = page_order(page);
- area = &zone->free_area[order];
#ifdef CONFIG_DEBUG_VM
pr_info("remove from free list %lx %d %lx\n",
pfn, 1 << order, end_pfn);
#endif
- mt = get_pageblock_migratetype(page);
- del_page_from_free_area(page, area, mt);
+ del_page_from_free_area(page, &zone->free_area[order]);
for (i = 0; i < (1 << order); i++)
SetPageReserved((page+i));
pfn += (1 << order);
--
2.20.1
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PATCH v10 2/3] mm: Move buddy list manipulations into helpers
2019-02-19 15:24 ` Vlastimil Babka
@ 2019-02-19 17:21 ` Dan Williams
0 siblings, 0 replies; 12+ messages in thread
From: Dan Williams @ 2019-02-19 17:21 UTC (permalink / raw)
To: Vlastimil Babka
Cc: Andrew Morton, Michal Hocko, Dave Hansen, Linux MM,
Linux Kernel Mailing List, Keith Busch
On Tue, Feb 19, 2019 at 7:24 AM Vlastimil Babka <vbabka@suse.cz> wrote:
>
> On 2/1/19 6:15 AM, Dan Williams wrote:
> > In preparation for runtime randomization of the zone lists, take all
> > (well, most of) the list_*() functions in the buddy allocator and put
> > them in helper functions. Provide a common control point for injecting
> > additional behavior when freeing pages.
> >
> > Acked-by: Michal Hocko <mhocko@suse.com>
> > Cc: Dave Hansen <dave.hansen@linux.intel.com>
> > Signed-off-by: Dan Williams <dan.j.williams@intel.com>
>
> Here's another fixlet to fold into mm-move-buddy-list-manipulations-into-helpers.patch
> This time not critical.
>
> ----8<----
> From 05aaff61f62f86e646c4a2581fe2ff63ff66a199 Mon Sep 17 00:00:00 2001
> From: Vlastimil Babka <vbabka@suse.cz>
> Date: Tue, 19 Feb 2019 16:20:33 +0100
> Subject: [PATCH] mm: Move buddy list manipulations into helpers-fix2
>
> del_page_from_free_area() migratetype parameter is unused, remove it.
Looks good,
Acked-by: Dan Williams <dan.j.williams@intel.com>
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v10 3/3] mm: Maintain randomization of page free lists
2019-02-01 5:15 [PATCH v10 0/3] mm: Randomize free memory Dan Williams
2019-02-01 5:15 ` [PATCH v10 1/3] mm: Shuffle initial free memory to improve memory-side-cache utilization Dan Williams
2019-02-01 5:15 ` [PATCH v10 2/3] mm: Move buddy list manipulations into helpers Dan Williams
@ 2019-02-01 5:15 ` Dan Williams
2019-02-01 9:58 ` Michal Hocko
2 siblings, 1 reply; 12+ messages in thread
From: Dan Williams @ 2019-02-01 5:15 UTC (permalink / raw)
To: akpm
Cc: Michal Hocko, Dave Hansen, Kees Cook, linux-mm, linux-kernel,
keith.busch
When freeing a page with an order >= shuffle_page_order randomly select
the front or back of the list for insertion.
While the mm tries to defragment physical pages into huge pages this can
tend to make the page allocator more predictable over time. Inject the
front-back randomness to preserve the initial randomness established by
shuffle_free_memory() when the kernel was booted.
The overhead of this manipulation is constrained by only being applied
for MAX_ORDER sized pages by default.
Cc: Michal Hocko <mhocko@suse.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
include/linux/mmzone.h | 12 ++++++++++++
mm/page_alloc.c | 11 +++++++++--
mm/shuffle.c | 23 +++++++++++++++++++++++
mm/shuffle.h | 12 ++++++++++++
4 files changed, 56 insertions(+), 2 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 2274e43933ae..a3cb9a21196d 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -116,6 +116,18 @@ static inline void add_to_free_area_tail(struct page *page, struct free_area *ar
area->nr_free++;
}
+#ifdef CONFIG_SHUFFLE_PAGE_ALLOCATOR
+/* Used to preserve page allocation order entropy */
+void add_to_free_area_random(struct page *page, struct free_area *area,
+ int migratetype);
+#else
+static inline void add_to_free_area_random(struct page *page,
+ struct free_area *area, int migratetype)
+{
+ add_to_free_area(page, area, migratetype);
+}
+#endif
+
/* Used for pages which are on another list */
static inline void move_to_free_area(struct page *page, struct free_area *area,
int migratetype)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 3fd0df403766..2a0969e3b0eb 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -43,6 +43,7 @@
#include <linux/mempolicy.h>
#include <linux/memremap.h>
#include <linux/stop_machine.h>
+#include <linux/random.h>
#include <linux/sort.h>
#include <linux/pfn.h>
#include <linux/backing-dev.h>
@@ -889,7 +890,8 @@ static inline void __free_one_page(struct page *page,
* so it's less likely to be used soon and more likely to be merged
* as a higher order page
*/
- if ((order < MAX_ORDER-2) && pfn_valid_within(buddy_pfn)) {
+ if ((order < MAX_ORDER-2) && pfn_valid_within(buddy_pfn)
+ && !is_shuffle_order(order)) {
struct page *higher_page, *higher_buddy;
combined_pfn = buddy_pfn & pfn;
higher_page = page + (combined_pfn - pfn);
@@ -903,7 +905,12 @@ static inline void __free_one_page(struct page *page,
}
}
- add_to_free_area(page, &zone->free_area[order], migratetype);
+ if (is_shuffle_order(order))
+ add_to_free_area_random(page, &zone->free_area[order],
+ migratetype);
+ else
+ add_to_free_area(page, &zone->free_area[order], migratetype);
+
}
/*
diff --git a/mm/shuffle.c b/mm/shuffle.c
index 8badf4f0a852..19bbf3e37fb6 100644
--- a/mm/shuffle.c
+++ b/mm/shuffle.c
@@ -168,3 +168,26 @@ void __meminit __shuffle_free_memory(pg_data_t *pgdat)
for (z = pgdat->node_zones; z < pgdat->node_zones + MAX_NR_ZONES; z++)
shuffle_zone(z);
}
+
+void add_to_free_area_random(struct page *page, struct free_area *area,
+ int migratetype)
+{
+ static u64 rand;
+ static u8 rand_bits;
+
+ /*
+ * The lack of locking is deliberate. If 2 threads race to
+ * update the rand state it just adds to the entropy.
+ */
+ if (rand_bits == 0) {
+ rand_bits = 64;
+ rand = get_random_u64();
+ }
+
+ if (rand & 1)
+ add_to_free_area(page, area, migratetype);
+ else
+ add_to_free_area_tail(page, area, migratetype);
+ rand_bits--;
+ rand >>= 1;
+}
diff --git a/mm/shuffle.h b/mm/shuffle.h
index 644c8ee97b9e..fc1e327ae22d 100644
--- a/mm/shuffle.h
+++ b/mm/shuffle.h
@@ -36,6 +36,13 @@ static inline void shuffle_zone(struct zone *z)
return;
__shuffle_zone(z);
}
+
+static inline bool is_shuffle_order(int order)
+{
+ if (!static_branch_unlikely(&page_alloc_shuffle_key))
+ return false;
+ return order >= SHUFFLE_ORDER;
+}
#else
static inline void shuffle_free_memory(pg_data_t *pgdat)
{
@@ -48,5 +55,10 @@ static inline void shuffle_zone(struct zone *z)
static inline void page_alloc_shuffle(enum mm_shuffle_ctl ctl)
{
}
+
+static inline bool is_shuffle_order(int order)
+{
+ return false;
+}
#endif
#endif /* _MM_SHUFFLE_H */
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PATCH v10 3/3] mm: Maintain randomization of page free lists
2019-02-01 5:15 ` [PATCH v10 3/3] mm: Maintain randomization of page free lists Dan Williams
@ 2019-02-01 9:58 ` Michal Hocko
0 siblings, 0 replies; 12+ messages in thread
From: Michal Hocko @ 2019-02-01 9:58 UTC (permalink / raw)
To: Dan Williams
Cc: akpm, Dave Hansen, Kees Cook, linux-mm, linux-kernel, keith.busch
On Thu 31-01-19 21:15:27, Dan Williams wrote:
> When freeing a page with an order >= shuffle_page_order randomly select
> the front or back of the list for insertion.
>
> While the mm tries to defragment physical pages into huge pages this can
> tend to make the page allocator more predictable over time. Inject the
> front-back randomness to preserve the initial randomness established by
> shuffle_free_memory() when the kernel was booted.
>
> The overhead of this manipulation is constrained by only being applied
> for MAX_ORDER sized pages by default.
>
> Cc: Michal Hocko <mhocko@suse.com>
> Cc: Dave Hansen <dave.hansen@linux.intel.com>
> Reviewed-by: Kees Cook <keescook@chromium.org>
> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
> ---
> include/linux/mmzone.h | 12 ++++++++++++
> mm/page_alloc.c | 11 +++++++++--
> mm/shuffle.c | 23 +++++++++++++++++++++++
> mm/shuffle.h | 12 ++++++++++++
> 4 files changed, 56 insertions(+), 2 deletions(-)
>
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index 2274e43933ae..a3cb9a21196d 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -116,6 +116,18 @@ static inline void add_to_free_area_tail(struct page *page, struct free_area *ar
> area->nr_free++;
> }
>
> +#ifdef CONFIG_SHUFFLE_PAGE_ALLOCATOR
> +/* Used to preserve page allocation order entropy */
> +void add_to_free_area_random(struct page *page, struct free_area *area,
> + int migratetype);
> +#else
> +static inline void add_to_free_area_random(struct page *page,
> + struct free_area *area, int migratetype)
> +{
> + add_to_free_area(page, area, migratetype);
> +}
> +#endif
> +
> /* Used for pages which are on another list */
> static inline void move_to_free_area(struct page *page, struct free_area *area,
> int migratetype)
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 3fd0df403766..2a0969e3b0eb 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -43,6 +43,7 @@
> #include <linux/mempolicy.h>
> #include <linux/memremap.h>
> #include <linux/stop_machine.h>
> +#include <linux/random.h>
> #include <linux/sort.h>
> #include <linux/pfn.h>
> #include <linux/backing-dev.h>
> @@ -889,7 +890,8 @@ static inline void __free_one_page(struct page *page,
> * so it's less likely to be used soon and more likely to be merged
> * as a higher order page
> */
> - if ((order < MAX_ORDER-2) && pfn_valid_within(buddy_pfn)) {
> + if ((order < MAX_ORDER-2) && pfn_valid_within(buddy_pfn)
> + && !is_shuffle_order(order)) {
> struct page *higher_page, *higher_buddy;
> combined_pfn = buddy_pfn & pfn;
> higher_page = page + (combined_pfn - pfn);
> @@ -903,7 +905,12 @@ static inline void __free_one_page(struct page *page,
> }
> }
>
> - add_to_free_area(page, &zone->free_area[order], migratetype);
> + if (is_shuffle_order(order))
> + add_to_free_area_random(page, &zone->free_area[order],
> + migratetype);
> + else
> + add_to_free_area(page, &zone->free_area[order], migratetype);
> +
> }
>
> /*
> diff --git a/mm/shuffle.c b/mm/shuffle.c
> index 8badf4f0a852..19bbf3e37fb6 100644
> --- a/mm/shuffle.c
> +++ b/mm/shuffle.c
> @@ -168,3 +168,26 @@ void __meminit __shuffle_free_memory(pg_data_t *pgdat)
> for (z = pgdat->node_zones; z < pgdat->node_zones + MAX_NR_ZONES; z++)
> shuffle_zone(z);
> }
> +
> +void add_to_free_area_random(struct page *page, struct free_area *area,
> + int migratetype)
> +{
> + static u64 rand;
> + static u8 rand_bits;
> +
> + /*
> + * The lack of locking is deliberate. If 2 threads race to
> + * update the rand state it just adds to the entropy.
> + */
> + if (rand_bits == 0) {
> + rand_bits = 64;
> + rand = get_random_u64();
> + }
> +
> + if (rand & 1)
> + add_to_free_area(page, area, migratetype);
> + else
> + add_to_free_area_tail(page, area, migratetype);
> + rand_bits--;
> + rand >>= 1;
> +}
> diff --git a/mm/shuffle.h b/mm/shuffle.h
> index 644c8ee97b9e..fc1e327ae22d 100644
> --- a/mm/shuffle.h
> +++ b/mm/shuffle.h
> @@ -36,6 +36,13 @@ static inline void shuffle_zone(struct zone *z)
> return;
> __shuffle_zone(z);
> }
> +
> +static inline bool is_shuffle_order(int order)
> +{
> + if (!static_branch_unlikely(&page_alloc_shuffle_key))
> + return false;
> + return order >= SHUFFLE_ORDER;
> +}
> #else
> static inline void shuffle_free_memory(pg_data_t *pgdat)
> {
> @@ -48,5 +55,10 @@ static inline void shuffle_zone(struct zone *z)
> static inline void page_alloc_shuffle(enum mm_shuffle_ctl ctl)
> {
> }
> +
> +static inline bool is_shuffle_order(int order)
> +{
> + return false;
> +}
> #endif
> #endif /* _MM_SHUFFLE_H */
>
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 12+ messages in thread