* [RFC 01/43] mm: add PKRAM API stubs and Kconfig
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 02/43] mm: PKRAM: implement node load and save functions Anthony Yznaga
` (41 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Preserved-across-kexec memory or PKRAM is a method for saving memory
pages of the currently executing kernel and restoring them after kexec
boot into a new one. This can be utilized for preserving guest VM state,
large in-memory databases, process memory, etc. across reboot. While
DRAM-as-PMEM or actual persistent memory could be used to accomplish
these things, PKRAM provides the latency of DRAM with the flexibility
of dynamically determining the amount of memory to preserve.
The proposed API:
* Preserved memory is divided into nodes which can be saved or loaded
independently of each other. The nodes are identified by unique name
strings. A PKRAM node is created when save is initiated by calling
pkram_prepare_save(). A PKRAM node is removed when load is initiated by
calling pkram_prepare_load(). See below
* A node is further divided into objects. An object represents a
grouping of associated pages and any relevant metadata preserved
with them. For example, the pages and attributes of a file.
* For saving/loading data from a PKRAM node/object an instance of the
pkram_stream struct is used. The struct is initialized by calling
pkram_prepare_save() for saving data or pkram_prepare_load() for
loading data. After save (load) is complete, pkram_finish_save()
(pkram_finish_load()) must be called. If an error occurred during
save, the saved data and the PKRAM node may be freed by calling
pkram_discard_save() instead of pkram_finish_save().
* Both page data and byte data can separately be streamed to a PKRAM
object. pkram_save_page() and pkram_load_page() are used to stream
page data while pkram_write() and pkram_read() are used to stream byte
data.
A sequence of operations for saving/loading data from PKRAM would
look like:
* For saving data to PKRAM:
/* create a PKRAM node and do initial stream setup */
pkram_prepare_save()
/* create a PKRAM object associated with the PKRAM node and complete stream initialization */
pkram_prepare_save_obj()
/* save data to the node/object */
pkram_save_page()[,...] /* for page stream, or
pkram_write()[,...] * ... for byte stream */
pkram_finish_save_obj()
/* commit the save or discard and delete the node */
pkram_finish_save() /* on success, or
pkram_discard_save() * ... in case of error */
* For loading data from PKRAM:
/* remove a PKRAM node from the list and do initial stream setup */
pkram_prepare_load()
/* Remove a PKRAM object from the node and complete stream initializtion for loading data from it. */
pkram_prepare_load_obj()
/* load data from the node/object */
pkram_load_page()[,...] /* for page stream, or
pkram_read()[,...] * ... for byte stream */
/* free the object */
pkram_finish_load_obj()
/* free the node */
pkram_finish_load()
Originally-by: Vladimir Davydov <vdavydov.dev@gmail.com>
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/pkram.h | 32 ++++++++++
mm/Kconfig | 9 +++
mm/Makefile | 1 +
mm/pkram.c | 169 ++++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 211 insertions(+)
create mode 100644 include/linux/pkram.h
create mode 100644 mm/pkram.c
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
new file mode 100644
index 000000000000..4c4e13311ec8
--- /dev/null
+++ b/include/linux/pkram.h
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_PKRAM_H
+#define _LINUX_PKRAM_H
+
+#include <linux/gfp.h>
+#include <linux/types.h>
+#include <linux/mm_types.h>
+
+struct pkram_stream;
+
+#define PKRAM_NAME_MAX 256 /* including nul */
+
+int pkram_prepare_save(struct pkram_stream *ps, const char *name,
+ gfp_t gfp_mask);
+int pkram_prepare_save_obj(struct pkram_stream *ps);
+void pkram_finish_save(struct pkram_stream *ps);
+void pkram_finish_save_obj(struct pkram_stream *ps);
+void pkram_discard_save(struct pkram_stream *ps);
+
+int pkram_prepare_load(struct pkram_stream *ps, const char *name);
+int pkram_prepare_load_obj(struct pkram_stream *ps);
+void pkram_finish_load(struct pkram_stream *ps);
+void pkram_finish_load_obj(struct pkram_stream *ps);
+
+int pkram_save_page(struct pkram_stream *ps, struct page *page, short flags);
+struct page *pkram_load_page(struct pkram_stream *ps, unsigned long *index,
+ short *flags);
+
+ssize_t pkram_write(struct pkram_stream *ps, const void *buf, size_t count);
+size_t pkram_read(struct pkram_stream *ps, void *buf, size_t count);
+
+#endif /* _LINUX_PKRAM_H */
diff --git a/mm/Kconfig b/mm/Kconfig
index c1acc34c1c35..bddf20ecf6e1 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -867,4 +867,13 @@ config ARCH_HAS_HUGEPD
config MAPPING_DIRTY_HELPERS
bool
+config PKRAM
+ bool "Preserved-over-kexec memory storage"
+ default n
+ help
+ This option adds the kernel API that enables saving memory pages of
+ the currently executing kernel and restoring them after a kexec in
+ the newly booted one. This can be utilized for speeding up reboot by
+ leaving process memory and/or FS caches in-place.
+
endmenu
diff --git a/mm/Makefile b/mm/Makefile
index fccd3756b25f..59cd381194af 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -112,3 +112,4 @@ obj-$(CONFIG_MEMFD_CREATE) += memfd.o
obj-$(CONFIG_MAPPING_DIRTY_HELPERS) += mapping_dirty_helpers.o
obj-$(CONFIG_PTDUMP_CORE) += ptdump.o
obj-$(CONFIG_PAGE_REPORTING) += page_reporting.o
+obj-$(CONFIG_PKRAM) += pkram.o
diff --git a/mm/pkram.c b/mm/pkram.c
new file mode 100644
index 000000000000..d6f2f79d4852
--- /dev/null
+++ b/mm/pkram.c
@@ -0,0 +1,169 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/err.h>
+#include <linux/gfp.h>
+#include <linux/kernel.h>
+#include <linux/mm.h>
+#include <linux/pkram.h>
+#include <linux/types.h>
+
+/**
+ * Create a preserved memory node with name @name and initialize stream @ps
+ * for saving data to it.
+ *
+ * @gfp_mask specifies the memory allocation mask to be used when saving data.
+ *
+ * Returns 0 on success, -errno on failure.
+ *
+ * After the save has finished, pkram_finish_save() (or pkram_discard_save() in
+ * case of failure) is to be called.
+ */
+int pkram_prepare_save(struct pkram_stream *ps, const char *name, gfp_t gfp_mask)
+{
+ return -ENOSYS;
+}
+
+/**
+ * Create a preserved memory object and initialize stream @ps for saving data
+ * to it.
+ *
+ * Returns 0 on success, -errno on failure.
+ *
+ * After the save has finished, pkram_finish_save_obj() (or pkram_discard_save()
+ * in case of failure) is to be called.
+ */
+int pkram_prepare_save_obj(struct pkram_stream *ps)
+{
+ return -ENOSYS;
+}
+
+/**
+ * Commit the object started with pkram_prepare_save_obj() to preserved memory.
+ */
+void pkram_finish_save_obj(struct pkram_stream *ps)
+{
+ BUG();
+}
+
+/**
+ * Commit the save to preserved memory started with pkram_prepare_save().
+ * After the call, the stream may not be used any more.
+ */
+void pkram_finish_save(struct pkram_stream *ps)
+{
+ BUG();
+}
+
+/**
+ * Cancel the save to preserved memory started with pkram_prepare_save() and
+ * destroy the corresponding preserved memory node freeing any data already
+ * saved to it.
+ */
+void pkram_discard_save(struct pkram_stream *ps)
+{
+ BUG();
+}
+
+/**
+ * Remove the preserved memory node with name @name and initialize stream @ps
+ * for loading data from it.
+ *
+ * Returns 0 on success, -errno on failure.
+ *
+ * After the load has finished, pkram_finish_load() is to be called.
+ */
+int pkram_prepare_load(struct pkram_stream *ps, const char *name)
+{
+ return -ENOSYS;
+}
+
+/**
+ * Remove the next preserved memory object from the stream @ps and
+ * initialize stream @ps for loading data from it.
+ *
+ * Returns 0 on success, -errno on failure.
+ *
+ * After the load has finished, pkram_finish_load_obj() is to be called.
+ */
+int pkram_prepare_load_obj(struct pkram_stream *ps)
+{
+ return -ENOSYS;
+}
+
+/**
+ * Finish the load of a preserved memory object started with
+ * pkram_prepare_load_obj() freeing the object and any data that has not
+ * been loaded from it.
+ */
+void pkram_finish_load_obj(struct pkram_stream *ps)
+{
+ BUG();
+}
+
+/**
+ * Finish the load from preserved memory started with pkram_prepare_load()
+ * freeing the corresponding preserved memory node and any data that has
+ * not been loaded from it.
+ */
+void pkram_finish_load(struct pkram_stream *ps)
+{
+ BUG();
+}
+
+/**
+ * Save page @page to the preserved memory node and object associated with
+ * stream @ps. The stream must have been initialized with pkram_prepare_save()
+ * and pkram_prepare_save_obj().
+ *
+ * @flags specifies supplemental page state to be preserved.
+ *
+ * Returns 0 on success, -errno on failure.
+ */
+int pkram_save_page(struct pkram_stream *ps, struct page *page, short flags)
+{
+ return -ENOSYS;
+}
+
+/**
+ * Load the next page from the preserved memory node and object associated
+ * with stream @ps. The stream must have been initialized with
+ * pkram_prepare_load() and pkram_prepare_load_obj().
+ *
+ * If not NULL, @index is initialized with the preserved mapping offset of the
+ * page loaded.
+ * If not NULL, @flags is initialized with preserved supplemental state of the
+ * page loaded.
+ *
+ * Returns the page loaded or NULL if the node is empty.
+ *
+ * The page loaded has its refcount incremented.
+ */
+struct page *pkram_load_page(struct pkram_stream *ps, unsigned long *index, short *flags)
+{
+ return NULL;
+}
+
+/**
+ * Copy @count bytes from @buf to the preserved memory node and object
+ * associated with stream @ps. The stream must have been initialized with
+ * pkram_prepare_save() and pkram_prepare_save_obj().
+ *
+ * On success, returns the number of bytes written, which is always equal to
+ * @count. On failure, -errno is returned.
+ */
+ssize_t pkram_write(struct pkram_stream *ps, const void *buf, size_t count)
+{
+ return -ENOSYS;
+}
+
+/**
+ * Copy up to @count bytes from the preserved memory node and object
+ * associated with stream @ps to @buf. The stream must have been initialized
+ * with pkram_prepare_load() and pkram_prepare_load_obj().
+ *
+ * Returns the number of bytes read, which may be less than @count if the node
+ * has fewer bytes available.
+ */
+size_t pkram_read(struct pkram_stream *ps, void *buf, size_t count)
+{
+ return 0;
+}
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 02/43] mm: PKRAM: implement node load and save functions
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
2020-05-07 0:41 ` [RFC 01/43] mm: add PKRAM API stubs and Kconfig Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 03/43] mm: PKRAM: implement object " Anthony Yznaga
` (40 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Preserved memory is divided into nodes which can be saved and loaded
independently of each other. PKRAM nodes are kept on a list and
identified by unique names. Whenever a save operation is initiated by
calling pkram_prepare_save(), a new node is created and linked to the
list. When the save operation has been committed by calling
pkram_finish_save(), the node becomes loadable. A load operation can be
then initiated by calling pkram_prepare_load() which deletes the node
from the list and prepares the corresponding stream for loading data
from it. After the load has been finished, the pkram_finish_load()
function must be called to free the node. Nodes are also deleted when a
save operation is discarded, i.e. pkram_discard_save() is called instead
of pkram_finish_save().
Originally-by: Vladimir Davydov <vdavydov.dev@gmail.com>
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/pkram.h | 7 ++-
mm/pkram.c | 148 ++++++++++++++++++++++++++++++++++++++++++++++++--
2 files changed, 149 insertions(+), 6 deletions(-)
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index 4c4e13311ec8..83a0579e4c1c 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -6,7 +6,12 @@
#include <linux/types.h>
#include <linux/mm_types.h>
-struct pkram_stream;
+struct pkram_node;
+
+struct pkram_stream {
+ gfp_t gfp_mask;
+ struct pkram_node *node;
+};
#define PKRAM_NAME_MAX 256 /* including nul */
diff --git a/mm/pkram.c b/mm/pkram.c
index d6f2f79d4852..5c57126353ff 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -2,16 +2,85 @@
#include <linux/err.h>
#include <linux/gfp.h>
#include <linux/kernel.h>
+#include <linux/list.h>
#include <linux/mm.h>
+#include <linux/mutex.h>
#include <linux/pkram.h>
+#include <linux/string.h>
#include <linux/types.h>
+/*
+ * Preserved memory is divided into nodes that can be saved or loaded
+ * independently of each other. The nodes are identified by unique name
+ * strings.
+ *
+ * The structure occupies a memory page.
+ */
+struct pkram_node {
+ __u32 flags;
+
+ __u8 name[PKRAM_NAME_MAX];
+};
+
+#define PKRAM_SAVE 1
+#define PKRAM_LOAD 2
+#define PKRAM_ACCMODE_MASK 3
+
+static LIST_HEAD(pkram_nodes); /* linked through page::lru */
+static DEFINE_MUTEX(pkram_mutex); /* serializes open/close */
+
+static inline struct page *pkram_alloc_page(gfp_t gfp_mask)
+{
+ return alloc_page(gfp_mask);
+}
+
+static inline void pkram_free_page(void *addr)
+{
+ free_page((unsigned long)addr);
+}
+
+static inline void pkram_insert_node(struct pkram_node *node)
+{
+ list_add(&virt_to_page(node)->lru, &pkram_nodes);
+}
+
+static inline void pkram_delete_node(struct pkram_node *node)
+{
+ list_del(&virt_to_page(node)->lru);
+}
+
+static struct pkram_node *pkram_find_node(const char *name)
+{
+ struct page *page;
+ struct pkram_node *node;
+
+ list_for_each_entry(page, &pkram_nodes, lru) {
+ node = page_address(page);
+ if (strcmp(node->name, name) == 0)
+ return node;
+ }
+ return NULL;
+}
+
+static void pkram_stream_init(struct pkram_stream *ps,
+ struct pkram_node *node, gfp_t gfp_mask)
+{
+ memset(ps, 0, sizeof(*ps));
+ ps->gfp_mask = gfp_mask;
+ ps->node = node;
+}
+
/**
* Create a preserved memory node with name @name and initialize stream @ps
* for saving data to it.
*
* @gfp_mask specifies the memory allocation mask to be used when saving data.
*
+ * Error values:
+ * %ENAMETOOLONG: name len >= PKRAM_NAME_MAX
+ * %ENOMEM: insufficient memory available
+ * %EEXIST: node with specified name already exists
+ *
* Returns 0 on success, -errno on failure.
*
* After the save has finished, pkram_finish_save() (or pkram_discard_save() in
@@ -19,7 +88,34 @@
*/
int pkram_prepare_save(struct pkram_stream *ps, const char *name, gfp_t gfp_mask)
{
- return -ENOSYS;
+ struct page *page;
+ struct pkram_node *node;
+ int err = 0;
+
+ if (strlen(name) >= PKRAM_NAME_MAX)
+ return -ENAMETOOLONG;
+
+ page = pkram_alloc_page(GFP_KERNEL | __GFP_ZERO);
+ if (!page)
+ return -ENOMEM;
+ node = page_address(page);
+
+ node->flags = PKRAM_SAVE;
+ strcpy(node->name, name);
+
+ mutex_lock(&pkram_mutex);
+ if (!pkram_find_node(name))
+ pkram_insert_node(node);
+ else
+ err = -EEXIST;
+ mutex_unlock(&pkram_mutex);
+ if (err) {
+ __free_page(page);
+ return err;
+ }
+
+ pkram_stream_init(ps, node, gfp_mask);
+ return 0;
}
/**
@@ -50,7 +146,12 @@ void pkram_finish_save_obj(struct pkram_stream *ps)
*/
void pkram_finish_save(struct pkram_stream *ps)
{
- BUG();
+ struct pkram_node *node = ps->node;
+
+ BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_SAVE);
+
+ smp_wmb();
+ node->flags &= ~PKRAM_ACCMODE_MASK;
}
/**
@@ -60,7 +161,15 @@ void pkram_finish_save(struct pkram_stream *ps)
*/
void pkram_discard_save(struct pkram_stream *ps)
{
- BUG();
+ struct pkram_node *node = ps->node;
+
+ BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_SAVE);
+
+ mutex_lock(&pkram_mutex);
+ pkram_delete_node(node);
+ mutex_unlock(&pkram_mutex);
+
+ pkram_free_page(node);
}
/**
@@ -69,11 +178,36 @@ void pkram_discard_save(struct pkram_stream *ps)
*
* Returns 0 on success, -errno on failure.
*
+ * Error values:
+ * %ENOENT: node with specified name does not exist
+ * %EBUSY: save to required node has not finished yet
+ *
* After the load has finished, pkram_finish_load() is to be called.
*/
int pkram_prepare_load(struct pkram_stream *ps, const char *name)
{
- return -ENOSYS;
+ struct pkram_node *node;
+ int err = 0;
+
+ mutex_lock(&pkram_mutex);
+ node = pkram_find_node(name);
+ if (!node) {
+ err = -ENOENT;
+ goto out_unlock;
+ }
+ if (node->flags & PKRAM_ACCMODE_MASK) {
+ err = -EBUSY;
+ goto out_unlock;
+ }
+ pkram_delete_node(node);
+out_unlock:
+ mutex_unlock(&pkram_mutex);
+ if (err)
+ return err;
+
+ node->flags |= PKRAM_LOAD;
+ pkram_stream_init(ps, node, 0);
+ return 0;
}
/**
@@ -106,7 +240,11 @@ void pkram_finish_load_obj(struct pkram_stream *ps)
*/
void pkram_finish_load(struct pkram_stream *ps)
{
- BUG();
+ struct pkram_node *node = ps->node;
+
+ BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_LOAD);
+
+ pkram_free_page(node);
}
/**
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 03/43] mm: PKRAM: implement object load and save functions
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
2020-05-07 0:41 ` [RFC 01/43] mm: add PKRAM API stubs and Kconfig Anthony Yznaga
2020-05-07 0:41 ` [RFC 02/43] mm: PKRAM: implement node load and save functions Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 04/43] mm: PKRAM: implement page stream operations Anthony Yznaga
` (39 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
PKRAM nodes are further divided into a list of objects. After a save
operation has been initiated for a node, a save operation for an object
associated with the node is initiated by calling pkram_prepare_save_obj().
A new object is created and linked to the node. The save operation for
the object is committed by calling pkram_finish_save_obj(). After a load
operation has been initiated, pkram_prepare_load_obj() is called to
delete the next object from the node and prepare the corresponding
stream for loading data from it. After the load of object has been
finished, pkram_finish_load_obj() is called to free the object. Objects
are also deleted when a save operation is discarded.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/pkram.h | 1 +
mm/pkram.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++++---
2 files changed, 74 insertions(+), 4 deletions(-)
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index 83a0579e4c1c..fabde2cd8203 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -11,6 +11,7 @@ struct pkram_node;
struct pkram_stream {
gfp_t gfp_mask;
struct pkram_node *node;
+ struct pkram_obj *obj;
};
#define PKRAM_NAME_MAX 256 /* including nul */
diff --git a/mm/pkram.c b/mm/pkram.c
index 5c57126353ff..4934ffd8b019 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -6,9 +6,14 @@
#include <linux/mm.h>
#include <linux/mutex.h>
#include <linux/pkram.h>
+#include <linux/sched.h>
#include <linux/string.h>
#include <linux/types.h>
+struct pkram_obj {
+ __u64 obj_pfn; /* points to the next object in the list */
+};
+
/*
* Preserved memory is divided into nodes that can be saved or loaded
* independently of each other. The nodes are identified by unique name
@@ -18,6 +23,7 @@
*/
struct pkram_node {
__u32 flags;
+ __u64 obj_pfn; /* points to the first obj of the node */
__u8 name[PKRAM_NAME_MAX];
};
@@ -62,6 +68,21 @@ static struct pkram_node *pkram_find_node(const char *name)
return NULL;
}
+static void pkram_truncate_node(struct pkram_node *node)
+{
+ unsigned long obj_pfn;
+ struct pkram_obj *obj;
+
+ obj_pfn = node->obj_pfn;
+ while (obj_pfn) {
+ obj = pfn_to_kaddr(obj_pfn);
+ obj_pfn = obj->obj_pfn;
+ pkram_free_page(obj);
+ cond_resched();
+ }
+ node->obj_pfn = 0;
+}
+
static void pkram_stream_init(struct pkram_stream *ps,
struct pkram_node *node, gfp_t gfp_mask)
{
@@ -70,6 +91,11 @@ static void pkram_stream_init(struct pkram_stream *ps,
ps->node = node;
}
+static void pkram_stream_init_obj(struct pkram_stream *ps, struct pkram_obj *obj)
+{
+ ps->obj = obj;
+}
+
/**
* Create a preserved memory node with name @name and initialize stream @ps
* for saving data to it.
@@ -124,12 +150,31 @@ int pkram_prepare_save(struct pkram_stream *ps, const char *name, gfp_t gfp_mask
*
* Returns 0 on success, -errno on failure.
*
+ * Error values:
+ * %ENOMEM: insufficient memory available
+ *
* After the save has finished, pkram_finish_save_obj() (or pkram_discard_save()
* in case of failure) is to be called.
*/
int pkram_prepare_save_obj(struct pkram_stream *ps)
{
- return -ENOSYS;
+ struct pkram_node *node = ps->node;
+ struct pkram_obj *obj;
+ struct page *page;
+
+ BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_SAVE);
+
+ page = pkram_alloc_page(GFP_KERNEL | __GFP_ZERO);
+ if (!page)
+ return -ENOMEM;
+ obj = page_address(page);
+
+ if (node->obj_pfn)
+ obj->obj_pfn = node->obj_pfn;
+ node->obj_pfn = page_to_pfn(page);
+
+ pkram_stream_init_obj(ps, obj);
+ return 0;
}
/**
@@ -137,7 +182,9 @@ int pkram_prepare_save_obj(struct pkram_stream *ps)
*/
void pkram_finish_save_obj(struct pkram_stream *ps)
{
- BUG();
+ struct pkram_node *node = ps->node;
+
+ BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_SAVE);
}
/**
@@ -169,6 +216,7 @@ void pkram_discard_save(struct pkram_stream *ps)
pkram_delete_node(node);
mutex_unlock(&pkram_mutex);
+ pkram_truncate_node(node);
pkram_free_page(node);
}
@@ -216,11 +264,26 @@ int pkram_prepare_load(struct pkram_stream *ps, const char *name)
*
* Returns 0 on success, -errno on failure.
*
+ * Error values:
+ * %ENODATA: Stream @ps has no preserved memory objects
+ *
* After the load has finished, pkram_finish_load_obj() is to be called.
*/
int pkram_prepare_load_obj(struct pkram_stream *ps)
{
- return -ENOSYS;
+ struct pkram_node *node = ps->node;
+ struct pkram_obj *obj;
+
+ BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_LOAD);
+
+ if (!node->obj_pfn)
+ return -ENODATA;
+
+ obj = pfn_to_kaddr(node->obj_pfn);
+ node->obj_pfn = obj->obj_pfn;
+
+ pkram_stream_init_obj(ps, obj);
+ return 0;
}
/**
@@ -230,7 +293,12 @@ int pkram_prepare_load_obj(struct pkram_stream *ps)
*/
void pkram_finish_load_obj(struct pkram_stream *ps)
{
- BUG();
+ struct pkram_node *node = ps->node;
+ struct pkram_obj *obj = ps->obj;
+
+ BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_LOAD);
+
+ pkram_free_page(obj);
}
/**
@@ -244,6 +312,7 @@ void pkram_finish_load(struct pkram_stream *ps)
BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_LOAD);
+ pkram_truncate_node(node);
pkram_free_page(node);
}
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 04/43] mm: PKRAM: implement page stream operations
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (2 preceding siblings ...)
2020-05-07 0:41 ` [RFC 03/43] mm: PKRAM: implement object " Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 05/43] mm: PKRAM: support preserving transparent hugepages Anthony Yznaga
` (38 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Using the pkram_save_page() function, one can populate PKRAM objects with
memory pages which can later be loaded using the pkram_load_page()
function. Saving a memory page to PKRAM is accomplished by recording
its pfn and incrementing its refcount so that it will not be freed after
the last user puts it.
Originally-by: Vladimir Davydov <vdavydov.dev@gmail.com>
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/pkram.h | 5 ++
mm/pkram.c | 219 +++++++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 221 insertions(+), 3 deletions(-)
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index fabde2cd8203..f338d1c2aeb6 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -12,6 +12,11 @@ struct pkram_stream {
gfp_t gfp_mask;
struct pkram_node *node;
struct pkram_obj *obj;
+
+ struct pkram_link *link; /* current link */
+ unsigned int entry_idx; /* next entry in link */
+
+ unsigned long next_index;
};
#define PKRAM_NAME_MAX 256 /* including nul */
diff --git a/mm/pkram.c b/mm/pkram.c
index 4934ffd8b019..ab3053ca3539 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -1,6 +1,7 @@
// SPDX-License-Identifier: GPL-2.0
#include <linux/err.h>
#include <linux/gfp.h>
+#include <linux/io.h>
#include <linux/kernel.h>
#include <linux/list.h>
#include <linux/mm.h>
@@ -10,8 +11,38 @@
#include <linux/string.h>
#include <linux/types.h>
+#include "internal.h"
+
+
+/*
+ * Represents a reference to a data page saved to PKRAM.
+ */
+typedef __u64 pkram_entry_t;
+
+#define PKRAM_ENTRY_FLAGS_SHIFT 0x5
+#define PKRAM_ENTRY_FLAGS_MASK 0x7f
+
+/*
+ * Keeps references to data pages saved to PKRAM.
+ * The structure occupies a memory page.
+ */
+struct pkram_link {
+ __u64 link_pfn; /* points to the next link of the object */
+ __u64 index; /* mapping index of first pkram_entry_t */
+
+ /*
+ * the array occupies the rest of the link page; if the link is not
+ * full, the rest of the array must be filled with zeros
+ */
+ pkram_entry_t entry[0];
+};
+
+#define PKRAM_LINK_ENTRIES_MAX \
+ ((PAGE_SIZE-sizeof(struct pkram_link))/sizeof(pkram_entry_t))
+
struct pkram_obj {
- __u64 obj_pfn; /* points to the next object in the list */
+ __u64 link_pfn; /* points to the first link of the object */
+ __u64 obj_pfn; /* points to the next object in the list */
};
/*
@@ -19,6 +50,10 @@ struct pkram_obj {
* independently of each other. The nodes are identified by unique name
* strings.
*
+ * References to data pages saved to a preserved memory node are kept in a
+ * singly-linked list of PKRAM link structures (see above), the node has a
+ * pointer to the head of.
+ *
* The structure occupies a memory page.
*/
struct pkram_node {
@@ -68,6 +103,37 @@ static struct pkram_node *pkram_find_node(const char *name)
return NULL;
}
+static void pkram_truncate_link(struct pkram_link *link)
+{
+ struct page *page;
+ pkram_entry_t p;
+ int i;
+
+ for (i = 0; i < PKRAM_LINK_ENTRIES_MAX; i++) {
+ p = link->entry[i];
+ if (!p)
+ continue;
+ page = pfn_to_page(PHYS_PFN(p));
+ put_page(page);
+ }
+}
+
+static void pkram_truncate_obj(struct pkram_obj *obj)
+{
+ unsigned long link_pfn;
+ struct pkram_link *link;
+
+ link_pfn = obj->link_pfn;
+ while (link_pfn) {
+ link = pfn_to_kaddr(link_pfn);
+ pkram_truncate_link(link);
+ link_pfn = link->link_pfn;
+ pkram_free_page(link);
+ cond_resched();
+ }
+ obj->link_pfn = 0;
+}
+
static void pkram_truncate_node(struct pkram_node *node)
{
unsigned long obj_pfn;
@@ -76,6 +142,7 @@ static void pkram_truncate_node(struct pkram_node *node)
obj_pfn = node->obj_pfn;
while (obj_pfn) {
obj = pfn_to_kaddr(obj_pfn);
+ pkram_truncate_obj(obj);
obj_pfn = obj->obj_pfn;
pkram_free_page(obj);
cond_resched();
@@ -83,6 +150,26 @@ static void pkram_truncate_node(struct pkram_node *node)
node->obj_pfn = 0;
}
+static void pkram_add_link(struct pkram_link *link, struct pkram_obj *obj)
+{
+ link->link_pfn = obj->link_pfn;
+ obj->link_pfn = page_to_pfn(virt_to_page(link));
+}
+
+static struct pkram_link *pkram_remove_link(struct pkram_obj *obj)
+{
+ struct pkram_link *current_link;
+
+ if (!obj->link_pfn)
+ return NULL;
+
+ current_link = pfn_to_kaddr(obj->link_pfn);
+ obj->link_pfn = current_link->link_pfn;
+ current_link->link_pfn = 0;
+
+ return current_link;
+}
+
static void pkram_stream_init(struct pkram_stream *ps,
struct pkram_node *node, gfp_t gfp_mask)
{
@@ -94,6 +181,9 @@ static void pkram_stream_init(struct pkram_stream *ps,
static void pkram_stream_init_obj(struct pkram_stream *ps, struct pkram_obj *obj)
{
ps->obj = obj;
+ ps->link = NULL;
+ ps->entry_idx = 0;
+ ps->next_index = 0;
}
/**
@@ -295,9 +385,28 @@ void pkram_finish_load_obj(struct pkram_stream *ps)
{
struct pkram_node *node = ps->node;
struct pkram_obj *obj = ps->obj;
+ struct pkram_link *link = ps->link;
BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_LOAD);
+ /*
+ * If link is not null, then loading stopped within a pkram_link
+ * unexpectedly.
+ */
+ if (link) {
+ unsigned long link_pfn;
+
+ link_pfn = page_to_pfn(virt_to_page(link));
+ while (link_pfn) {
+ link = pfn_to_kaddr(link_pfn);
+ pkram_truncate_link(link);
+ link_pfn = link->link_pfn;
+ pkram_free_page(link);
+ cond_resched();
+ }
+ }
+
+ pkram_truncate_obj(obj);
pkram_free_page(obj);
}
@@ -316,6 +425,44 @@ void pkram_finish_load(struct pkram_stream *ps)
pkram_free_page(node);
}
+/*
+ * Insert page to PKRAM node allocating a new PKRAM link if necessary.
+ */
+static int __pkram_save_page(struct pkram_stream *ps,
+ struct page *page, short flags, unsigned long index)
+{
+ struct pkram_link *link = ps->link;
+ struct pkram_obj *obj = ps->obj;
+ pkram_entry_t p;
+
+ if (!link || ps->entry_idx >= PKRAM_LINK_ENTRIES_MAX ||
+ index != ps->next_index) {
+ struct page *link_page;
+
+ link_page = pkram_alloc_page((ps->gfp_mask & GFP_RECLAIM_MASK) |
+ __GFP_ZERO);
+ if (!link_page)
+ return -ENOMEM;
+
+ ps->link = link = page_address(link_page);
+ pkram_add_link(link, obj);
+
+ ps->entry_idx = 0;
+
+ ps->next_index = link->index = index;
+ }
+
+ ps->next_index++;
+
+ get_page(page);
+ p = page_to_phys(page);
+ p |= ((flags & PKRAM_ENTRY_FLAGS_MASK) << PKRAM_ENTRY_FLAGS_SHIFT);
+ link->entry[ps->entry_idx] = p;
+ ps->entry_idx++;
+
+ return 0;
+}
+
/**
* Save page @page to the preserved memory node and object associated with
* stream @ps. The stream must have been initialized with pkram_prepare_save()
@@ -324,10 +471,72 @@ void pkram_finish_load(struct pkram_stream *ps)
* @flags specifies supplemental page state to be preserved.
*
* Returns 0 on success, -errno on failure.
+ *
+ * Error values:
+ * %ENOMEM: insufficient amount of memory available
+ *
+ * Saving a page to preserved memory is simply incrementing its refcount so
+ * that it will not get freed after the last user puts it. That means it is
+ * safe to use the page as usual after it has been saved.
*/
int pkram_save_page(struct pkram_stream *ps, struct page *page, short flags)
{
- return -ENOSYS;
+ struct pkram_node *node = ps->node;
+
+ BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_SAVE);
+
+ BUG_ON(PageCompound(page));
+
+ return __pkram_save_page(ps, page, flags, page->index);
+}
+
+/*
+ * Extract the next page from preserved memory freeing a PKRAM link if it
+ * becomes empty.
+ */
+static struct page *__pkram_load_page(struct pkram_stream *ps, unsigned long *index, short *flags)
+{
+ struct pkram_link *link = ps->link;
+ struct page *page;
+ pkram_entry_t p;
+ short flgs;
+
+ if (!link) {
+ link = pkram_remove_link(ps->obj);
+ if (!link)
+ return NULL;
+
+ ps->link = link;
+ ps->entry_idx = 0;
+ ps->next_index = link->index;
+ }
+
+ BUG_ON(ps->entry_idx >= PKRAM_LINK_ENTRIES_MAX);
+
+ p = link->entry[ps->entry_idx];
+ BUG_ON(!p);
+
+ flgs = (p >> PKRAM_ENTRY_FLAGS_SHIFT) & PKRAM_ENTRY_FLAGS_MASK;
+ page = pfn_to_page(PHYS_PFN(p));
+
+ if (flags)
+ *flags = flgs;
+ if (index)
+ *index = ps->next_index;
+
+ ps->next_index++;
+
+ /* clear to avoid double free (see pkram_truncate_link()) */
+ link->entry[ps->entry_idx] = 0;
+
+ ps->entry_idx++;
+ if (ps->entry_idx >= PKRAM_LINK_ENTRIES_MAX ||
+ !link->entry[ps->entry_idx]) {
+ ps->link = NULL;
+ pkram_free_page(link);
+ }
+
+ return page;
}
/**
@@ -346,7 +555,11 @@ int pkram_save_page(struct pkram_stream *ps, struct page *page, short flags)
*/
struct page *pkram_load_page(struct pkram_stream *ps, unsigned long *index, short *flags)
{
- return NULL;
+ struct pkram_node *node = ps->node;
+
+ BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_LOAD);
+
+ return __pkram_load_page(ps, index, flags);
}
/**
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 05/43] mm: PKRAM: support preserving transparent hugepages
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (3 preceding siblings ...)
2020-05-07 0:41 ` [RFC 04/43] mm: PKRAM: implement page stream operations Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 06/43] mm: PKRAM: implement byte stream operations Anthony Yznaga
` (37 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Support preserving a transparent hugepage by recording the page order and
a flag indicating it is a THP. Use these values when the page is
restored to reconstruct the THP.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/pkram.h | 2 ++
mm/pkram.c | 20 ++++++++++++++++----
2 files changed, 18 insertions(+), 4 deletions(-)
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index f338d1c2aeb6..584cadb662b4 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -33,6 +33,8 @@ int pkram_prepare_load_obj(struct pkram_stream *ps);
void pkram_finish_load(struct pkram_stream *ps);
void pkram_finish_load_obj(struct pkram_stream *ps);
+#define PKRAM_PAGE_TRANS_HUGE 0x1 /* page is a transparent hugepage */
+
int pkram_save_page(struct pkram_stream *ps, struct page *page, short flags);
struct page *pkram_load_page(struct pkram_stream *ps, unsigned long *index,
short *flags);
diff --git a/mm/pkram.c b/mm/pkram.c
index ab3053ca3539..9164060e36f5 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -21,6 +21,7 @@ typedef __u64 pkram_entry_t;
#define PKRAM_ENTRY_FLAGS_SHIFT 0x5
#define PKRAM_ENTRY_FLAGS_MASK 0x7f
+#define PKRAM_ENTRY_ORDER_MASK 0x1f
/*
* Keeps references to data pages saved to PKRAM.
@@ -434,6 +435,7 @@ static int __pkram_save_page(struct pkram_stream *ps,
struct pkram_link *link = ps->link;
struct pkram_obj *obj = ps->obj;
pkram_entry_t p;
+ int order;
if (!link || ps->entry_idx >= PKRAM_LINK_ENTRIES_MAX ||
index != ps->next_index) {
@@ -452,10 +454,15 @@ static int __pkram_save_page(struct pkram_stream *ps,
ps->next_index = link->index = index;
}
- ps->next_index++;
+ if (PageTransHuge(page))
+ flags |= PKRAM_PAGE_TRANS_HUGE;
+
+ order = compound_order(page);
+ ps->next_index += (1 << order);
get_page(page);
p = page_to_phys(page);
+ p |= order;
p |= ((flags & PKRAM_ENTRY_FLAGS_MASK) << PKRAM_ENTRY_FLAGS_SHIFT);
link->entry[ps->entry_idx] = p;
ps->entry_idx++;
@@ -485,8 +492,6 @@ int pkram_save_page(struct pkram_stream *ps, struct page *page, short flags)
BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_SAVE);
- BUG_ON(PageCompound(page));
-
return __pkram_save_page(ps, page, flags, page->index);
}
@@ -499,6 +504,7 @@ static struct page *__pkram_load_page(struct pkram_stream *ps, unsigned long *in
struct pkram_link *link = ps->link;
struct page *page;
pkram_entry_t p;
+ int order;
short flgs;
if (!link) {
@@ -517,14 +523,20 @@ static struct page *__pkram_load_page(struct pkram_stream *ps, unsigned long *in
BUG_ON(!p);
flgs = (p >> PKRAM_ENTRY_FLAGS_SHIFT) & PKRAM_ENTRY_FLAGS_MASK;
+ order = p & PKRAM_ENTRY_ORDER_MASK;
page = pfn_to_page(PHYS_PFN(p));
+ if (flgs & PKRAM_PAGE_TRANS_HUGE) {
+ prep_compound_page(page, order);
+ prep_transhuge_page(page);
+ }
+
if (flags)
*flags = flgs;
if (index)
*index = ps->next_index;
- ps->next_index++;
+ ps->next_index += (1 << order);
/* clear to avoid double free (see pkram_truncate_link()) */
link->entry[ps->entry_idx] = 0;
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 06/43] mm: PKRAM: implement byte stream operations
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (4 preceding siblings ...)
2020-05-07 0:41 ` [RFC 05/43] mm: PKRAM: support preserving transparent hugepages Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 07/43] mm: PKRAM: link nodes by pfn before reboot Anthony Yznaga
` (36 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
This patch adds the ability to save arbitrary byte streams up to a
total length of one page to a PKRAM object using pkram_write() to be
restored later using pkram_read().
Originally-by: Vladimir Davydov <vdavydov.dev@gmail.com>
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/pkram.h | 4 +++
mm/pkram.c | 84 +++++++++++++++++++++++++++++++++++++++++++++++++--
2 files changed, 86 insertions(+), 2 deletions(-)
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index 584cadb662b4..a58dd2ea835a 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -17,6 +17,10 @@ struct pkram_stream {
unsigned int entry_idx; /* next entry in link */
unsigned long next_index;
+
+ /* byte data */
+ struct page *data_page;
+ unsigned int data_offset;
};
#define PKRAM_NAME_MAX 256 /* including nul */
diff --git a/mm/pkram.c b/mm/pkram.c
index 9164060e36f5..06b471eea0b0 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -1,6 +1,7 @@
// SPDX-License-Identifier: GPL-2.0
#include <linux/err.h>
#include <linux/gfp.h>
+#include <linux/highmem.h>
#include <linux/io.h>
#include <linux/kernel.h>
#include <linux/list.h>
@@ -42,6 +43,8 @@ struct pkram_link {
((PAGE_SIZE-sizeof(struct pkram_link))/sizeof(pkram_entry_t))
struct pkram_obj {
+ __u64 data_pfn; /* points to the byte data */
+ __u64 data_len; /* byte data size */
__u64 link_pfn; /* points to the first link of the object */
__u64 obj_pfn; /* points to the next object in the list */
};
@@ -407,6 +410,9 @@ void pkram_finish_load_obj(struct pkram_stream *ps)
}
}
+ if (ps->data_page)
+ pkram_free_page(page_address(ps->data_page));
+
pkram_truncate_obj(obj);
pkram_free_page(obj);
}
@@ -422,6 +428,9 @@ void pkram_finish_load(struct pkram_stream *ps)
BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_LOAD);
+ if (ps->data_page)
+ put_page(ps->data_page);
+
pkram_truncate_node(node);
pkram_free_page(node);
}
@@ -581,10 +590,41 @@ struct page *pkram_load_page(struct pkram_stream *ps, unsigned long *index, shor
*
* On success, returns the number of bytes written, which is always equal to
* @count. On failure, -errno is returned.
+ *
+ * Error values:
+ * %ENOMEM: insufficient amount of memory available
*/
ssize_t pkram_write(struct pkram_stream *ps, const void *buf, size_t count)
{
- return -ENOSYS;
+ struct pkram_node *node = ps->node;
+ struct pkram_obj *obj = ps->obj;
+ void *addr;
+
+ BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_SAVE);
+
+ if (!ps->data_page) {
+ struct page *page;
+
+ page = pkram_alloc_page((ps->gfp_mask & GFP_RECLAIM_MASK) |
+ __GFP_HIGHMEM | __GFP_ZERO);
+ if (!page)
+ return -ENOMEM;
+
+ ps->data_page = page;
+ ps->data_offset = 0;
+ obj->data_pfn = page_to_pfn(page);
+ }
+
+ BUG_ON(count > PAGE_SIZE - ps->data_offset);
+
+ addr = kmap_atomic(ps->data_page);
+ memcpy(addr + ps->data_offset, buf, count);
+ kunmap_atomic(addr);
+
+ obj->data_len += count;
+ ps->data_offset += count;
+
+ return count;
}
/**
@@ -597,5 +637,45 @@ ssize_t pkram_write(struct pkram_stream *ps, const void *buf, size_t count)
*/
size_t pkram_read(struct pkram_stream *ps, void *buf, size_t count)
{
- return 0;
+ struct pkram_node *node = ps->node;
+ struct pkram_obj *obj = ps->obj;
+ size_t copy_count;
+ char *addr;
+
+ BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_LOAD);
+
+ if (!count || !obj->data_len)
+ return 0;
+
+ if (!ps->data_page) {
+ struct page *page;
+
+ page = pfn_to_page(obj->data_pfn);
+ if (!page)
+ return 0;
+
+ ps->data_page = page;
+ ps->data_offset = 0;
+ obj->data_pfn = 0;
+ }
+
+ BUG_ON(count > PAGE_SIZE - ps->data_offset);
+
+ copy_count = min_t(size_t, count, PAGE_SIZE - ps->data_offset);
+ if (copy_count > obj->data_len)
+ copy_count = obj->data_len;
+
+ addr = kmap_atomic(ps->data_page);
+ memcpy(buf, addr + ps->data_offset, copy_count);
+ kunmap_atomic(addr);
+
+ obj->data_len -= copy_count;
+ ps->data_offset += copy_count;
+
+ if (!obj->data_len) {
+ pkram_free_page(page_address(ps->data_page));
+ ps->data_page = NULL;
+ }
+
+ return copy_count;
}
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 07/43] mm: PKRAM: link nodes by pfn before reboot
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (5 preceding siblings ...)
2020-05-07 0:41 ` [RFC 06/43] mm: PKRAM: implement byte stream operations Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 08/43] mm: PKRAM: introduce super block Anthony Yznaga
` (35 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Since page structs are used for linking PKRAM nodes and cleared
on boot, organize all PKRAM nodes into a list singly-linked by pfns
before reboot to facilitate the node list restore in the new kernel.
Originally-by: Vladimir Davydov <vdavydov.dev@gmail.com>
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
mm/pkram.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 50 insertions(+)
diff --git a/mm/pkram.c b/mm/pkram.c
index 06b471eea0b0..44fadb70acf6 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -2,12 +2,16 @@
#include <linux/err.h>
#include <linux/gfp.h>
#include <linux/highmem.h>
+#include <linux/init.h>
#include <linux/io.h>
#include <linux/kernel.h>
#include <linux/list.h>
#include <linux/mm.h>
+#include <linux/module.h>
#include <linux/mutex.h>
+#include <linux/notifier.h>
#include <linux/pkram.h>
+#include <linux/reboot.h>
#include <linux/sched.h>
#include <linux/string.h>
#include <linux/types.h>
@@ -58,11 +62,15 @@ struct pkram_obj {
* singly-linked list of PKRAM link structures (see above), the node has a
* pointer to the head of.
*
+ * To facilitate data restore in the new kernel, before reboot all PKRAM nodes
+ * are organized into a list singly-linked by pfn's (see pkram_reboot()).
+ *
* The structure occupies a memory page.
*/
struct pkram_node {
__u32 flags;
__u64 obj_pfn; /* points to the first obj of the node */
+ __u64 node_pfn; /* points to the next node in the node list */
__u8 name[PKRAM_NAME_MAX];
};
@@ -71,6 +79,10 @@ struct pkram_node {
#define PKRAM_LOAD 2
#define PKRAM_ACCMODE_MASK 3
+/*
+ * For convenience sake PKRAM nodes are kept in an auxiliary doubly-linked list
+ * connected through the lru field of the page struct.
+ */
static LIST_HEAD(pkram_nodes); /* linked through page::lru */
static DEFINE_MUTEX(pkram_mutex); /* serializes open/close */
@@ -679,3 +691,41 @@ size_t pkram_read(struct pkram_stream *ps, void *buf, size_t count)
return copy_count;
}
+
+/*
+ * Build the list of PKRAM nodes.
+ */
+static void __pkram_reboot(void)
+{
+ struct page *page;
+ struct pkram_node *node;
+ unsigned long node_pfn = 0;
+
+ list_for_each_entry_reverse(page, &pkram_nodes, lru) {
+ node = page_address(page);
+ if (WARN_ON(node->flags & PKRAM_ACCMODE_MASK))
+ continue;
+ node->node_pfn = node_pfn;
+ node_pfn = page_to_pfn(page);
+ }
+}
+
+static int pkram_reboot(struct notifier_block *notifier,
+ unsigned long val, void *v)
+{
+ if (val != SYS_RESTART)
+ return NOTIFY_DONE;
+ __pkram_reboot();
+ return NOTIFY_OK;
+}
+
+static struct notifier_block pkram_reboot_notifier = {
+ .notifier_call = pkram_reboot,
+};
+
+static int __init pkram_init(void)
+{
+ register_reboot_notifier(&pkram_reboot_notifier);
+ return 0;
+}
+module_init(pkram_init);
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 08/43] mm: PKRAM: introduce super block
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (6 preceding siblings ...)
2020-05-07 0:41 ` [RFC 07/43] mm: PKRAM: link nodes by pfn before reboot Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 09/43] PKRAM: build a physical mapping pagetable of pages to be preserved Anthony Yznaga
` (34 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
The PKRAM super block is the starting point for restoring preserved
memory. By providing the super block to the new kernel at boot time,
preserved memory can be reserved and made available to be restored.
To point the kernel to the location of the super block, one passes
its pfn via the 'pkram' boot param. For that purpose, the pkram super
block pfn is exported via /sys/kernel/pkram. If none is passed, any
preserved memory will not be kept, and a new super block will be
allocated.
Originally-by: Vladimir Davydov <vdavydov.dev@gmail.com>
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
mm/pkram.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 94 insertions(+), 2 deletions(-)
diff --git a/mm/pkram.c b/mm/pkram.c
index 44fadb70acf6..70f2219e6218 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -5,15 +5,18 @@
#include <linux/init.h>
#include <linux/io.h>
#include <linux/kernel.h>
+#include <linux/kobject.h>
#include <linux/list.h>
#include <linux/mm.h>
#include <linux/module.h>
#include <linux/mutex.h>
#include <linux/notifier.h>
+#include <linux/pfn.h>
#include <linux/pkram.h>
#include <linux/reboot.h>
#include <linux/sched.h>
#include <linux/string.h>
+#include <linux/sysfs.h>
#include <linux/types.h>
#include "internal.h"
@@ -80,12 +83,38 @@ struct pkram_node {
#define PKRAM_ACCMODE_MASK 3
/*
+ * The PKRAM super block contains data needed to restore the preserved memory
+ * structure on boot. The pointer to it (pfn) should be passed via the 'pkram'
+ * boot param if one wants to restore preserved data saved by the previously
+ * executing kernel. For that purpose the kernel exports the pfn via
+ * /sys/kernel/pkram. If none is passed, preserved memory if any will not be
+ * preserved and a new clean page will be allocated for the super block.
+ *
+ * The structure occupies a memory page.
+ */
+struct pkram_super_block {
+ __u64 node_pfn; /* first element of the node list */
+};
+
+static unsigned long pkram_sb_pfn __initdata;
+static struct pkram_super_block *pkram_sb;
+
+/*
* For convenience sake PKRAM nodes are kept in an auxiliary doubly-linked list
* connected through the lru field of the page struct.
*/
static LIST_HEAD(pkram_nodes); /* linked through page::lru */
static DEFINE_MUTEX(pkram_mutex); /* serializes open/close */
+/*
+ * The PKRAM super block pfn, see above.
+ */
+static int __init parse_pkram_sb_pfn(char *arg)
+{
+ return kstrtoul(arg, 16, &pkram_sb_pfn);
+}
+early_param("pkram", parse_pkram_sb_pfn);
+
static inline struct page *pkram_alloc_page(gfp_t gfp_mask)
{
return alloc_page(gfp_mask);
@@ -209,6 +238,7 @@ static void pkram_stream_init_obj(struct pkram_stream *ps, struct pkram_obj *obj
* @gfp_mask specifies the memory allocation mask to be used when saving data.
*
* Error values:
+ * %ENODEV: PKRAM not available
* %ENAMETOOLONG: name len >= PKRAM_NAME_MAX
* %ENOMEM: insufficient memory available
* %EEXIST: node with specified name already exists
@@ -224,6 +254,9 @@ int pkram_prepare_save(struct pkram_stream *ps, const char *name, gfp_t gfp_mask
struct pkram_node *node;
int err = 0;
+ if (!pkram_sb)
+ return -ENODEV;
+
if (strlen(name) >= PKRAM_NAME_MAX)
return -ENAMETOOLONG;
@@ -333,6 +366,7 @@ void pkram_discard_save(struct pkram_stream *ps)
* Returns 0 on success, -errno on failure.
*
* Error values:
+ * %ENODEV: PKRAM not available
* %ENOENT: node with specified name does not exist
* %EBUSY: save to required node has not finished yet
*
@@ -343,6 +377,9 @@ int pkram_prepare_load(struct pkram_stream *ps, const char *name)
struct pkram_node *node;
int err = 0;
+ if (!pkram_sb)
+ return -ENODEV;
+
mutex_lock(&pkram_mutex);
node = pkram_find_node(name);
if (!node) {
@@ -708,6 +745,7 @@ static void __pkram_reboot(void)
node->node_pfn = node_pfn;
node_pfn = page_to_pfn(page);
}
+ pkram_sb->node_pfn = node_pfn;
}
static int pkram_reboot(struct notifier_block *notifier,
@@ -715,7 +753,8 @@ static int pkram_reboot(struct notifier_block *notifier,
{
if (val != SYS_RESTART)
return NOTIFY_DONE;
- __pkram_reboot();
+ if (pkram_sb)
+ __pkram_reboot();
return NOTIFY_OK;
}
@@ -723,9 +762,62 @@ static struct notifier_block pkram_reboot_notifier = {
.notifier_call = pkram_reboot,
};
+static ssize_t show_pkram_sb_pfn(struct kobject *kobj,
+ struct kobj_attribute *attr, char *buf)
+{
+ unsigned long pfn = pkram_sb ? PFN_DOWN(__pa(pkram_sb)) : 0;
+
+ return sprintf(buf, "%lx\n", pfn);
+}
+
+static struct kobj_attribute pkram_sb_pfn_attr =
+ __ATTR(pkram, 0444, show_pkram_sb_pfn, NULL);
+
+static struct attribute *pkram_attrs[] = {
+ &pkram_sb_pfn_attr.attr,
+ NULL,
+};
+
+static struct attribute_group pkram_attr_group = {
+ .attrs = pkram_attrs,
+};
+
+/* returns non-zero on success */
+static int __init pkram_init_sb(void)
+{
+ unsigned long pfn;
+ struct pkram_node *node;
+
+ if (!pkram_sb) {
+ struct page *page;
+
+ page = pkram_alloc_page(GFP_KERNEL | __GFP_ZERO);
+ if (!page) {
+ pr_err("PKRAM: Failed to allocate super block\n");
+ return 0;
+ }
+ pkram_sb = page_address(page);
+ }
+
+ /*
+ * Build auxiliary doubly-linked list of nodes connected through
+ * page::lru for convenience sake.
+ */
+ pfn = pkram_sb->node_pfn;
+ while (pfn) {
+ node = pfn_to_kaddr(pfn);
+ pkram_insert_node(node);
+ pfn = node->node_pfn;
+ }
+ return 1;
+}
+
static int __init pkram_init(void)
{
- register_reboot_notifier(&pkram_reboot_notifier);
+ if (pkram_init_sb()) {
+ register_reboot_notifier(&pkram_reboot_notifier);
+ sysfs_update_group(kernel_kobj, &pkram_attr_group);
+ }
return 0;
}
module_init(pkram_init);
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 09/43] PKRAM: build a physical mapping pagetable of pages to be preserved
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (7 preceding siblings ...)
2020-05-07 0:41 ` [RFC 08/43] mm: PKRAM: introduce super block Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 10/43] PKRAM: add code for walking the preserved pages pagetable Anthony Yznaga
` (33 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Future patches will need a way to efficiently identify physically
contiguous ranges of preserved pages regardless of their virtual
addresses as well as a way to identify ranges that do not contain
preserved pages. To facilitate this all pages to be preserved across
kexec are added to an identity mapping-style pagetable that is passed
to the next kernel.
The pagetable makes use of the existing architecture definitions for
building a memory mapping pagetable with the primary difference being
that a bitmap is used to represent the presence or absence of preserved
pages at the PTE level.
In general both metadata pages and data pages must be added to the
pagetable. A mapping for a metadata page can be added when the page is
allocated, but there is an exception: for the pagetable pages themselves
mappings are added after they are allocated to avoid recursion.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
mm/pkram.c | 233 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 230 insertions(+), 3 deletions(-)
diff --git a/mm/pkram.c b/mm/pkram.c
index 70f2219e6218..5a7b8f61a55d 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -99,6 +99,12 @@ struct pkram_super_block {
static unsigned long pkram_sb_pfn __initdata;
static struct pkram_super_block *pkram_sb;
+static pgd_t *pkram_pgd;
+static DEFINE_SPINLOCK(pkram_pgd_lock);
+
+static int pkram_add_identity_map(struct page *page);
+static void pkram_remove_identity_map(struct page *page);
+
/*
* For convenience sake PKRAM nodes are kept in an auxiliary doubly-linked list
* connected through the lru field of the page struct.
@@ -115,13 +121,31 @@ static int __init parse_pkram_sb_pfn(char *arg)
}
early_param("pkram", parse_pkram_sb_pfn);
+static inline struct page *__pkram_alloc_page(gfp_t gfp_mask, bool add_to_map)
+{
+ struct page *page;
+ int err;
+
+ page = alloc_page(gfp_mask);
+ if (page && add_to_map) {
+ err = pkram_add_identity_map(page);
+ if (err) {
+ __free_page(page);
+ page = NULL;
+ }
+ }
+
+ return page;
+}
+
static inline struct page *pkram_alloc_page(gfp_t gfp_mask)
{
- return alloc_page(gfp_mask);
+ return __pkram_alloc_page(gfp_mask, true);
}
static inline void pkram_free_page(void *addr)
{
+ pkram_remove_identity_map(virt_to_page(addr));
free_page((unsigned long)addr);
}
@@ -159,6 +183,7 @@ static void pkram_truncate_link(struct pkram_link *link)
if (!p)
continue;
page = pfn_to_page(PHYS_PFN(p));
+ pkram_remove_identity_map(page);
put_page(page);
}
}
@@ -547,10 +572,15 @@ static int __pkram_save_page(struct pkram_stream *ps,
int pkram_save_page(struct pkram_stream *ps, struct page *page, short flags)
{
struct pkram_node *node = ps->node;
+ int err;
BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_SAVE);
- return __pkram_save_page(ps, page, flags, page->index);
+ err = __pkram_save_page(ps, page, flags, page->index);
+ if (!err)
+ err = pkram_add_identity_map(page);
+
+ return err;
}
/*
@@ -599,6 +629,8 @@ static struct page *__pkram_load_page(struct pkram_stream *ps, unsigned long *in
/* clear to avoid double free (see pkram_truncate_link()) */
link->entry[ps->entry_idx] = 0;
+ pkram_remove_identity_map(page);
+
ps->entry_idx++;
if (ps->entry_idx >= PKRAM_LINK_ENTRIES_MAX ||
!link->entry[ps->entry_idx]) {
@@ -791,7 +823,7 @@ static int __init pkram_init_sb(void)
if (!pkram_sb) {
struct page *page;
- page = pkram_alloc_page(GFP_KERNEL | __GFP_ZERO);
+ page = __pkram_alloc_page(GFP_KERNEL | __GFP_ZERO, false);
if (!page) {
pr_err("PKRAM: Failed to allocate super block\n");
return 0;
@@ -821,3 +853,198 @@ static int __init pkram_init(void)
return 0;
}
module_init(pkram_init);
+
+static unsigned long *pkram_alloc_pte_bitmap(void)
+{
+ return page_address(__pkram_alloc_page(GFP_KERNEL | __GFP_ZERO, false));
+}
+
+static void pkram_free_pte_bitmap(void *bitmap)
+{
+ pkram_remove_identity_map(virt_to_page(bitmap));
+ free_page((unsigned long)bitmap);
+}
+
+#define set_p4d(p4dp, p4d) WRITE_ONCE(*(p4dp), (p4d))
+
+static int pkram_add_identity_map(struct page *page)
+{
+ unsigned long orig_paddr, paddr;
+ unsigned long *bitmap;
+ int result = -ENOMEM;
+ unsigned int index;
+ struct page *pg;
+ LIST_HEAD(list);
+ pgd_t *pgd;
+ p4d_t *p4d;
+ pud_t *pud;
+ pmd_t *pmd;
+
+ if (!pkram_pgd) {
+ spin_lock(&pkram_pgd_lock);
+ if (!pkram_pgd) {
+ pg = __pkram_alloc_page(GFP_KERNEL | __GFP_ZERO, false);
+ if (!pg)
+ goto err;
+ pkram_pgd = page_address(pg);
+ }
+ spin_unlock(&pkram_pgd_lock);
+ }
+
+ orig_paddr = paddr = __pa(page_address(page));
+again:
+ pgd = pkram_pgd;
+ pgd += pgd_index(paddr);
+ if (pgd_none(*pgd)) {
+ spin_lock(&pkram_pgd_lock);
+ if (pgd_none(*pgd)) {
+ pg = __pkram_alloc_page(GFP_KERNEL|__GFP_ZERO, false);
+ if (!pg)
+ goto err;
+ list_add(&pg->lru, &list);
+ p4d = page_address(pg);
+ set_pgd(pgd, __pgd(__pa(p4d)));
+ }
+ spin_unlock(&pkram_pgd_lock);
+ }
+ p4d = p4d_offset(pgd, paddr);
+ if (p4d_none(*p4d)) {
+ spin_lock(&pkram_pgd_lock);
+ if (p4d_none(*p4d)) {
+ pg = __pkram_alloc_page(GFP_KERNEL|__GFP_ZERO, false);
+ if (!pg)
+ goto err;
+ list_add(&pg->lru, &list);
+ pud = page_address(pg);
+ set_p4d(p4d, __p4d(__pa(pud)));
+ }
+ spin_unlock(&pkram_pgd_lock);
+ }
+ pud = pud_offset(p4d, paddr);
+ if (pud_none(*pud)) {
+ spin_lock(&pkram_pgd_lock);
+ if (pud_none(*pud)) {
+ pg = __pkram_alloc_page(GFP_KERNEL|__GFP_ZERO, false);
+ if (!pg)
+ goto err;
+ list_add(&pg->lru, &list);
+ pmd = page_address(pg);
+ set_pud(pud, __pud(__pa(pmd)));
+ }
+ spin_unlock(&pkram_pgd_lock);
+ }
+ pmd = pmd_offset(pud, paddr);
+ if (pmd_none(*pmd)) {
+ spin_lock(&pkram_pgd_lock);
+ if (pmd_none(*pmd)) {
+ if (PageTransHuge(page)) {
+ set_pmd(pmd, pmd_mkhuge(*pmd));
+ spin_unlock(&pkram_pgd_lock);
+ goto next;
+ }
+ bitmap = pkram_alloc_pte_bitmap();
+ if (!bitmap)
+ goto err;
+ pg = virt_to_page(bitmap);
+ list_add(&pg->lru, &list);
+ set_pmd(pmd, __pmd(__pa(bitmap)));
+ } else {
+ BUG_ON(pmd_large(*pmd));
+ bitmap = __va(pmd_val(*pmd));
+ }
+ spin_unlock(&pkram_pgd_lock);
+ } else {
+ BUG_ON(pmd_large(*pmd));
+ bitmap = __va(pmd_val(*pmd));
+ }
+
+ index = pte_index(paddr);
+ BUG_ON(test_bit(index, bitmap));
+ set_bit(index, bitmap);
+ smp_mb__after_atomic();
+ if (bitmap_full(bitmap, PTRS_PER_PTE))
+ set_pmd(pmd, pmd_mkhuge(*pmd));
+next:
+ /* Add mappings for any pagetable pages that were allocated */
+ if (!list_empty(&list)) {
+ page = list_first_entry(&list, struct page, lru);
+ list_del_init(&page->lru);
+ paddr = __pa(page_address(page));
+ goto again;
+ }
+
+ return 0;
+err:
+ spin_unlock(&pkram_pgd_lock);
+ while (!list_empty(&list)) {
+ pg = list_first_entry(&list, struct page, lru);
+ list_del_init(&pg->lru);
+ }
+ return result;
+}
+
+static void pkram_remove_identity_map(struct page *page)
+{
+ unsigned long *bitmap;
+ unsigned long paddr;
+ unsigned int index;
+ pgd_t *pgd;
+ p4d_t *p4d;
+ pud_t *pud;
+ pmd_t *pmd;
+
+ /*
+ * pkram_pgd will be null when freeing metadata pages after a reboot
+ */
+ if (!pkram_pgd)
+ return;
+
+ paddr = __pa(page_address(page));
+ pgd = pkram_pgd;
+ pgd += pgd_index(paddr);
+ if (pgd_none(*pgd)) {
+ WARN_ONCE(1, "PKRAM: %s: no pgd for 0x%lx\n", __func__, paddr);
+ return;
+ }
+ p4d = p4d_offset(pgd, paddr);
+ if (p4d_none(*p4d)) {
+ WARN_ONCE(1, "PKRAM: %s: no p4d for 0x%lx\n", __func__, paddr);
+ return;
+ }
+ pud = pud_offset(p4d, paddr);
+ if (pud_none(*pud)) {
+ WARN_ONCE(1, "PKRAM: %s: no pud for 0x%lx\n", __func__, paddr);
+ return;
+ }
+ pmd = pmd_offset(pud, paddr);
+ if (pmd_none(*pmd)) {
+ WARN_ONCE(1, "PKRAM: %s: no pmd for 0x%lx\n", __func__, paddr);
+ return;
+ }
+ if (PageTransHuge(page)) {
+ BUG_ON(!pmd_large(*pmd));
+ pmd_clear(pmd);
+ return;
+ }
+
+ if (pmd_large(*pmd)) {
+ spin_lock(&pkram_pgd_lock);
+ if (pmd_large(*pmd))
+ set_pmd(pmd, __pmd(pte_val(pte_clrhuge(*(pte_t *)pmd))));
+ spin_unlock(&pkram_pgd_lock);
+ }
+
+ bitmap = __va(pmd_val(*pmd));
+ index = pte_index(paddr);
+ clear_bit(index, bitmap);
+ smp_mb__after_atomic();
+
+ spin_lock(&pkram_pgd_lock);
+ if (!pmd_none(*pmd) && bitmap_empty(bitmap, PTRS_PER_PTE)) {
+ pmd_clear(pmd);
+ spin_unlock(&pkram_pgd_lock);
+ pkram_free_pte_bitmap(bitmap);
+ } else {
+ spin_unlock(&pkram_pgd_lock);
+ }
+}
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 10/43] PKRAM: add code for walking the preserved pages pagetable
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (8 preceding siblings ...)
2020-05-07 0:41 ` [RFC 09/43] PKRAM: build a physical mapping pagetable of pages to be preserved Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 11/43] PKRAM: pass the preserved pages pagetable to the next kernel Anthony Yznaga
` (32 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Add the ability to walk the pkram pagetable from high to low addresses
and execute a callback for each contiguous range of preserved or not
preserved memory found. The reason for walking high to low is to align
with high to low memblock allocation when finding holes that memblocks
can safely be allocated from as will be seen in a later patch.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/pkram.h | 15 +++++
mm/Makefile | 2 +-
mm/pkram_pagetable.c | 169 ++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 185 insertions(+), 1 deletion(-)
create mode 100644 mm/pkram_pagetable.c
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index a58dd2ea835a..b6fa973d37cc 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -25,6 +25,21 @@ struct pkram_stream {
#define PKRAM_NAME_MAX 256 /* including nul */
+struct pkram_pg_state {
+ int (*range_cb)(struct pkram_pg_state *state, unsigned long base,
+ unsigned long size);
+ unsigned long curr_addr;
+ unsigned long end_addr;
+ unsigned long min_addr;
+ unsigned long max_addr;
+ unsigned long min_size;
+ bool tracking;
+ bool find_holes;
+ unsigned long retval;
+};
+
+void pkram_walk_pgt_rev(struct pkram_pg_state *st, pgd_t *pgd);
+
int pkram_prepare_save(struct pkram_stream *ps, const char *name,
gfp_t gfp_mask);
int pkram_prepare_save_obj(struct pkram_stream *ps);
diff --git a/mm/Makefile b/mm/Makefile
index 59cd381194af..c4ad1c56e237 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -112,4 +112,4 @@ obj-$(CONFIG_MEMFD_CREATE) += memfd.o
obj-$(CONFIG_MAPPING_DIRTY_HELPERS) += mapping_dirty_helpers.o
obj-$(CONFIG_PTDUMP_CORE) += ptdump.o
obj-$(CONFIG_PAGE_REPORTING) += page_reporting.o
-obj-$(CONFIG_PKRAM) += pkram.o
+obj-$(CONFIG_PKRAM) += pkram.o pkram_pagetable.o
diff --git a/mm/pkram_pagetable.c b/mm/pkram_pagetable.c
new file mode 100644
index 000000000000..d31aa36207ba
--- /dev/null
+++ b/mm/pkram_pagetable.c
@@ -0,0 +1,169 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bitops.h>
+#include <asm/pgtable.h>
+#include <linux/pkram.h>
+
+#define pgd_none(a) (pgtable_l5_enabled() ? pgd_none(a) : p4d_none(__p4d(pgd_val(a))))
+
+static int note_page_rev(struct pkram_pg_state *st, unsigned long curr_size, bool present)
+{
+ unsigned long curr_addr = st->curr_addr;
+ bool track_page = present ^ st->find_holes;
+
+ if (!st->tracking && track_page) {
+ unsigned long end_addr = curr_addr + curr_size;
+
+ if (end_addr <= st->min_addr)
+ return 1;
+
+ st->end_addr = min(end_addr, st->max_addr);
+ st->tracking = true;
+ } else if (st->tracking) {
+ unsigned long base, size;
+
+ /* Continue tracking if lower bound has not been reached */
+ if (track_page && curr_addr && curr_addr >= st->min_addr)
+ return 0;
+
+ if (!track_page)
+ base = max(curr_addr + curr_size, st->min_addr);
+ else
+ base = st->min_addr;
+
+ size = st->end_addr - base;
+ st->tracking = false;
+
+ return st->range_cb(st, base, size);
+ }
+
+ return 0;
+}
+
+static int walk_pte_level_rev(struct pkram_pg_state *st, pmd_t addr, unsigned long P)
+{
+ unsigned long *bitmap;
+ int present;
+ int i, ret;
+
+ bitmap = __va(pmd_val(addr));
+ for (i = PTRS_PER_PTE - 1; i >= 0; i--) {
+ unsigned long curr_addr = P + i * PAGE_SIZE;
+
+ if (curr_addr >= st->max_addr)
+ continue;
+ st->curr_addr = curr_addr;
+
+ present = test_bit(i, bitmap);
+ ret = note_page_rev(st, PAGE_SIZE, present);
+ if (ret)
+ break;
+ }
+
+ return ret;
+}
+
+static int walk_pmd_level_rev(struct pkram_pg_state *st, pud_t addr, unsigned long P)
+{
+ pmd_t *start;
+ int i, ret;
+
+ start = (pmd_t *)pud_page_vaddr(addr) + PTRS_PER_PMD - 1;
+ for (i = PTRS_PER_PMD - 1; i >= 0; i--, start--) {
+ unsigned long curr_addr = P + i * PMD_SIZE;
+
+ if (curr_addr >= st->max_addr)
+ continue;
+ st->curr_addr = curr_addr;
+
+ if (!pmd_none(*start)) {
+ if (pmd_large(*start))
+ ret = note_page_rev(st, PMD_SIZE, true);
+ else
+ ret = walk_pte_level_rev(st, *start, curr_addr);
+ } else
+ ret = note_page_rev(st, PMD_SIZE, false);
+ if (ret)
+ break;
+ }
+
+ return ret;
+}
+
+static int walk_pud_level_rev(struct pkram_pg_state *st, p4d_t addr, unsigned long P)
+{
+ pud_t *start;
+ int i, ret;
+
+ start = (pud_t *)p4d_page_vaddr(addr) + PTRS_PER_PUD - 1;
+ for (i = PTRS_PER_PUD - 1; i >= 0 ; i--, start--) {
+ unsigned long curr_addr = P + i * PUD_SIZE;
+
+ if (curr_addr >= st->max_addr)
+ continue;
+ st->curr_addr = curr_addr;
+
+ if (!pud_none(*start)) {
+ if (pud_large(*start))
+ ret = note_page_rev(st, PUD_SIZE, true);
+ else
+ ret = walk_pmd_level_rev(st, *start, curr_addr);
+ } else
+ ret = note_page_rev(st, PUD_SIZE, false);
+ if (ret)
+ break;
+ }
+
+ return ret;
+}
+
+static int walk_p4d_level_rev(struct pkram_pg_state *st, pgd_t addr, unsigned long P)
+{
+ p4d_t *start;
+ int i, ret;
+
+ if (PTRS_PER_P4D == 1)
+ return walk_pud_level_rev(st, __p4d(pgd_val(addr)), P);
+
+ start = (p4d_t *)pgd_page_vaddr(addr) + PTRS_PER_P4D - 1;
+ for (i = PTRS_PER_P4D - 1; i >= 0; i--, start--) {
+ unsigned long curr_addr = P + i * P4D_SIZE;
+
+ if (curr_addr >= st->max_addr)
+ continue;
+ st->curr_addr = curr_addr;
+
+ if (!p4d_none(*start)) {
+ if (p4d_large(*start))
+ ret = note_page_rev(st, P4D_SIZE, true);
+ else
+ ret = walk_pud_level_rev(st, *start, curr_addr);
+ } else
+ ret = note_page_rev(st, P4D_SIZE, false);
+ if (ret)
+ break;
+ }
+
+ return ret;
+}
+
+void pkram_walk_pgt_rev(struct pkram_pg_state *st, pgd_t *pgd)
+{
+ pgd_t *start;
+ int i, ret;
+
+ start = pgd + PTRS_PER_PGD - 1;
+ for (i = PTRS_PER_PGD - 1; i >= 0; i--, start--) {
+ unsigned long curr_addr = i * PGDIR_SIZE;
+
+ if (curr_addr >= st->max_addr)
+ continue;
+ st->curr_addr = curr_addr;
+
+ if (!pgd_none(*start))
+ ret = walk_p4d_level_rev(st, *start, curr_addr);
+ else
+ ret = note_page_rev(st, PGDIR_SIZE, false);
+ if (ret)
+ break;
+ }
+}
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 11/43] PKRAM: pass the preserved pages pagetable to the next kernel
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (9 preceding siblings ...)
2020-05-07 0:41 ` [RFC 10/43] PKRAM: add code for walking the preserved pages pagetable Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 12/43] mm: PKRAM: reserve preserved memory at boot Anthony Yznaga
` (31 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Add a pointer to the pagetable to the pkram_super_block page.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
mm/pkram.c | 20 +++++++++++++-------
1 file changed, 13 insertions(+), 7 deletions(-)
diff --git a/mm/pkram.c b/mm/pkram.c
index 5a7b8f61a55d..54b2779d0813 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -94,6 +94,7 @@ struct pkram_node {
*/
struct pkram_super_block {
__u64 node_pfn; /* first element of the node list */
+ __u64 pgd_pfn;
};
static unsigned long pkram_sb_pfn __initdata;
@@ -769,15 +770,20 @@ static void __pkram_reboot(void)
struct page *page;
struct pkram_node *node;
unsigned long node_pfn = 0;
-
- list_for_each_entry_reverse(page, &pkram_nodes, lru) {
- node = page_address(page);
- if (WARN_ON(node->flags & PKRAM_ACCMODE_MASK))
- continue;
- node->node_pfn = node_pfn;
- node_pfn = page_to_pfn(page);
+ unsigned long pgd_pfn = 0;
+
+ if (pkram_pgd) {
+ list_for_each_entry_reverse(page, &pkram_nodes, lru) {
+ node = page_address(page);
+ if (WARN_ON(node->flags & PKRAM_ACCMODE_MASK))
+ continue;
+ node->node_pfn = node_pfn;
+ node_pfn = page_to_pfn(page);
+ }
+ pgd_pfn = page_to_pfn(virt_to_page(pkram_pgd));
}
pkram_sb->node_pfn = node_pfn;
+ pkram_sb->pgd_pfn = pgd_pfn;
}
static int pkram_reboot(struct notifier_block *notifier,
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 12/43] mm: PKRAM: reserve preserved memory at boot
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (10 preceding siblings ...)
2020-05-07 0:41 ` [RFC 11/43] PKRAM: pass the preserved pages pagetable to the next kernel Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 13/43] mm: PKRAM: free preserved pages pagetable Anthony Yznaga
` (30 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Keep preserved pages from being recycled during boot by adding them
to the memblock reserved list during early boot. If memory reservation
fails (e.g. a region has already been reserved), all preserved pages
are dropped.
For efficiency the preserved pages pagetable is used to identify and
reserve by the contiguous ranges present rather than a page at a time.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
arch/x86/kernel/setup.c | 3 +
arch/x86/mm/init_64.c | 2 +
include/linux/pkram.h | 8 +++
mm/pkram.c | 179 +++++++++++++++++++++++++++++++++++++++++++++++-
4 files changed, 189 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
index 4b3fa6cd3106..851515753ad9 100644
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -14,6 +14,7 @@
#include <linux/iscsi_ibft.h>
#include <linux/memblock.h>
#include <linux/pci.h>
+#include <linux/pkram.h>
#include <linux/root_dev.h>
#include <linux/sfi.h>
#include <linux/hugetlb.h>
@@ -1158,6 +1159,8 @@ void __init setup_arch(char **cmdline_p)
initmem_init();
dma_contiguous_reserve(max_pfn_mapped << PAGE_SHIFT);
+ pkram_reserve();
+
if (boot_cpu_has(X86_FEATURE_GBPAGES))
hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 3b289c2f75cd..ae569ef6bd7d 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -33,6 +33,7 @@
#include <linux/nmi.h>
#include <linux/gfp.h>
#include <linux/kcore.h>
+#include <linux/pkram.h>
#include <asm/processor.h>
#include <asm/bios_ebda.h>
@@ -1244,6 +1245,7 @@ void __init mem_init(void)
after_bootmem = 1;
x86_init.hyper.init_after_bootmem();
+ totalram_pages_add(pkram_reserved_pages);
/*
* Must be done after boot memory is put on freelist, because here we
* might set fields in deferred struct pages that have not yet been
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index b6fa973d37cc..1b475f6e1598 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -61,4 +61,12 @@ struct page *pkram_load_page(struct pkram_stream *ps, unsigned long *index,
ssize_t pkram_write(struct pkram_stream *ps, const void *buf, size_t count);
size_t pkram_read(struct pkram_stream *ps, void *buf, size_t count);
+#ifdef CONFIG_PKRAM
+extern unsigned long pkram_reserved_pages;
+void pkram_reserve(void);
+#else
+#define pkram_reserved_pages 0UL
+static inline void pkram_reserve(void) { }
+#endif
+
#endif /* _LINUX_PKRAM_H */
diff --git a/mm/pkram.c b/mm/pkram.c
index 54b2779d0813..2c323154df76 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -7,6 +7,7 @@
#include <linux/kernel.h>
#include <linux/kobject.h>
#include <linux/list.h>
+#include <linux/memblock.h>
#include <linux/mm.h>
#include <linux/module.h>
#include <linux/mutex.h>
@@ -105,6 +106,7 @@ static DEFINE_SPINLOCK(pkram_pgd_lock);
static int pkram_add_identity_map(struct page *page);
static void pkram_remove_identity_map(struct page *page);
+static int pkram_reserve_page_ranges(pgd_t *pgd);
/*
* For convenience sake PKRAM nodes are kept in an auxiliary doubly-linked list
@@ -113,6 +115,9 @@ static void pkram_remove_identity_map(struct page *page);
static LIST_HEAD(pkram_nodes); /* linked through page::lru */
static DEFINE_MUTEX(pkram_mutex); /* serializes open/close */
+unsigned long __initdata pkram_reserved_pages;
+static bool pkram_reservation_in_progress;
+
/*
* The PKRAM super block pfn, see above.
*/
@@ -122,6 +127,102 @@ static int __init parse_pkram_sb_pfn(char *arg)
}
early_param("pkram", parse_pkram_sb_pfn);
+static void * __init pkram_map_meta(unsigned long pfn)
+{
+ if (pfn >= max_low_pfn)
+ return ERR_PTR(-EINVAL);
+ return pfn_to_kaddr(pfn);
+}
+
+static int __init pkram_reserve_page(unsigned long pfn)
+{
+ phys_addr_t base, size;
+ int err = 0;
+
+ if (pfn >= max_pfn)
+ return -EINVAL;
+
+ base = PFN_PHYS(pfn);
+ size = PAGE_SIZE;
+
+ if (memblock_is_region_reserved(base, size) ||
+ memblock_reserve(base, size) < 0)
+ err = -EBUSY;
+
+ if (!err)
+ pkram_reserved_pages++;
+
+ return err;
+}
+
+static void __init pkram_unreserve_page(unsigned long pfn)
+{
+ memblock_free(PFN_PHYS(pfn), PAGE_SIZE);
+ pkram_reserved_pages--;
+}
+
+/*
+ * Reserved pages that belong to preserved memory.
+ *
+ * This function should be called at boot time as early as possible to prevent
+ * preserved memory from being recycled.
+ */
+void __init pkram_reserve(void)
+{
+ int err = 0;
+
+ if (!pkram_sb_pfn)
+ return;
+
+ pr_info("PKRAM: Examining preserved memory...\n");
+ pkram_reservation_in_progress = true;
+
+ err = pkram_reserve_page(pkram_sb_pfn);
+ if (err)
+ goto out;
+ pkram_sb = pkram_map_meta(pkram_sb_pfn);
+ if (IS_ERR(pkram_sb)) {
+ pkram_unreserve_page(pkram_sb_pfn);
+ err = PTR_ERR(pkram_sb);
+ goto out;
+ }
+
+ /* An empty pkram_sb is not an error */
+ if (!pkram_sb->node_pfn) {
+ pkram_unreserve_page(pkram_sb_pfn);
+ pkram_sb = NULL;
+ goto done;
+ }
+
+ err = pkram_reserve_page(pkram_sb->pgd_pfn);
+ if (err) {
+ pr_warn("PKRAM: pgd_pfn=0x%llx already reserved\n",
+ pkram_sb->pgd_pfn);
+ pkram_unreserve_page(pkram_sb_pfn);
+ goto out;
+ }
+ pkram_pgd = pfn_to_kaddr(pkram_sb->pgd_pfn);
+ err = pkram_reserve_page_ranges(pkram_pgd);
+ if (err) {
+ pkram_unreserve_page(pkram_sb->pgd_pfn);
+ pkram_unreserve_page(pkram_sb_pfn);
+ pkram_pgd = NULL;
+ }
+
+out:
+ pkram_reservation_in_progress = false;
+
+ if (err) {
+ pr_err("PKRAM: Reservation failed: %d\n", err);
+ WARN_ON(pkram_reserved_pages > 0);
+ pkram_sb = NULL;
+ return;
+ }
+
+done:
+ pr_info("PKRAM: %lu pages reserved\n", pkram_reserved_pages);
+}
+
static inline struct page *__pkram_alloc_page(gfp_t gfp_mask, bool add_to_map)
{
struct page *page;
@@ -146,6 +247,11 @@ static inline struct page *pkram_alloc_page(gfp_t gfp_mask)
static inline void pkram_free_page(void *addr)
{
+ /*
+ * The page may have the reserved bit set since preserved pages
+ * are reserved early in boot.
+ */
+ ClearPageReserved(virt_to_page(addr));
pkram_remove_identity_map(virt_to_page(addr));
free_page((unsigned long)addr);
}
@@ -184,6 +290,11 @@ static void pkram_truncate_link(struct pkram_link *link)
if (!p)
continue;
page = pfn_to_page(PHYS_PFN(p));
+ /*
+ * The page may have the reserved bit set since preserved pages
+ * are reserved early in boot.
+ */
+ ClearPageReserved(page);
pkram_remove_identity_map(page);
put_page(page);
}
@@ -593,7 +704,7 @@ static struct page *__pkram_load_page(struct pkram_stream *ps, unsigned long *in
struct pkram_link *link = ps->link;
struct page *page;
pkram_entry_t p;
- int order;
+ int i, order;
short flgs;
if (!link) {
@@ -615,6 +726,12 @@ static struct page *__pkram_load_page(struct pkram_stream *ps, unsigned long *in
order = p & PKRAM_ENTRY_ORDER_MASK;
page = pfn_to_page(PHYS_PFN(p));
+ for (i = 0; i < (1 << order); i++) {
+ struct page *pg = page + i;
+
+ ClearPageReserved(pg);
+ }
+
if (flgs & PKRAM_PAGE_TRANS_HUGE) {
prep_compound_page(page, order);
prep_transhuge_page(page);
@@ -735,6 +852,7 @@ size_t pkram_read(struct pkram_stream *ps, void *buf, size_t count)
page = pfn_to_page(obj->data_pfn);
if (!page)
return 0;
+ ClearPageReserved(page);
ps->data_page = page;
ps->data_offset = 0;
@@ -782,8 +900,15 @@ static void __pkram_reboot(void)
}
pgd_pfn = page_to_pfn(virt_to_page(pkram_pgd));
}
- pkram_sb->node_pfn = node_pfn;
- pkram_sb->pgd_pfn = pgd_pfn;
+ /*
+ * Zero out pkram_sb completely since it may have been passed from
+ * the previous boot.
+ */
+ memset(pkram_sb, 0, PAGE_SIZE);
+ if (node_pfn) {
+ pkram_sb->node_pfn = node_pfn;
+ pkram_sb->pgd_pfn = pgd_pfn;
+ }
}
static int pkram_reboot(struct notifier_block *notifier,
@@ -867,6 +992,7 @@ static unsigned long *pkram_alloc_pte_bitmap(void)
static void pkram_free_pte_bitmap(void *bitmap)
{
+ ClearPageReserved(virt_to_page(bitmap));
pkram_remove_identity_map(virt_to_page(bitmap));
free_page((unsigned long)bitmap);
}
@@ -1054,3 +1180,50 @@ static void pkram_remove_identity_map(struct page *page)
spin_unlock(&pkram_pgd_lock);
}
}
+
+static int __init pkram_reserve_range_cb(struct pkram_pg_state *st, unsigned long base, unsigned long size)
+{
+ if (memblock_is_region_reserved(base, size) ||
+ memblock_reserve(base, size) < 0) {
+ pr_warn("PKRAM: reservations exist in [0x%lx,0x%lx]\n", base, base + size - 1);
+ /*
+ * Set a lower bound so another walk can undo the earlier,
+ * successful reservations.
+ */
+ st->min_addr = base + size;
+ st->retval = -EBUSY;
+ return 1;
+ }
+
+ pkram_reserved_pages += (size >> PAGE_SHIFT);
+ return 0;
+}
+
+static int __init pkram_unreserve_range_cb(struct pkram_pg_state *st, unsigned long base, unsigned long size)
+{
+ memblock_free(base, size);
+ pkram_reserved_pages -= (size >> PAGE_SHIFT);
+ return 0;
+}
+
+/*
+ * Walk the preserved pages pagetable and reserve each present address range.
+ */
+static int __init pkram_reserve_page_ranges(pgd_t *pgd)
+{
+ struct pkram_pg_state st = {
+ .range_cb = pkram_reserve_range_cb,
+ .max_addr = PHYS_ADDR_MAX,
+ };
+ int err = 0;
+
+ pkram_walk_pgt_rev(&st, pgd);
+ if ((int)st.retval < 0) {
+ err = st.retval;
+ st.retval = 0;
+ st.range_cb = pkram_unreserve_range_cb;
+ pkram_walk_pgt_rev(&st, pgd);
+ }
+
+ return err;
+}
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 13/43] mm: PKRAM: free preserved pages pagetable
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (11 preceding siblings ...)
2020-05-07 0:41 ` [RFC 12/43] mm: PKRAM: reserve preserved memory at boot Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 14/43] mm: memblock: PKRAM: prevent memblock resize from clobbering preserved pages Anthony Yznaga
` (29 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
After the page ranges in the pagetable have been reserved the pagetable
is no longer needed. Rather than free it during early boot by unreserving
page-sized blocks which can be inefficient when dealing with a large number
of blocks, wait until the page structs have been initialized and free them
as pages.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
arch/x86/mm/init_64.c | 1 +
include/linux/pkram.h | 3 ++
mm/pkram.c | 11 +++++++
mm/pkram_pagetable.c | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 97 insertions(+)
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index ae569ef6bd7d..72662615977b 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -1245,6 +1245,7 @@ void __init mem_init(void)
after_bootmem = 1;
x86_init.hyper.init_after_bootmem();
+ pkram_free_pgt();
totalram_pages_add(pkram_reserved_pages);
/*
* Must be done after boot memory is put on freelist, because here we
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index 1b475f6e1598..edc5d8bef9d3 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -39,6 +39,7 @@ struct pkram_pg_state {
};
void pkram_walk_pgt_rev(struct pkram_pg_state *st, pgd_t *pgd);
+void pkram_free_pgt_walk_pgd(pgd_t *pgd);
int pkram_prepare_save(struct pkram_stream *ps, const char *name,
gfp_t gfp_mask);
@@ -64,9 +65,11 @@ size_t pkram_read(struct pkram_stream *ps, void *buf, size_t count);
#ifdef CONFIG_PKRAM
extern unsigned long pkram_reserved_pages;
void pkram_reserve(void);
+void pkram_free_pgt(void);
#else
#define pkram_reserved_pages 0UL
static inline void pkram_reserve(void) { }
+static inline void pkram_free_pgt(void) { }
#endif
#endif /* _LINUX_PKRAM_H */
diff --git a/mm/pkram.c b/mm/pkram.c
index 2c323154df76..dd3c89614010 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -1227,3 +1227,14 @@ static int __init pkram_reserve_page_ranges(pgd_t *pgd)
return err;
}
+
+void pkram_free_pgt(void)
+{
+ if (!pkram_pgd)
+ return;
+
+ pkram_free_pgt_walk_pgd(pkram_pgd);
+
+ __free_pages_core(virt_to_page(pkram_pgd), 0);
+ pkram_pgd = NULL;
+}
diff --git a/mm/pkram_pagetable.c b/mm/pkram_pagetable.c
index d31aa36207ba..7033e9b1c47f 100644
--- a/mm/pkram_pagetable.c
+++ b/mm/pkram_pagetable.c
@@ -3,6 +3,8 @@
#include <asm/pgtable.h>
#include <linux/pkram.h>
+#include "internal.h"
+
#define pgd_none(a) (pgtable_l5_enabled() ? pgd_none(a) : p4d_none(__p4d(pgd_val(a))))
static int note_page_rev(struct pkram_pg_state *st, unsigned long curr_size, bool present)
@@ -167,3 +169,83 @@ void pkram_walk_pgt_rev(struct pkram_pg_state *st, pgd_t *pgd)
break;
}
}
+
+static void pkram_free_pgt_walk_pmd(pud_t addr)
+{
+ unsigned long bitmap_pa;
+ struct page *page;
+ pmd_t *start;
+ int i;
+
+ start = (pmd_t *)pud_page_vaddr(addr);
+ for (i = 0; i < PTRS_PER_PMD; i++, start++) {
+ if (!pmd_none(*start)) {
+ bitmap_pa = pte_val(pte_clrhuge(*(pte_t *)start));
+ if (pmd_large(*start) && !bitmap_pa)
+ continue;
+ page = virt_to_page(__va(bitmap_pa));
+ __free_pages_core(page, 0);
+ }
+ }
+}
+
+static void pkram_free_pgt_walk_pud(p4d_t addr)
+{
+ struct page *page;
+ pud_t *start;
+ int i;
+
+ start = (pud_t *)p4d_page_vaddr(addr);
+ for (i = 0; i < PTRS_PER_PUD; i++, start++) {
+ if (!pud_none(*start)) {
+ if (pud_large(*start)) {
+ WARN_ONCE(1, "PKRAM: unexpected pud hugepage\n");
+ continue;
+ }
+ pkram_free_pgt_walk_pmd(*start);
+ page = virt_to_page(__va(pud_val(*start)));
+ __free_pages_core(page, 0);
+ }
+ }
+}
+
+static void pkram_free_pgt_walk_p4d(pgd_t addr)
+{
+ struct page *page;
+ p4d_t *start;
+ int i;
+
+ if (PTRS_PER_P4D == 1)
+ return pkram_free_pgt_walk_pud(__p4d(pgd_val(addr)));
+
+ start = (p4d_t *)pgd_page_vaddr(addr);
+ for (i = 0; i < PTRS_PER_P4D; i++, start++) {
+ if (!p4d_none(*start)) {
+ if (p4d_large(*start)) {
+ WARN_ONCE(1, "PKRAM: unexpected p4d hugepage\n");
+ continue;
+ }
+ pkram_free_pgt_walk_pud(*start);
+ page = virt_to_page(__va(p4d_val(*start)));
+ __free_pages_core(page, 0);
+ }
+ }
+}
+
+/*
+ * Free the pagetable passed from the previous boot.
+ */
+void pkram_free_pgt_walk_pgd(pgd_t *pgd)
+{
+ pgd_t *start = pgd;
+ struct page *page;
+ int i;
+
+ for (i = 0; i < PTRS_PER_PGD; i++, start++) {
+ if (!pgd_none(*start)) {
+ pkram_free_pgt_walk_p4d(*start);
+ page = virt_to_page(__va(pgd_val(*start)));
+ __free_pages_core(page, 0);
+ }
+ }
+}
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 14/43] mm: memblock: PKRAM: prevent memblock resize from clobbering preserved pages
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (12 preceding siblings ...)
2020-05-07 0:41 ` [RFC 13/43] mm: PKRAM: free preserved pages pagetable Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-11 13:57 ` Mike Rapoport
2020-05-07 0:41 ` [RFC 15/43] PKRAM: provide a way to ban pages from use by PKRAM Anthony Yznaga
` (28 subsequent siblings)
42 siblings, 1 reply; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
The size of the memblock reserved array may be increased while preserved
pages are being reserved. When this happens, preserved pages that have
not yet been reserved are at risk for being clobbered when space for a
larger array is allocated.
When called from memblock_double_array(), a wrapper around
memblock_find_in_range() walks the preserved pages pagetable to find
sufficiently sized ranges without preserved pages and passes them to
memblock_find_in_range().
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/pkram.h | 3 +++
mm/memblock.c | 15 +++++++++++++--
mm/pkram.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 67 insertions(+), 2 deletions(-)
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index edc5d8bef9d3..409022e1472f 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -62,6 +62,9 @@ struct page *pkram_load_page(struct pkram_stream *ps, unsigned long *index,
ssize_t pkram_write(struct pkram_stream *ps, const void *buf, size_t count);
size_t pkram_read(struct pkram_stream *ps, void *buf, size_t count);
+phys_addr_t pkram_memblock_find_in_range(phys_addr_t start, phys_addr_t end,
+ phys_addr_t size, phys_addr_t align);
+
#ifdef CONFIG_PKRAM
extern unsigned long pkram_reserved_pages;
void pkram_reserve(void);
diff --git a/mm/memblock.c b/mm/memblock.c
index c79ba6f9920c..69ae883b8d21 100644
--- a/mm/memblock.c
+++ b/mm/memblock.c
@@ -16,6 +16,7 @@
#include <linux/kmemleak.h>
#include <linux/seq_file.h>
#include <linux/memblock.h>
+#include <linux/pkram.h>
#include <asm/sections.h>
#include <linux/io.h>
@@ -349,6 +350,16 @@ phys_addr_t __init_memblock memblock_find_in_range(phys_addr_t start,
return ret;
}
+phys_addr_t __init_memblock __memblock_find_in_range(phys_addr_t start,
+ phys_addr_t end, phys_addr_t size,
+ phys_addr_t align)
+{
+ if (IS_ENABLED(CONFIG_PKRAM))
+ return pkram_memblock_find_in_range(start, end, size, align);
+ else
+ return memblock_find_in_range(start, end, size, align);
+}
+
static void __init_memblock memblock_remove_region(struct memblock_type *type, unsigned long r)
{
type->total_size -= type->regions[r].size;
@@ -447,11 +458,11 @@ static int __init_memblock memblock_double_array(struct memblock_type *type,
if (type != &memblock.reserved)
new_area_start = new_area_size = 0;
- addr = memblock_find_in_range(new_area_start + new_area_size,
+ addr = __memblock_find_in_range(new_area_start + new_area_size,
memblock.current_limit,
new_alloc_size, PAGE_SIZE);
if (!addr && new_area_size)
- addr = memblock_find_in_range(0,
+ addr = __memblock_find_in_range(0,
min(new_area_start, memblock.current_limit),
new_alloc_size, PAGE_SIZE);
diff --git a/mm/pkram.c b/mm/pkram.c
index dd3c89614010..e49c9bcd3854 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -1238,3 +1238,54 @@ void pkram_free_pgt(void)
__free_pages_core(virt_to_page(pkram_pgd), 0);
pkram_pgd = NULL;
}
+
+static int __init_memblock pkram_memblock_find_cb(struct pkram_pg_state *st, unsigned long base, unsigned long size)
+{
+ unsigned long end = base + size;
+ unsigned long addr;
+
+ if (size < st->min_size)
+ return 0;
+
+ addr = memblock_find_in_range(base, end, st->min_size, PAGE_SIZE);
+ if (!addr)
+ return 0;
+
+ st->retval = addr;
+ return 1;
+}
+
+/*
+ * It may be necessary to allocate a larger reserved memblock array
+ * while populating it with ranges of preserved pages. To avoid
+ * trampling preserved pages that have not yet been added to the
+ * memblock reserved list this function implements a wrapper around
+ * memblock_find_in_range() that restricts searches to subranges
+ * that do not contain preserved pages.
+ */
+phys_addr_t __init_memblock pkram_memblock_find_in_range(phys_addr_t start,
+ phys_addr_t end, phys_addr_t size,
+ phys_addr_t align)
+{
+ struct pkram_pg_state st = {
+ .range_cb = pkram_memblock_find_cb,
+ .min_addr = start,
+ .max_addr = end,
+ .min_size = PAGE_ALIGN(size),
+ .find_holes = true,
+ };
+
+ if (!pkram_reservation_in_progress)
+ return memblock_find_in_range(start, end, size, align);
+
+ if (!pkram_pgd) {
+ WARN_ONCE(1, "No preserved pages pagetable\n");
+ return memblock_find_in_range(start, end, size, align);
+ }
+
+ WARN_ONCE(memblock_bottom_up(), "PKRAM: bottom up memblock allocation not yet supported\n");
+
+ pkram_walk_pgt_rev(&st, pkram_pgd);
+
+ return st.retval;
+}
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* Re: [RFC 14/43] mm: memblock: PKRAM: prevent memblock resize from clobbering preserved pages
2020-05-07 0:41 ` [RFC 14/43] mm: memblock: PKRAM: prevent memblock resize from clobbering preserved pages Anthony Yznaga
@ 2020-05-11 13:57 ` Mike Rapoport
2020-05-11 23:29 ` Anthony Yznaga
0 siblings, 1 reply; 50+ messages in thread
From: Mike Rapoport @ 2020-05-11 13:57 UTC (permalink / raw)
To: Anthony Yznaga
Cc: linux-mm, linux-kernel, willy, corbet, tglx, mingo, bp, x86, hpa,
dave.hansen, luto, peterz, akpm, hughd, ebiederm, masahiroy,
ardb, ndesaulniers, dima, daniel.kiper, nivedita,
rafael.j.wysocki, dan.j.williams, zhenzhong.duan, jroedel, bhe,
guro, Thomas.Lendacky, andriy.shevchenko, keescook, hannes,
minchan, mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq,
vdavydov.dev, jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li,
paul.c.lai, ashok.raj, linux-fsdevel, linux-doc, kexec
On Wed, May 06, 2020 at 05:41:40PM -0700, Anthony Yznaga wrote:
> The size of the memblock reserved array may be increased while preserved
> pages are being reserved. When this happens, preserved pages that have
> not yet been reserved are at risk for being clobbered when space for a
> larger array is allocated.
> When called from memblock_double_array(), a wrapper around
> memblock_find_in_range() walks the preserved pages pagetable to find
> sufficiently sized ranges without preserved pages and passes them to
> memblock_find_in_range().
I'd suggest to create an array of memblock_region's that will contain
the PKRAM ranges before kexec and pass this array to the new kernel.
Then, somewhere in start_kerenel() replace replace
memblock.reserved->regions with that array.
> Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
> ---
> include/linux/pkram.h | 3 +++
> mm/memblock.c | 15 +++++++++++++--
> mm/pkram.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++
> 3 files changed, 67 insertions(+), 2 deletions(-)
>
> diff --git a/include/linux/pkram.h b/include/linux/pkram.h
> index edc5d8bef9d3..409022e1472f 100644
> --- a/include/linux/pkram.h
> +++ b/include/linux/pkram.h
> @@ -62,6 +62,9 @@ struct page *pkram_load_page(struct pkram_stream *ps, unsigned long *index,
> ssize_t pkram_write(struct pkram_stream *ps, const void *buf, size_t count);
> size_t pkram_read(struct pkram_stream *ps, void *buf, size_t count);
>
> +phys_addr_t pkram_memblock_find_in_range(phys_addr_t start, phys_addr_t end,
> + phys_addr_t size, phys_addr_t align);
> +
> #ifdef CONFIG_PKRAM
> extern unsigned long pkram_reserved_pages;
> void pkram_reserve(void);
> diff --git a/mm/memblock.c b/mm/memblock.c
> index c79ba6f9920c..69ae883b8d21 100644
> --- a/mm/memblock.c
> +++ b/mm/memblock.c
> @@ -16,6 +16,7 @@
> #include <linux/kmemleak.h>
> #include <linux/seq_file.h>
> #include <linux/memblock.h>
> +#include <linux/pkram.h>
>
> #include <asm/sections.h>
> #include <linux/io.h>
> @@ -349,6 +350,16 @@ phys_addr_t __init_memblock memblock_find_in_range(phys_addr_t start,
> return ret;
> }
>
> +phys_addr_t __init_memblock __memblock_find_in_range(phys_addr_t start,
> + phys_addr_t end, phys_addr_t size,
> + phys_addr_t align)
> +{
> + if (IS_ENABLED(CONFIG_PKRAM))
> + return pkram_memblock_find_in_range(start, end, size, align);
> + else
> + return memblock_find_in_range(start, end, size, align);
> +}
> +
> static void __init_memblock memblock_remove_region(struct memblock_type *type, unsigned long r)
> {
> type->total_size -= type->regions[r].size;
> @@ -447,11 +458,11 @@ static int __init_memblock memblock_double_array(struct memblock_type *type,
> if (type != &memblock.reserved)
> new_area_start = new_area_size = 0;
>
> - addr = memblock_find_in_range(new_area_start + new_area_size,
> + addr = __memblock_find_in_range(new_area_start + new_area_size,
> memblock.current_limit,
> new_alloc_size, PAGE_SIZE);
> if (!addr && new_area_size)
> - addr = memblock_find_in_range(0,
> + addr = __memblock_find_in_range(0,
> min(new_area_start, memblock.current_limit),
> new_alloc_size, PAGE_SIZE);
>
> diff --git a/mm/pkram.c b/mm/pkram.c
> index dd3c89614010..e49c9bcd3854 100644
> --- a/mm/pkram.c
> +++ b/mm/pkram.c
> @@ -1238,3 +1238,54 @@ void pkram_free_pgt(void)
> __free_pages_core(virt_to_page(pkram_pgd), 0);
> pkram_pgd = NULL;
> }
> +
> +static int __init_memblock pkram_memblock_find_cb(struct pkram_pg_state *st, unsigned long base, unsigned long size)
> +{
> + unsigned long end = base + size;
> + unsigned long addr;
> +
> + if (size < st->min_size)
> + return 0;
> +
> + addr = memblock_find_in_range(base, end, st->min_size, PAGE_SIZE);
> + if (!addr)
> + return 0;
> +
> + st->retval = addr;
> + return 1;
> +}
> +
> +/*
> + * It may be necessary to allocate a larger reserved memblock array
> + * while populating it with ranges of preserved pages. To avoid
> + * trampling preserved pages that have not yet been added to the
> + * memblock reserved list this function implements a wrapper around
> + * memblock_find_in_range() that restricts searches to subranges
> + * that do not contain preserved pages.
> + */
> +phys_addr_t __init_memblock pkram_memblock_find_in_range(phys_addr_t start,
> + phys_addr_t end, phys_addr_t size,
> + phys_addr_t align)
> +{
> + struct pkram_pg_state st = {
> + .range_cb = pkram_memblock_find_cb,
> + .min_addr = start,
> + .max_addr = end,
> + .min_size = PAGE_ALIGN(size),
> + .find_holes = true,
> + };
> +
> + if (!pkram_reservation_in_progress)
> + return memblock_find_in_range(start, end, size, align);
> +
> + if (!pkram_pgd) {
> + WARN_ONCE(1, "No preserved pages pagetable\n");
> + return memblock_find_in_range(start, end, size, align);
> + }
> +
> + WARN_ONCE(memblock_bottom_up(), "PKRAM: bottom up memblock allocation not yet supported\n");
> +
> + pkram_walk_pgt_rev(&st, pkram_pgd);
> +
> + return st.retval;
> +}
> --
> 2.13.3
>
--
Sincerely yours,
Mike.
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [RFC 14/43] mm: memblock: PKRAM: prevent memblock resize from clobbering preserved pages
2020-05-11 13:57 ` Mike Rapoport
@ 2020-05-11 23:29 ` Anthony Yznaga
0 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-11 23:29 UTC (permalink / raw)
To: Mike Rapoport
Cc: linux-mm, linux-kernel, willy, corbet, tglx, mingo, bp, x86, hpa,
dave.hansen, luto, peterz, akpm, hughd, ebiederm, masahiroy,
ardb, ndesaulniers, dima, daniel.kiper, nivedita,
rafael.j.wysocki, dan.j.williams, zhenzhong.duan, jroedel, bhe,
guro, Thomas.Lendacky, andriy.shevchenko, keescook, hannes,
minchan, mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq,
vdavydov.dev, jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li,
paul.c.lai, ashok.raj, linux-fsdevel, linux-doc, kexec
On 5/11/20 6:57 AM, Mike Rapoport wrote:
> On Wed, May 06, 2020 at 05:41:40PM -0700, Anthony Yznaga wrote:
>> The size of the memblock reserved array may be increased while preserved
>> pages are being reserved. When this happens, preserved pages that have
>> not yet been reserved are at risk for being clobbered when space for a
>> larger array is allocated.
>> When called from memblock_double_array(), a wrapper around
>> memblock_find_in_range() walks the preserved pages pagetable to find
>> sufficiently sized ranges without preserved pages and passes them to
>> memblock_find_in_range().
> I'd suggest to create an array of memblock_region's that will contain
> the PKRAM ranges before kexec and pass this array to the new kernel.
> Then, somewhere in start_kerenel() replace replace
> memblock.reserved->regions with that array.
I'll look into doing this. Thanks!
Anthony
>
>> Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
>> ---
>> include/linux/pkram.h | 3 +++
>> mm/memblock.c | 15 +++++++++++++--
>> mm/pkram.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++
>> 3 files changed, 67 insertions(+), 2 deletions(-)
>>
>> diff --git a/include/linux/pkram.h b/include/linux/pkram.h
>> index edc5d8bef9d3..409022e1472f 100644
>> --- a/include/linux/pkram.h
>> +++ b/include/linux/pkram.h
>> @@ -62,6 +62,9 @@ struct page *pkram_load_page(struct pkram_stream *ps, unsigned long *index,
>> ssize_t pkram_write(struct pkram_stream *ps, const void *buf, size_t count);
>> size_t pkram_read(struct pkram_stream *ps, void *buf, size_t count);
>>
>> +phys_addr_t pkram_memblock_find_in_range(phys_addr_t start, phys_addr_t end,
>> + phys_addr_t size, phys_addr_t align);
>> +
>> #ifdef CONFIG_PKRAM
>> extern unsigned long pkram_reserved_pages;
>> void pkram_reserve(void);
>> diff --git a/mm/memblock.c b/mm/memblock.c
>> index c79ba6f9920c..69ae883b8d21 100644
>> --- a/mm/memblock.c
>> +++ b/mm/memblock.c
>> @@ -16,6 +16,7 @@
>> #include <linux/kmemleak.h>
>> #include <linux/seq_file.h>
>> #include <linux/memblock.h>
>> +#include <linux/pkram.h>
>>
>> #include <asm/sections.h>
>> #include <linux/io.h>
>> @@ -349,6 +350,16 @@ phys_addr_t __init_memblock memblock_find_in_range(phys_addr_t start,
>> return ret;
>> }
>>
>> +phys_addr_t __init_memblock __memblock_find_in_range(phys_addr_t start,
>> + phys_addr_t end, phys_addr_t size,
>> + phys_addr_t align)
>> +{
>> + if (IS_ENABLED(CONFIG_PKRAM))
>> + return pkram_memblock_find_in_range(start, end, size, align);
>> + else
>> + return memblock_find_in_range(start, end, size, align);
>> +}
>> +
>> static void __init_memblock memblock_remove_region(struct memblock_type *type, unsigned long r)
>> {
>> type->total_size -= type->regions[r].size;
>> @@ -447,11 +458,11 @@ static int __init_memblock memblock_double_array(struct memblock_type *type,
>> if (type != &memblock.reserved)
>> new_area_start = new_area_size = 0;
>>
>> - addr = memblock_find_in_range(new_area_start + new_area_size,
>> + addr = __memblock_find_in_range(new_area_start + new_area_size,
>> memblock.current_limit,
>> new_alloc_size, PAGE_SIZE);
>> if (!addr && new_area_size)
>> - addr = memblock_find_in_range(0,
>> + addr = __memblock_find_in_range(0,
>> min(new_area_start, memblock.current_limit),
>> new_alloc_size, PAGE_SIZE);
>>
>> diff --git a/mm/pkram.c b/mm/pkram.c
>> index dd3c89614010..e49c9bcd3854 100644
>> --- a/mm/pkram.c
>> +++ b/mm/pkram.c
>> @@ -1238,3 +1238,54 @@ void pkram_free_pgt(void)
>> __free_pages_core(virt_to_page(pkram_pgd), 0);
>> pkram_pgd = NULL;
>> }
>> +
>> +static int __init_memblock pkram_memblock_find_cb(struct pkram_pg_state *st, unsigned long base, unsigned long size)
>> +{
>> + unsigned long end = base + size;
>> + unsigned long addr;
>> +
>> + if (size < st->min_size)
>> + return 0;
>> +
>> + addr = memblock_find_in_range(base, end, st->min_size, PAGE_SIZE);
>> + if (!addr)
>> + return 0;
>> +
>> + st->retval = addr;
>> + return 1;
>> +}
>> +
>> +/*
>> + * It may be necessary to allocate a larger reserved memblock array
>> + * while populating it with ranges of preserved pages. To avoid
>> + * trampling preserved pages that have not yet been added to the
>> + * memblock reserved list this function implements a wrapper around
>> + * memblock_find_in_range() that restricts searches to subranges
>> + * that do not contain preserved pages.
>> + */
>> +phys_addr_t __init_memblock pkram_memblock_find_in_range(phys_addr_t start,
>> + phys_addr_t end, phys_addr_t size,
>> + phys_addr_t align)
>> +{
>> + struct pkram_pg_state st = {
>> + .range_cb = pkram_memblock_find_cb,
>> + .min_addr = start,
>> + .max_addr = end,
>> + .min_size = PAGE_ALIGN(size),
>> + .find_holes = true,
>> + };
>> +
>> + if (!pkram_reservation_in_progress)
>> + return memblock_find_in_range(start, end, size, align);
>> +
>> + if (!pkram_pgd) {
>> + WARN_ONCE(1, "No preserved pages pagetable\n");
>> + return memblock_find_in_range(start, end, size, align);
>> + }
>> +
>> + WARN_ONCE(memblock_bottom_up(), "PKRAM: bottom up memblock allocation not yet supported\n");
>> +
>> + pkram_walk_pgt_rev(&st, pkram_pgd);
>> +
>> + return st.retval;
>> +}
>> --
>> 2.13.3
>>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [RFC 15/43] PKRAM: provide a way to ban pages from use by PKRAM
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (13 preceding siblings ...)
2020-05-07 0:41 ` [RFC 14/43] mm: memblock: PKRAM: prevent memblock resize from clobbering preserved pages Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 16/43] kexec: PKRAM: prevent kexec clobbering preserved pages in some cases Anthony Yznaga
` (27 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Not all memory ranges can be used for saving preserved over-kexec data.
For example, a kexec kernel may be loaded before pages are preserved.
The memory regions where the kexec segments will be copied to on kexec
must not contain preserved pages or else they will be clobbered.
Originally-by: Vladimir Davydov <vdavydov.dev@gmail.com>
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/pkram.h | 2 +
mm/pkram.c | 210 ++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 212 insertions(+)
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index 409022e1472f..1ba48442ef8e 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -69,10 +69,12 @@ phys_addr_t pkram_memblock_find_in_range(phys_addr_t start, phys_addr_t end,
extern unsigned long pkram_reserved_pages;
void pkram_reserve(void);
void pkram_free_pgt(void);
+void pkram_ban_region(unsigned long start, unsigned long end);
#else
#define pkram_reserved_pages 0UL
static inline void pkram_reserve(void) { }
static inline void pkram_free_pgt(void) { }
+static inline void pkram_ban_region(unsigned long start, unsigned long end) { }
#endif
#endif /* _LINUX_PKRAM_H */
diff --git a/mm/pkram.c b/mm/pkram.c
index e49c9bcd3854..60863c8ecbab 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -119,6 +119,28 @@ unsigned long __initdata pkram_reserved_pages;
static bool pkram_reservation_in_progress;
/*
+ * For tracking a region of memory that PKRAM is not allowed to use.
+ */
+struct banned_region {
+ unsigned long start, end; /* pfn, inclusive */
+};
+
+#define MAX_NR_BANNED (32 + MAX_NUMNODES * 2)
+
+static unsigned int nr_banned; /* number of banned regions */
+
+/* banned regions; arranged in ascending order, do not overlap */
+static struct banned_region banned[MAX_NR_BANNED];
+/*
+ * If a page allocated for PKRAM turns out to belong to a banned region,
+ * it is placed on the banned_pages list so subsequent allocation attempts
+ * do not encounter it again. The list is shrunk when system memory is low.
+ */
+static LIST_HEAD(banned_pages); /* linked through page::lru */
+static DEFINE_SPINLOCK(banned_pages_lock);
+static unsigned long nr_banned_pages;
+
+/*
* The PKRAM super block pfn, see above.
*/
static int __init parse_pkram_sb_pfn(char *arg)
@@ -223,12 +245,120 @@ void __init pkram_reserve(void)
pr_info("PKRAM: %lu pages reserved\n", pkram_reserved_pages);
}
+/*
+ * Ban pfn range [start..end] (inclusive) from use in PKRAM.
+ */
+void pkram_ban_region(unsigned long start, unsigned long end)
+{
+ int i, merged = -1;
+
+ if (pkram_reservation_in_progress)
+ return;
+
+ /* first try to merge the region with an existing one */
+ for (i = nr_banned - 1; i >= 0 && start <= banned[i].end + 1; i--) {
+ if (end + 1 >= banned[i].start) {
+ start = min(banned[i].start, start);
+ end = max(banned[i].end, end);
+ if (merged < 0)
+ merged = i;
+ } else
+ /*
+ * Regions are arranged in ascending order and do not
+ * intersect so the merged region cannot jump over its
+ * predecessors.
+ */
+ BUG_ON(merged >= 0);
+ }
+
+ i++;
+
+ if (merged >= 0) {
+ banned[i].start = start;
+ banned[i].end = end;
+ /* shift if merged with more than one region */
+ memmove(banned + i + 1, banned + merged + 1,
+ sizeof(*banned) * (nr_banned - merged - 1));
+ nr_banned -= merged - i;
+ return;
+ }
+
+ /*
+ * The region does not intersect with an existing one;
+ * try to create a new one.
+ */
+ if (nr_banned == MAX_NR_BANNED) {
+ pr_err("PKRAM: Failed to ban %lu-%lu: "
+ "Too many banned regions\n", start, end);
+ return;
+ }
+
+ memmove(banned + i + 1, banned + i,
+ sizeof(*banned) * (nr_banned - i));
+ banned[i].start = start;
+ banned[i].end = end;
+ nr_banned++;
+}
+
+static void pkram_show_banned(void)
+{
+ int i;
+ unsigned long n, total = 0;
+
+ pr_info("PKRAM: banned regions:\n");
+ for (i = 0; i < nr_banned; i++) {
+ n = banned[i].end - banned[i].start + 1;
+ pr_info("%4d: [%08lx - %08lx] %ld pages\n",
+ i, banned[i].start, banned[i].end, n);
+ total += n;
+ }
+ pr_info("Total banned: %ld pages in %d regions\n",
+ total, nr_banned);
+}
+
+/*
+ * Returns true if the page may not be used for storing preserved data.
+ */
+static bool pkram_page_banned(struct page *page)
+{
+ unsigned long epfn, pfn = page_to_pfn(page);
+ int l = 0, r = nr_banned - 1, m;
+
+ epfn = pfn + compound_nr(page) - 1;
+
+ /* do binary search */
+ while (l <= r) {
+ m = (l + r) / 2;
+ if (epfn < banned[m].start)
+ r = m - 1;
+ else if (pfn > banned[m].end)
+ l = m + 1;
+ else
+ return true;
+ }
+ return false;
+}
+
static inline struct page *__pkram_alloc_page(gfp_t gfp_mask, bool add_to_map)
{
struct page *page;
+ LIST_HEAD(list);
+ unsigned long len = 0;
int err;
page = alloc_page(gfp_mask);
+ while (page && pkram_page_banned(page)) {
+ len++;
+ list_add(&page->lru, &list);
+ page = alloc_page(gfp_mask);
+ }
+ if (len > 0) {
+ spin_lock(&banned_pages_lock);
+ nr_banned_pages += len;
+ list_splice(&list, &banned_pages);
+ spin_unlock(&banned_pages_lock);
+ }
+
if (page && add_to_map) {
err = pkram_add_identity_map(page);
if (err) {
@@ -256,6 +386,53 @@ static inline void pkram_free_page(void *addr)
free_page((unsigned long)addr);
}
+static void __banned_pages_shrink(unsigned long nr_to_scan)
+{
+ struct page *page;
+
+ if (nr_to_scan <= 0)
+ return;
+
+ while (nr_banned_pages > 0) {
+ BUG_ON(list_empty(&banned_pages));
+ page = list_first_entry(&banned_pages, struct page, lru);
+ list_del(&page->lru);
+ __free_page(page);
+ nr_banned_pages--;
+ nr_to_scan--;
+ if (!nr_to_scan)
+ break;
+ }
+}
+
+static unsigned long
+banned_pages_count(struct shrinker *shrink, struct shrink_control *sc)
+{
+ return nr_banned_pages;
+}
+
+static unsigned long
+banned_pages_scan(struct shrinker *shrink, struct shrink_control *sc)
+{
+ int nr_left = nr_banned_pages;
+
+ if (!sc->nr_to_scan || !nr_left)
+ return nr_left;
+
+ spin_lock(&banned_pages_lock);
+ __banned_pages_shrink(sc->nr_to_scan);
+ nr_left = nr_banned_pages;
+ spin_unlock(&banned_pages_lock);
+
+ return nr_left;
+}
+
+static struct shrinker banned_pages_shrinker = {
+ .count_objects = banned_pages_count,
+ .scan_objects = banned_pages_scan,
+ .seeks = DEFAULT_SEEKS,
+};
+
static inline void pkram_insert_node(struct pkram_node *node)
{
list_add(&virt_to_page(node)->lru, &pkram_nodes);
@@ -665,6 +842,32 @@ static int __pkram_save_page(struct pkram_stream *ps,
return 0;
}
+static int __pkram_save_page_copy(struct pkram_stream *ps, struct page *page,
+ short flags)
+{
+ int nr_pages = compound_nr(page);
+ pgoff_t index = page->index;
+ int i, err;
+
+ for (i = 0; i < nr_pages; i++, index++) {
+ struct page *p = page + i;
+ struct page *new;
+
+ new = pkram_alloc_page(ps->gfp_mask);
+ if (!new)
+ return -ENOMEM;
+
+ copy_highpage(new, p);
+ err = __pkram_save_page(ps, new, flags, index);
+ put_page(new);
+
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
/**
* Save page @page to the preserved memory node and object associated with
* stream @ps. The stream must have been initialized with pkram_prepare_save()
@@ -688,6 +891,10 @@ int pkram_save_page(struct pkram_stream *ps, struct page *page, short flags)
BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_SAVE);
+ /* if page is banned, relocate it */
+ if (pkram_page_banned(page))
+ return __pkram_save_page_copy(ps, page, flags);
+
err = __pkram_save_page(ps, page, flags, page->index);
if (!err)
err = pkram_add_identity_map(page);
@@ -891,6 +1098,7 @@ static void __pkram_reboot(void)
unsigned long pgd_pfn = 0;
if (pkram_pgd) {
+ pkram_show_banned();
list_for_each_entry_reverse(page, &pkram_nodes, lru) {
node = page_address(page);
if (WARN_ON(node->flags & PKRAM_ACCMODE_MASK))
@@ -957,6 +1165,7 @@ static int __init pkram_init_sb(void)
page = __pkram_alloc_page(GFP_KERNEL | __GFP_ZERO, false);
if (!page) {
pr_err("PKRAM: Failed to allocate super block\n");
+ __banned_pages_shrink(ULONG_MAX);
return 0;
}
pkram_sb = page_address(page);
@@ -979,6 +1188,7 @@ static int __init pkram_init(void)
{
if (pkram_init_sb()) {
register_reboot_notifier(&pkram_reboot_notifier);
+ register_shrinker(&banned_pages_shrinker);
sysfs_update_group(kernel_kobj, &pkram_attr_group);
}
return 0;
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 16/43] kexec: PKRAM: prevent kexec clobbering preserved pages in some cases
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (14 preceding siblings ...)
2020-05-07 0:41 ` [RFC 15/43] PKRAM: provide a way to ban pages from use by PKRAM Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 17/43] PKRAM: provide a way to check if a memory range has preserved pages Anthony Yznaga
` (26 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
When loading a kernel for kexec, dynamically update the list of physical
ranges that are not to be used for storing preserved pages with the ranges
where kexec segments will be copied to on reboot. This ensures no pages
preserved after the new kernel has been loaded will reside in these ranges
on reboot.
Not yet handled is the case where pages have been preserved before a
kexec kernel is loaded. This will be covered by a later patch.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
kernel/kexec.c | 9 +++++++++
kernel/kexec_file.c | 10 ++++++++++
2 files changed, 19 insertions(+)
diff --git a/kernel/kexec.c b/kernel/kexec.c
index f977786fe498..c44598fc42a1 100644
--- a/kernel/kexec.c
+++ b/kernel/kexec.c
@@ -16,6 +16,7 @@
#include <linux/syscalls.h>
#include <linux/vmalloc.h>
#include <linux/slab.h>
+#include <linux/pkram.h>
#include "kexec_internal.h"
@@ -163,6 +164,14 @@ static int do_kexec_load(unsigned long entry, unsigned long nr_segments,
if (ret)
goto out;
+ for (i = 0; i < nr_segments; i++) {
+ unsigned long mem = image->segment[i].mem;
+ size_t memsz = image->segment[i].memsz;
+
+ if (memsz)
+ pkram_ban_region(PFN_DOWN(mem), PFN_UP(mem + memsz) - 1);
+ }
+
/* Install the new kernel and uninstall the old */
image = xchg(dest_image, image);
diff --git a/kernel/kexec_file.c b/kernel/kexec_file.c
index faa74d5f6941..f57f72237859 100644
--- a/kernel/kexec_file.c
+++ b/kernel/kexec_file.c
@@ -26,6 +26,8 @@
#include <linux/kernel.h>
#include <linux/syscalls.h>
#include <linux/vmalloc.h>
+#include <linux/pkram.h>
+
#include "kexec_internal.h"
static int kexec_calculate_store_digests(struct kimage *image);
@@ -445,6 +447,14 @@ SYSCALL_DEFINE5(kexec_file_load, int, kernel_fd, int, initrd_fd,
if (ret)
goto out;
+ for (i = 0; i < image->nr_segments; i++) {
+ unsigned long mem = image->segment[i].mem;
+ size_t memsz = image->segment[i].memsz;
+
+ if (memsz)
+ pkram_ban_region(PFN_DOWN(mem), PFN_UP(mem + memsz) - 1);
+ }
+
/*
* Free up any temporary buffers allocated which are not needed
* after image has been loaded
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 17/43] PKRAM: provide a way to check if a memory range has preserved pages
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (15 preceding siblings ...)
2020-05-07 0:41 ` [RFC 16/43] kexec: PKRAM: prevent kexec clobbering preserved pages in some cases Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 18/43] kexec: PKRAM: avoid clobbering already " Anthony Yznaga
` (25 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
When a kernel is loaded for kexec the address ranges where the kexec
segments will be copied to may conflict with pages already set to be
preserved. Provide a way to determine if preserved pages exist in a
specified range.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/pkram.h | 2 ++
mm/pkram.c | 25 +++++++++++++++++++++++++
2 files changed, 27 insertions(+)
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index 1ba48442ef8e..1cd518843d7a 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -70,11 +70,13 @@ extern unsigned long pkram_reserved_pages;
void pkram_reserve(void);
void pkram_free_pgt(void);
void pkram_ban_region(unsigned long start, unsigned long end);
+int pkram_has_preserved_pages(unsigned long start, unsigned long end);
#else
#define pkram_reserved_pages 0UL
static inline void pkram_reserve(void) { }
static inline void pkram_free_pgt(void) { }
static inline void pkram_ban_region(unsigned long start, unsigned long end) { }
+static inline int pkram_has_preserved_pages(unsigned long start, unsigned long end) { return 0; }
#endif
#endif /* _LINUX_PKRAM_H */
diff --git a/mm/pkram.c b/mm/pkram.c
index 60863c8ecbab..0aaaf9b79682 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -1499,3 +1499,28 @@ phys_addr_t __init_memblock pkram_memblock_find_in_range(phys_addr_t start,
return st.retval;
}
+
+static int pkram_has_preserved_pages_cb(struct pkram_pg_state *st, unsigned long base, unsigned long size)
+{
+ st->retval = 1;
+ return 1;
+}
+
+/*
+ * Check whether the memory range [start, end) contains preserved pages.
+ */
+int pkram_has_preserved_pages(unsigned long start, unsigned long end)
+{
+ struct pkram_pg_state st = {
+ .range_cb = pkram_has_preserved_pages_cb,
+ .min_addr = start,
+ .max_addr = end,
+ };
+
+ if (!pkram_pgd)
+ return 0;
+
+ pkram_walk_pgt_rev(&st, pkram_pgd);
+
+ return st.retval;
+}
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 18/43] kexec: PKRAM: avoid clobbering already preserved pages
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (16 preceding siblings ...)
2020-05-07 0:41 ` [RFC 17/43] PKRAM: provide a way to check if a memory range has preserved pages Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 19/43] mm: PKRAM: allow preserved memory to be freed from userspace Anthony Yznaga
` (24 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Ensure destination ranges of the kexec segments do not overlap
with any kernel pages marked to be preserved across kexec.
For kexec_load, return EADDRNOTAVAIL if overlap is detected.
For kexec_file_load, skip ranges containing preserved pages when
seaching for available ranges to use.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
kernel/kexec_core.c | 3 +++
kernel/kexec_file.c | 5 +++++
2 files changed, 8 insertions(+)
diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c
index c19c0dad1ebe..8c24b546352e 100644
--- a/kernel/kexec_core.c
+++ b/kernel/kexec_core.c
@@ -37,6 +37,7 @@
#include <linux/compiler.h>
#include <linux/hugetlb.h>
#include <linux/frame.h>
+#include <linux/pkram.h>
#include <asm/page.h>
#include <asm/sections.h>
@@ -176,6 +177,8 @@ int sanity_check_segment_list(struct kimage *image)
return -EADDRNOTAVAIL;
if (mend >= KEXEC_DESTINATION_MEMORY_LIMIT)
return -EADDRNOTAVAIL;
+ if (pkram_has_preserved_pages(mstart, mend))
+ return -EADDRNOTAVAIL;
}
/* Verify our destination addresses do not overlap.
diff --git a/kernel/kexec_file.c b/kernel/kexec_file.c
index f57f72237859..7b14e1b1a178 100644
--- a/kernel/kexec_file.c
+++ b/kernel/kexec_file.c
@@ -498,6 +498,11 @@ static int locate_mem_hole_top_down(unsigned long start, unsigned long end,
continue;
}
+ if (pkram_has_preserved_pages(temp_start, temp_end + 1)) {
+ temp_start = temp_start - PAGE_SIZE;
+ continue;
+ }
+
/* We found a suitable memory range */
break;
} while (1);
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 19/43] mm: PKRAM: allow preserved memory to be freed from userspace
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (17 preceding siblings ...)
2020-05-07 0:41 ` [RFC 18/43] kexec: PKRAM: avoid clobbering already " Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 20/43] PKRAM: disable feature when running the kdump kernel Anthony Yznaga
` (23 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
To free all space utilized for preserved memory, one can write 0 to
/sys/kernel/pkram. This will destroy all PKRAM nodes that are not
currently being read or written.
Originally-by: Vladimir Davydov <vdavydov.dev@gmail.com>
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
mm/pkram.c | 39 ++++++++++++++++++++++++++++++++++++++-
1 file changed, 38 insertions(+), 1 deletion(-)
diff --git a/mm/pkram.c b/mm/pkram.c
index 0aaaf9b79682..95e691382721 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -509,6 +509,32 @@ static void pkram_truncate_node(struct pkram_node *node)
node->obj_pfn = 0;
}
+/*
+ * Free all nodes that are not under operation.
+ */
+static void pkram_truncate(void)
+{
+ struct page *page, *tmp;
+ struct pkram_node *node;
+ LIST_HEAD(dispose);
+
+ mutex_lock(&pkram_mutex);
+ list_for_each_entry_safe(page, tmp, &pkram_nodes, lru) {
+ node = page_address(page);
+ if (!(node->flags & PKRAM_ACCMODE_MASK))
+ list_move(&page->lru, &dispose);
+ }
+ mutex_unlock(&pkram_mutex);
+
+ while (!list_empty(&dispose)) {
+ page = list_first_entry(&dispose, struct page, lru);
+ list_del(&page->lru);
+ node = page_address(page);
+ pkram_truncate_node(node);
+ pkram_free_page(node);
+ }
+}
+
static void pkram_add_link(struct pkram_link *link, struct pkram_obj *obj)
{
link->link_pfn = obj->link_pfn;
@@ -1141,8 +1167,19 @@ static ssize_t show_pkram_sb_pfn(struct kobject *kobj,
return sprintf(buf, "%lx\n", pfn);
}
+static ssize_t store_pkram_sb_pfn(struct kobject *kobj,
+ struct kobj_attribute *attr, const char *buf, size_t count)
+{
+ int val;
+
+ if (kstrtoint(buf, 0, &val) || val)
+ return -EINVAL;
+ pkram_truncate();
+ return count;
+}
+
static struct kobj_attribute pkram_sb_pfn_attr =
- __ATTR(pkram, 0444, show_pkram_sb_pfn, NULL);
+ __ATTR(pkram, 0644, show_pkram_sb_pfn, store_pkram_sb_pfn);
static struct attribute *pkram_attrs[] = {
&pkram_sb_pfn_attr.attr,
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 20/43] PKRAM: disable feature when running the kdump kernel
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (18 preceding siblings ...)
2020-05-07 0:41 ` [RFC 19/43] mm: PKRAM: allow preserved memory to be freed from userspace Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 21/43] x86/KASLR: PKRAM: support physical kaslr Anthony Yznaga
` (22 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
The kdump kernel should not preserve or restore pages.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
mm/pkram.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/mm/pkram.c b/mm/pkram.c
index 95e691382721..4d4d836fea53 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -1,4 +1,5 @@
// SPDX-License-Identifier: GPL-2.0
+#include <linux/crash_dump.h>
#include <linux/err.h>
#include <linux/gfp.h>
#include <linux/highmem.h>
@@ -193,7 +194,7 @@ void __init pkram_reserve(void)
{
int err = 0;
- if (!pkram_sb_pfn)
+ if (!pkram_sb_pfn || is_kdump_kernel())
return;
pr_info("PKRAM: Examining preserved memory...\n");
@@ -305,6 +306,9 @@ static void pkram_show_banned(void)
int i;
unsigned long n, total = 0;
+ if (is_kdump_kernel())
+ return;
+
pr_info("PKRAM: banned regions:\n");
for (i = 0; i < nr_banned; i++) {
n = banned[i].end - banned[i].start + 1;
@@ -1223,7 +1227,7 @@ static int __init pkram_init_sb(void)
static int __init pkram_init(void)
{
- if (pkram_init_sb()) {
+ if (!is_kdump_kernel() && pkram_init_sb()) {
register_reboot_notifier(&pkram_reboot_notifier);
register_shrinker(&banned_pages_shrinker);
sysfs_update_group(kernel_kobj, &pkram_attr_group);
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 21/43] x86/KASLR: PKRAM: support physical kaslr
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (19 preceding siblings ...)
2020-05-07 0:41 ` [RFC 20/43] PKRAM: disable feature when running the kdump kernel Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 17:51 ` Kees Cook
2020-05-07 0:41 ` [RFC 22/43] mm: shmem: introduce shmem_insert_page Anthony Yznaga
` (21 subsequent siblings)
42 siblings, 1 reply; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Avoid regions of memory that contain preserved pages when computing
slots used to select where to put the decompressed kernel.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
arch/x86/boot/compressed/Makefile | 3 +
arch/x86/boot/compressed/kaslr.c | 67 ++++++----
arch/x86/boot/compressed/misc.h | 19 +++
arch/x86/boot/compressed/pkram.c | 252 ++++++++++++++++++++++++++++++++++++++
4 files changed, 320 insertions(+), 21 deletions(-)
create mode 100644 arch/x86/boot/compressed/pkram.c
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index 5f7c262bcc99..ba0d76c53574 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -84,6 +84,9 @@ ifdef CONFIG_X86_64
vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr_64.o
vmlinux-objs-y += $(obj)/mem_encrypt.o
vmlinux-objs-y += $(obj)/pgtable_64.o
+ifdef CONFIG_RANDOMIZE_BASE
+ vmlinux-objs-$(CONFIG_PKRAM) += $(obj)/pkram.o
+endif
endif
vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o
diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c
index d7408af55738..3f0a6fb15ac2 100644
--- a/arch/x86/boot/compressed/kaslr.c
+++ b/arch/x86/boot/compressed/kaslr.c
@@ -613,31 +613,16 @@ static unsigned long slots_fetch_random(void)
return 0;
}
-static void __process_mem_region(struct mem_vector *entry,
- unsigned long minimum,
- unsigned long image_size)
+void ___process_mem_region(struct mem_vector *entry,
+ unsigned long minimum,
+ unsigned long image_size)
{
struct mem_vector region, overlap;
- unsigned long start_orig, end;
+ unsigned long start_orig;
struct mem_vector cur_entry;
- /* On 32-bit, ignore entries entirely above our maximum. */
- if (IS_ENABLED(CONFIG_X86_32) && entry->start >= KERNEL_IMAGE_SIZE)
- return;
-
- /* Ignore entries entirely below our minimum. */
- if (entry->start + entry->size < minimum)
- return;
-
- /* Ignore entries above memory limit */
- end = min(entry->size + entry->start, mem_limit);
- if (entry->start >= end)
- return;
- cur_entry.start = entry->start;
- cur_entry.size = end - entry->start;
-
- region.start = cur_entry.start;
- region.size = cur_entry.size;
+ region.start = cur_entry.start = entry->start;
+ region.size = cur_entry.size = entry->size;
/* Give up if slot area array is full. */
while (slot_area_index < MAX_SLOT_AREA) {
@@ -691,6 +676,39 @@ static void __process_mem_region(struct mem_vector *entry,
}
}
+static void __process_mem_region(struct mem_vector *entry,
+ unsigned long minimum,
+ unsigned long image_size)
+{
+ struct mem_vector region, overlap;
+ unsigned long start_orig, end;
+ struct mem_vector cur_entry;
+
+ /* On 32-bit, ignore entries entirely above our maximum. */
+ if (IS_ENABLED(CONFIG_X86_32) && entry->start >= KERNEL_IMAGE_SIZE)
+ return;
+
+ /* Ignore entries entirely below our minimum. */
+ if (entry->start + entry->size < minimum)
+ return;
+
+ /* Ignore entries above memory limit */
+ end = min(entry->size + entry->start, mem_limit);
+ if (entry->start >= end)
+ return;
+ cur_entry.start = entry->start;
+ cur_entry.size = end - entry->start;
+
+ /* Return if region can't contain decompressed kernel */
+ if (cur_entry.size < image_size)
+ return;
+
+ if (pkram_enabled())
+ return pkram_process_mem_region(&cur_entry, minimum, image_size);
+ else
+ return ___process_mem_region(&cur_entry, minimum, image_size);
+}
+
static bool process_mem_region(struct mem_vector *region,
unsigned long long minimum,
unsigned long long image_size)
@@ -902,6 +920,8 @@ void choose_random_location(unsigned long input,
return;
}
+ pkram_init();
+
#ifdef CONFIG_X86_5LEVEL
if (__read_cr4() & X86_CR4_LA57) {
__pgtable_l5_enabled = 1;
@@ -952,3 +972,8 @@ void choose_random_location(unsigned long input,
random_addr = find_random_virt_addr(LOAD_PHYSICAL_ADDR, output_size);
*virt_addr = random_addr;
}
+
+int slot_areas_full(void)
+{
+ return slot_area_index == MAX_SLOT_AREA;
+}
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 726e264410ff..ca1a8ae5ebe9 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -117,6 +117,25 @@ static inline void console_init(void)
{ }
#endif
+void ___process_mem_region(struct mem_vector *entry,
+ unsigned long minimum,
+ unsigned long image_size);
+
+#ifdef CONFIG_PKRAM
+void pkram_init(void);
+int pkram_enabled(void);
+void pkram_process_mem_region(struct mem_vector *entry,
+ unsigned long minimum,
+ unsigned long image_size);
+#else
+static inline void pkram_init(void) { }
+static inline int pkram_enabled(void) { return 0; }
+static inline void pkram_process_mem_region(struct mem_vector *entry,
+ unsigned long minimum,
+ unsigned long image_size)
+{ ___process_mem_region(entry, minimum, image_size); }
+#endif
+
void set_sev_encryption_mask(void);
/* acpi.c */
diff --git a/arch/x86/boot/compressed/pkram.c b/arch/x86/boot/compressed/pkram.c
new file mode 100644
index 000000000000..5fc1e26909de
--- /dev/null
+++ b/arch/x86/boot/compressed/pkram.c
@@ -0,0 +1,252 @@
+// SPDX-License-Identifier: GPL-2.0
+#define __pa(x) ((unsigned long)(x))
+#define __va(x) ((void *)((unsigned long)(x)))
+
+#include "misc.h"
+#include <asm/pgtable.h>
+
+struct pkram_super_block {
+ __u64 node_pfn;
+ __u64 pgd_pfn;
+};
+
+static unsigned long long pkram_sb_pfn;
+static struct pkram_super_block *pkram_sb;
+static pgd_t *pkram_pgd;
+
+struct pg_state {
+ int (*range_cb)(struct pg_state *state, unsigned long base,
+ unsigned long size);
+ unsigned long curr_addr;
+ unsigned long start_addr;
+ unsigned long min_addr;
+ unsigned long max_addr;
+ unsigned long min_size;
+ unsigned long minimum;
+ bool tracking;
+ bool find_holes;
+};
+
+int pkram_enabled(void)
+{
+ return pkram_pgd ? 1 : 0;
+}
+
+void pkram_init(void)
+{
+ char arg[32];
+
+ if (cmdline_find_option("pkram", arg, sizeof(arg)) > 0) {
+ if (kstrtoull(arg, 16, &pkram_sb_pfn) != 0)
+ return;
+ } else
+ return;
+
+ pkram_sb = (struct pkram_super_block *)(pkram_sb_pfn << PAGE_SHIFT);
+
+ if (pkram_sb)
+ pkram_pgd = (pgd_t *)(pkram_sb->pgd_pfn << PAGE_SHIFT);
+}
+
+static int note_page(struct pg_state *st, int present)
+{
+ unsigned long curr_addr = st->curr_addr;
+ bool track_page = present ^ st->find_holes;
+
+ if (!st->tracking && track_page) {
+ if (curr_addr >= st->max_addr)
+ return 1;
+ /*
+ * curr_addr can be < min_addr if the page straddles the
+ * boundary
+ */
+ st->start_addr = max(curr_addr, st->min_addr);
+ st->tracking = true;
+ } else if (st->tracking) {
+ unsigned long base, size;
+ int ret;
+
+ /* Continue tracking if upper bound has not been reached */
+ if (track_page && curr_addr < st->max_addr)
+ return 0;
+
+ curr_addr = min(curr_addr, st->max_addr);
+
+ base = st->start_addr;
+ size = curr_addr - st->start_addr;
+ st->tracking = false;
+
+ ret = st->range_cb(st, base, size);
+
+ if (curr_addr == st->max_addr)
+ return 1;
+ else
+ return ret;
+ }
+
+ return 0;
+}
+
+static int walk_pte_level(struct pg_state *st, pmd_t addr, unsigned long P)
+{
+ unsigned long *bitmap;
+ int present;
+ int i, ret;
+
+ bitmap = __va(pmd_val(addr));
+ for (i = 0; i < PTRS_PER_PTE; i++) {
+ unsigned long curr_addr = P + i * PAGE_SIZE;
+
+ if (curr_addr < st->min_addr)
+ continue;
+ st->curr_addr = curr_addr;
+ present = test_bit(i, bitmap);
+ ret = note_page(st, present);
+ if (ret)
+ break;
+ }
+
+ return ret;
+}
+
+static int walk_pmd_level(struct pg_state *st, pud_t addr, unsigned long P)
+{
+ pmd_t *start;
+ int i, ret;
+
+ start = (pmd_t *)pud_page_vaddr(addr);
+ for (i = 0; i < PTRS_PER_PMD; i++, start++) {
+ unsigned long curr_addr = P + i * PMD_SIZE;
+
+ if (curr_addr + PMD_SIZE <= st->min_addr)
+ continue;
+ st->curr_addr = curr_addr;
+ if (!pmd_none(*start)) {
+ if (pmd_large(*start))
+ ret = note_page(st, true);
+ else
+ ret = walk_pte_level(st, *start, curr_addr);
+ } else
+ ret = note_page(st, false);
+ if (ret)
+ break;
+ }
+
+ return ret;
+}
+
+static int walk_pud_level(struct pg_state *st, p4d_t addr, unsigned long P)
+{
+ pud_t *start;
+ int i, ret;
+
+ start = (pud_t *)p4d_page_vaddr(addr);
+ for (i = 0; i < PTRS_PER_PUD; i++, start++) {
+ unsigned long curr_addr = P + i * PUD_SIZE;
+
+ if (curr_addr + PUD_SIZE <= st->min_addr)
+ continue;
+ st->curr_addr = curr_addr;
+ if (!pud_none(*start)) {
+ if (pud_large(*start))
+ ret = note_page(st, true);
+ else
+ ret = walk_pmd_level(st, *start, curr_addr);
+ } else
+ ret = note_page(st, false);
+ if (ret)
+ break;
+ }
+
+ return ret;
+}
+
+static int walk_p4d_level(struct pg_state *st, pgd_t addr, unsigned long P)
+{
+ p4d_t *start;
+ int i, ret;
+
+ if (PTRS_PER_P4D == 1)
+ return walk_pud_level(st, __p4d(pgd_val(addr)), P);
+
+ start = (p4d_t *)pgd_page_vaddr(addr);
+ for (i = 0; i < PTRS_PER_P4D; i++, start++) {
+ unsigned long curr_addr = P + i * P4D_SIZE;
+
+ if (curr_addr + P4D_SIZE <= st->min_addr)
+ continue;
+ st->curr_addr = curr_addr;
+ if (!p4d_none(*start)) {
+ if (p4d_large(*start))
+ ret = note_page(st, true);
+ else
+ ret = walk_pud_level(st, *start, curr_addr);
+ } else
+ ret = note_page(st, false);
+ if (ret)
+ break;
+ }
+
+ return ret;
+}
+
+#define pgd_large(a) (pgtable_l5_enabled() ? pgd_large(a) : p4d_large(__p4d(pgd_val(a))))
+#define pgd_none(a) (pgtable_l5_enabled() ? pgd_none(a) : p4d_none(__p4d(pgd_val(a))))
+
+static int walk_pgd_level(struct pg_state *st, pgd_t *pgd)
+{
+ pgd_t *start = pgd;
+ int i, ret = 0;
+
+ for (i = 0; i < PTRS_PER_PGD; i++, start++) {
+ unsigned long curr_addr = i * PGDIR_SIZE;
+
+ if (curr_addr + PGDIR_SIZE <= st->min_addr)
+ continue;
+ st->curr_addr = curr_addr;
+ if (!pgd_none(*start))
+ ret = walk_p4d_level(st, *start, curr_addr);
+ else
+ ret = note_page(st, false);
+ if (ret)
+ break;
+ }
+
+ return ret;
+}
+
+extern int slot_areas_full(void);
+
+static int pkram_process_mem_region_cb(struct pg_state *st, unsigned long base, unsigned long size)
+{
+ struct mem_vector region = {
+ .start = base,
+ .size = size,
+ };
+
+ if (size < st->min_size)
+ return 0;
+
+ ___process_mem_region(®ion, st->minimum, st->min_size);
+
+ if (slot_areas_full())
+ return 1;
+
+ return 0;
+}
+
+void pkram_process_mem_region(struct mem_vector *entry,
+ unsigned long minimum,
+ unsigned long image_size)
+{
+ struct pg_state st = {
+ .range_cb = pkram_process_mem_region_cb,
+ .min_addr = max((unsigned long)entry->start, minimum),
+ .max_addr = entry->start + entry->size,
+ .min_size = image_size,
+ .minimum = minimum,
+ .find_holes = true,
+ };
+
+ walk_pgd_level(&st, pkram_pgd);
+}
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* Re: [RFC 21/43] x86/KASLR: PKRAM: support physical kaslr
2020-05-07 0:41 ` [RFC 21/43] x86/KASLR: PKRAM: support physical kaslr Anthony Yznaga
@ 2020-05-07 17:51 ` Kees Cook
2020-05-07 18:41 ` Anthony Yznaga
0 siblings, 1 reply; 50+ messages in thread
From: Kees Cook @ 2020-05-07 17:51 UTC (permalink / raw)
To: Anthony Yznaga
Cc: linux-mm, linux-kernel, willy, corbet, tglx, mingo, bp, x86, hpa,
dave.hansen, luto, peterz, rppt, akpm, hughd, ebiederm,
masahiroy, ardb, ndesaulniers, dima, daniel.kiper, nivedita,
rafael.j.wysocki, dan.j.williams, zhenzhong.duan, jroedel, bhe,
guro, Thomas.Lendacky, andriy.shevchenko, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
On Wed, May 06, 2020 at 05:41:47PM -0700, Anthony Yznaga wrote:
> Avoid regions of memory that contain preserved pages when computing
> slots used to select where to put the decompressed kernel.
This is changing the slot-walking code instead of updating
mem_avoid_overlap() -- that's where the check for a "reserved" memory
area should live.
For example, this is how both mem_avoid_memmap() and the setup_data
memory areas are handled.
Is there a reason mem_avoid_overlap() can't be used here?
--
Kees Cook
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [RFC 21/43] x86/KASLR: PKRAM: support physical kaslr
2020-05-07 17:51 ` Kees Cook
@ 2020-05-07 18:41 ` Anthony Yznaga
0 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 18:41 UTC (permalink / raw)
To: Kees Cook
Cc: linux-mm, linux-kernel, willy, corbet, tglx, mingo, bp, x86, hpa,
dave.hansen, luto, peterz, rppt, akpm, hughd, ebiederm,
masahiroy, ardb, ndesaulniers, dima, daniel.kiper, nivedita,
rafael.j.wysocki, dan.j.williams, zhenzhong.duan, jroedel, bhe,
guro, Thomas.Lendacky, andriy.shevchenko, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
On 5/7/20 10:51 AM, Kees Cook wrote:
> On Wed, May 06, 2020 at 05:41:47PM -0700, Anthony Yznaga wrote:
>> Avoid regions of memory that contain preserved pages when computing
>> slots used to select where to put the decompressed kernel.
> This is changing the slot-walking code instead of updating
> mem_avoid_overlap() -- that's where the check for a "reserved" memory
> area should live.
>
> For example, this is how both mem_avoid_memmap() and the setup_data
> memory areas are handled.
>
> Is there a reason mem_avoid_overlap() can't be used here?
>
I was thinking it would be more efficient to process just
the regions that did not contain preserved pages rather
than checking for preserved pages in mem_avoid_overlap(),
but I see that may just be adding unnecessary complexity.
I'll investigate modifying mem_avoid_overlap().
Thank you for the comments!
Anthony
^ permalink raw reply [flat|nested] 50+ messages in thread
* [RFC 22/43] mm: shmem: introduce shmem_insert_page
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (20 preceding siblings ...)
2020-05-07 0:41 ` [RFC 21/43] x86/KASLR: PKRAM: support physical kaslr Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 23/43] mm: shmem: enable saving to PKRAM Anthony Yznaga
` (20 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
The function inserts a page into a shmem file at a specified offset.
The page can be a regular PAGE_SIZE page or a transparent huge page.
If there is something at the offset (page or swap), the function fails.
The function will be used by the next patch.
Originally-by: Vladimir Davydov <vdavydov.dev@gmail.com>
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/shmem_fs.h | 3 ++
mm/shmem.c | 95 ++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 98 insertions(+)
diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h
index 7a35a6901221..688b92cd4ec7 100644
--- a/include/linux/shmem_fs.h
+++ b/include/linux/shmem_fs.h
@@ -96,6 +96,9 @@ enum sgp_type {
extern int shmem_getpage(struct inode *inode, pgoff_t index,
struct page **pagep, enum sgp_type sgp);
+extern int shmem_insert_page(struct mm_struct *mm, struct inode *inode,
+ pgoff_t index, struct page *page);
+
static inline struct page *shmem_read_mapping_page(
struct address_space *mapping, pgoff_t index)
{
diff --git a/mm/shmem.c b/mm/shmem.c
index bd8840082c94..0a9a2166e51f 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -677,6 +677,101 @@ static void shmem_delete_from_page_cache(struct page *page, void *radswap)
BUG_ON(error);
}
+int shmem_insert_page(struct mm_struct *mm, struct inode *inode, pgoff_t index,
+ struct page *page)
+{
+ struct address_space *mapping = inode->i_mapping;
+ struct shmem_inode_info *info = SHMEM_I(inode);
+ struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
+ gfp_t gfp = mapping_gfp_mask(mapping);
+ int err;
+ int nr = 1;
+ struct mem_cgroup *memcg;
+ pgoff_t hindex = index;
+ bool on_lru = PageLRU(page);
+
+ if (index > (MAX_LFS_FILESIZE >> PAGE_SHIFT))
+ return -EFBIG;
+
+ if (PageTransHuge(page))
+ nr = HPAGE_PMD_NR;
+ else
+ nr = 1;
+retry:
+ err = 0;
+ if (!shmem_inode_acct_block(inode, nr))
+ err = -ENOSPC;
+ if (err) {
+ int retry = 5;
+
+ /*
+ * Try to reclaim some space by splitting a huge page
+ * beyond i_size on the filesystem.
+ */
+ while (retry--) {
+ int ret;
+
+ ret = shmem_unused_huge_shrink(sbinfo, NULL, 1);
+ if (ret == SHRINK_STOP)
+ break;
+ if (ret)
+ goto retry;
+ }
+ goto failed;
+ }
+
+ if (!on_lru) {
+ __SetPageLocked(page);
+ __SetPageSwapBacked(page);
+ } else {
+ lock_page(page);
+ }
+
+ if (PageTransHuge(page))
+ hindex = round_down(index, HPAGE_PMD_NR);
+ else
+ hindex = index;
+
+ __SetPageReferenced(page);
+
+ err = mem_cgroup_try_charge_delay(page, mm, gfp, &memcg,
+ PageTransHuge(page));
+ if (err)
+ goto out_unlock;
+
+ err = shmem_add_to_page_cache(page, mapping, hindex,
+ NULL, gfp & GFP_RECLAIM_MASK);
+ if (err) {
+ mem_cgroup_cancel_charge(page, memcg,
+ PageTransHuge(page));
+ goto out_unlock;
+ }
+ mem_cgroup_commit_charge(page, memcg, on_lru,
+ PageTransHuge(page));
+
+ if (!on_lru)
+ lru_cache_add_anon(page);
+
+ spin_lock(&info->lock);
+ info->alloced += compound_nr(page);
+ inode->i_blocks += BLOCKS_PER_PAGE << compound_order(page);
+ shmem_recalc_inode(inode);
+ spin_unlock(&info->lock);
+
+ flush_dcache_page(page);
+ SetPageUptodate(page);
+ set_page_dirty(page);
+
+ unlock_page(page);
+ return 0;
+
+out_unlock:
+ unlock_page(page);
+ shmem_inode_unacct_blocks(inode, nr);
+failed:
+ return err;
+}
+
/*
* Remove swap entry from page cache, free the swap and its page cache.
*/
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 23/43] mm: shmem: enable saving to PKRAM
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (21 preceding siblings ...)
2020-05-07 0:41 ` [RFC 22/43] mm: shmem: introduce shmem_insert_page Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 24/43] mm: shmem: prevent swapping of PKRAM-enabled tmpfs pages Anthony Yznaga
` (19 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
This patch illustrates how the PKRAM API can be used for preserving tmpfs.
Two options are added to tmpfs:
The 'pkram=' option specifies the PKRAM node to load/save the
filesystem tree from/to.
The 'preserve' option initiates preservation of a read-only
filesystem tree.
If the 'pkram=' options is passed on mount, shmem will look for the
corresponding PKRAM node and load the FS tree from it.
If the 'pkram=' options was passed on mount and the 'preserve' option is
passed on remount and the filesystem is read-only, shmem will save the
FS tree to the PKRAM node.
A typical usage scenario looks like:
# mount -t tmpfs -o pkram=mytmpfs none /mnt
# echo something > /mnt/smth
# mount -o remount ro,preserve /mnt
<possibly kexec>
# mount -t tmpfs -o pkram=mytmpfs none /mnt
# cat /mnt/smth
Each FS tree is saved into a PKRAM node, and each file is saved into a
PKRAM object. A byte stream written to the object is used for saving file
metadata (name, permissions, etc) while the page stream written to
the object accommodates file content pages and their offsets.
This implementation serves as a demonstration and therefore is
simplified: it supports only regular files in the root directory without
multiple hard links, and it does not save swapped out files and aborts if
any are found. However, it can be elaborated to fully support tmpfs.
Originally-by: Vladimir Davydov <vdavydov@parallels.com>
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/pkram.h | 1 +
include/linux/shmem_fs.h | 24 +++
mm/Makefile | 2 +-
mm/shmem.c | 66 ++++++++
mm/shmem_pkram.c | 381 +++++++++++++++++++++++++++++++++++++++++++++++
5 files changed, 473 insertions(+), 1 deletion(-)
create mode 100644 mm/shmem_pkram.c
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index 1cd518843d7a..b47b3aef16e3 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -17,6 +17,7 @@ struct pkram_stream {
unsigned int entry_idx; /* next entry in link */
unsigned long next_index;
+ struct address_space *mapping;
/* byte data */
struct page *data_page;
diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h
index 688b92cd4ec7..f2ce9937a8f2 100644
--- a/include/linux/shmem_fs.h
+++ b/include/linux/shmem_fs.h
@@ -26,6 +26,11 @@ struct shmem_inode_info {
struct inode vfs_inode;
};
+#define SHMEM_PKRAM_NAME_MAX 128
+struct shmem_pkram_info {
+ char name[SHMEM_PKRAM_NAME_MAX];
+};
+
struct shmem_sb_info {
unsigned long max_blocks; /* How many blocks are allowed */
struct percpu_counter used_blocks; /* How many are allocated */
@@ -40,6 +45,8 @@ struct shmem_sb_info {
spinlock_t shrinklist_lock; /* Protects shrinklist */
struct list_head shrinklist; /* List of shinkable inodes */
unsigned long shrinklist_len; /* Length of shrinklist */
+ struct shmem_pkram_info *pkram;
+ bool preserve; /* PKRAM-enabled data is preserved */
};
static inline struct shmem_inode_info *SHMEM_I(struct inode *inode)
@@ -99,6 +106,23 @@ extern int shmem_getpage(struct inode *inode, pgoff_t index,
extern int shmem_insert_page(struct mm_struct *mm, struct inode *inode,
pgoff_t index, struct page *page);
+#ifdef CONFIG_PKRAM
+extern int shmem_parse_pkram(const char *str, struct shmem_pkram_info **pkram);
+extern void shmem_show_pkram(struct seq_file *seq, struct shmem_pkram_info *pkram,
+ bool preserve);
+extern int shmem_save_pkram(struct super_block *sb);
+extern void shmem_load_pkram(struct super_block *sb);
+extern int shmem_release_pkram(struct super_block *sb);
+#else
+static inline int shmem_parse_pkram(const char *str,
+ struct shmem_pkram_info **pkram) { return 1; }
+static inline void shmem_show_pkram(struct seq_file *seq,
+ struct shmem_pkram_info *pkram, bool preserve) { }
+static inline int shmem_save_pkram(struct super_block *sb) { return 0; }
+static inline void shmem_load_pkram(struct super_block *sb) { }
+static inline int shmem_release_pkram(struct super_block *sb) { return 0; }
+#endif
+
static inline struct page *shmem_read_mapping_page(
struct address_space *mapping, pgoff_t index)
{
diff --git a/mm/Makefile b/mm/Makefile
index c4ad1c56e237..5c07ecaa5a38 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -112,4 +112,4 @@ obj-$(CONFIG_MEMFD_CREATE) += memfd.o
obj-$(CONFIG_MAPPING_DIRTY_HELPERS) += mapping_dirty_helpers.o
obj-$(CONFIG_PTDUMP_CORE) += ptdump.o
obj-$(CONFIG_PAGE_REPORTING) += page_reporting.o
-obj-$(CONFIG_PKRAM) += pkram.o pkram_pagetable.o
+obj-$(CONFIG_PKRAM) += pkram.o pkram_pagetable.o shmem_pkram.o
diff --git a/mm/shmem.c b/mm/shmem.c
index 0a9a2166e51f..9c28ef657cd1 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -112,14 +112,18 @@ struct shmem_options {
unsigned long long blocks;
unsigned long long inodes;
struct mempolicy *mpol;
+ struct shmem_pkram_info *pkram;
kuid_t uid;
kgid_t gid;
umode_t mode;
int huge;
+ bool preserve;
int seen;
#define SHMEM_SEEN_BLOCKS 1
#define SHMEM_SEEN_INODES 2
#define SHMEM_SEEN_HUGE 4
+#define SHMEM_SEEN_PKRAM 8
+#define SHMEM_SEEN_PRESERVE 16
};
#ifdef CONFIG_TMPFS
@@ -3467,6 +3471,8 @@ enum shmem_param {
Opt_mpol,
Opt_nr_blocks,
Opt_nr_inodes,
+ Opt_pkram,
+ Opt_preserve,
Opt_size,
Opt_uid,
};
@@ -3486,6 +3492,8 @@ const struct fs_parameter_spec shmem_fs_parameters[] = {
fsparam_string("mpol", Opt_mpol),
fsparam_string("nr_blocks", Opt_nr_blocks),
fsparam_string("nr_inodes", Opt_nr_inodes),
+ fsparam_string("pkram", Opt_pkram),
+ fsparam_flag_no("preserve", Opt_preserve),
fsparam_string("size", Opt_size),
fsparam_u32 ("uid", Opt_uid),
{}
@@ -3557,6 +3565,22 @@ static int shmem_parse_one(struct fs_context *fc, struct fs_parameter *param)
if (mpol_parse_str(param->string, &ctx->mpol))
goto bad_value;
break;
+ } else
+ goto unsupported_parameter;
+ case Opt_pkram:
+ if (IS_ENABLED(CONFIG_PKRAM)) {
+ kfree(ctx->pkram);
+ if (shmem_parse_pkram(param->string, &ctx->pkram))
+ goto bad_value;
+ ctx->seen |= SHMEM_SEEN_PKRAM;
+ break;
+ }
+ goto unsupported_parameter;
+ case Opt_preserve:
+ if (IS_ENABLED(CONFIG_PKRAM)) {
+ ctx->preserve = result.boolean;
+ ctx->seen |= SHMEM_SEEN_PRESERVE;
+ break;
}
goto unsupported_parameter;
}
@@ -3650,6 +3674,42 @@ static int shmem_reconfigure(struct fs_context *fc)
}
}
+ if (ctx->seen & SHMEM_SEEN_PRESERVE) {
+ if (!sbinfo->pkram && !(ctx->seen & SHMEM_SEEN_PKRAM)) {
+ err = "Cannot set preserve/nopreserve. Not enabled for PKRAM";
+ goto out;
+ }
+ if (ctx->preserve && !(fc->sb_flags & SB_RDONLY)) {
+ err = "Cannot preserve. Filesystem must be read-only";
+ goto out;
+ }
+ }
+
+ if (ctx->pkram) {
+ kfree(sbinfo->pkram);
+ sbinfo->pkram = ctx->pkram;
+ }
+
+ if (ctx->seen & SHMEM_SEEN_PRESERVE) {
+ int error;
+
+ if (!sbinfo->preserve && ctx->preserve) {
+ error = shmem_save_pkram(fc->root->d_sb);
+ if (error) {
+ err = "Failed to preserve";
+ goto out;
+ }
+ sbinfo->preserve = true;
+ } else if (sbinfo->preserve && !ctx->preserve) {
+ error = shmem_release_pkram(fc->root->d_sb);
+ if (error) {
+ err = "Failed to unpreserve";
+ goto out;
+ }
+ sbinfo->preserve = false;
+ }
+ }
+
if (ctx->seen & SHMEM_SEEN_HUGE)
sbinfo->huge = ctx->huge;
if (ctx->seen & SHMEM_SEEN_BLOCKS)
@@ -3667,6 +3727,7 @@ static int shmem_reconfigure(struct fs_context *fc)
sbinfo->mpol = ctx->mpol; /* transfers initial ref */
ctx->mpol = NULL;
}
+
spin_unlock(&sbinfo->stat_lock);
return 0;
out:
@@ -3697,6 +3758,7 @@ static int shmem_show_options(struct seq_file *seq, struct dentry *root)
seq_printf(seq, ",huge=%s", shmem_format_huge(sbinfo->huge));
#endif
shmem_show_mpol(seq, sbinfo->mpol);
+ shmem_show_pkram(seq, sbinfo->pkram, sbinfo->preserve);
return 0;
}
@@ -3708,6 +3770,7 @@ static void shmem_put_super(struct super_block *sb)
percpu_counter_destroy(&sbinfo->used_blocks);
mpol_put(sbinfo->mpol);
+ kfree(sbinfo->pkram);
kfree(sbinfo);
sb->s_fs_info = NULL;
}
@@ -3754,6 +3817,8 @@ static int shmem_fill_super(struct super_block *sb, struct fs_context *fc)
sbinfo->huge = ctx->huge;
sbinfo->mpol = ctx->mpol;
ctx->mpol = NULL;
+ sbinfo->pkram = ctx->pkram;
+ ctx->pkram = NULL;
spin_lock_init(&sbinfo->stat_lock);
if (percpu_counter_init(&sbinfo->used_blocks, 0, GFP_KERNEL))
@@ -3783,6 +3848,7 @@ static int shmem_fill_super(struct super_block *sb, struct fs_context *fc)
sb->s_root = d_make_root(inode);
if (!sb->s_root)
goto failed;
+ shmem_load_pkram(sb);
return 0;
failed:
diff --git a/mm/shmem_pkram.c b/mm/shmem_pkram.c
new file mode 100644
index 000000000000..3fa9cfbe0003
--- /dev/null
+++ b/mm/shmem_pkram.c
@@ -0,0 +1,381 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/crash_dump.h>
+#include <linux/dcache.h>
+#include <linux/err.h>
+#include <linux/fs.h>
+#include <linux/gfp.h>
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/mm.h>
+#include <linux/mount.h>
+#include <linux/mutex.h>
+#include <linux/namei.h>
+#include <linux/pagemap.h>
+#include <linux/pagevec.h>
+#include <linux/pkram.h>
+#include <linux/seq_file.h>
+#include <linux/shmem_fs.h>
+#include <linux/spinlock.h>
+#include <linux/string.h>
+#include <linux/time.h>
+#include <linux/types.h>
+#include <linux/uaccess.h>
+
+struct file_header {
+ __u32 mode;
+ kuid_t uid;
+ kgid_t gid;
+ __u32 namelen;
+ __u64 size;
+ __u64 atime;
+ __u64 mtime;
+ __u64 ctime;
+};
+
+int shmem_parse_pkram(const char *str, struct shmem_pkram_info **pkram)
+{
+ struct shmem_pkram_info *new;
+ size_t len;
+
+ len = strlen(str);
+ if (!len || len >= SHMEM_PKRAM_NAME_MAX)
+ return 1;
+ new = kzalloc(sizeof(*new), GFP_KERNEL);
+ if (!new)
+ return 1;
+ strcpy(new->name, str);
+ *pkram = new;
+ return 0;
+}
+
+void shmem_show_pkram(struct seq_file *seq, struct shmem_pkram_info *pkram, bool preserve)
+{
+ if (pkram) {
+ seq_printf(seq, ",pkram=%s", pkram->name);
+ seq_printf(seq, ",%s", preserve ? "preserve" : "nopreserve");
+ }
+}
+
+static int shmem_pkram_name(char *buf, size_t bufsize,
+ struct shmem_sb_info *sbinfo)
+{
+ if (snprintf(buf, bufsize, "shmem-%s", sbinfo->pkram->name) >= bufsize)
+ return -ENAMETOOLONG;
+ return 0;
+}
+
+static int save_page(struct page *page, struct pkram_stream *ps)
+{
+ int err = 0;
+
+ if (page)
+ err = pkram_save_page(ps, page, 0);
+
+ return err;
+}
+
+static int save_file_content(struct pkram_stream *ps)
+{
+ struct pagevec pvec;
+ pgoff_t indices[PAGEVEC_SIZE];
+ pgoff_t index = 0;
+ struct page *page;
+ int i, err = 0;
+
+ pagevec_init(&pvec);
+ for ( ; ; ) {
+ pvec.nr = find_get_entries(ps->mapping, index, PAGEVEC_SIZE,
+ pvec.pages, indices);
+ if (!pvec.nr)
+ break;
+ for (i = 0; i < pagevec_count(&pvec); i++) {
+ page = pvec.pages[i];
+ index = indices[i];
+
+ if (WARN_ON_ONCE(xa_is_value(page))) {
+ err = -EINVAL;
+ break;
+ }
+
+ lock_page(page);
+
+ if (PageTransTail(page)) {
+ WARN_ONCE(1, "PageTransTail returned true\n");
+ unlock_page(page);
+ continue;
+ }
+
+ BUG_ON(page->mapping != ps->mapping);
+ err = save_page(page, ps);
+
+ i += compound_nr(page) - 1;
+ index += compound_nr(page) - 1;
+
+ unlock_page(page);
+ if (err)
+ break;
+ }
+ pagevec_remove_exceptionals(&pvec);
+ pagevec_release(&pvec);
+ if (err)
+ break;
+ cond_resched();
+ index++;
+ }
+
+ return err;
+}
+
+static int save_file(struct dentry *dentry, struct pkram_stream *ps)
+{
+ struct inode *inode = dentry->d_inode;
+ umode_t mode = inode->i_mode;
+ struct file_header hdr;
+ ssize_t ret;
+
+ if (WARN_ON_ONCE(!S_ISREG(mode)))
+ return -EINVAL;
+ if (WARN_ON_ONCE(inode->i_nlink > 1))
+ return -EINVAL;
+
+ hdr.mode = mode;
+ hdr.uid = inode->i_uid;
+ hdr.gid = inode->i_gid;
+ hdr.namelen = dentry->d_name.len;
+ hdr.size = i_size_read(inode);
+ hdr.atime = timespec64_to_ns(&inode->i_atime);
+ hdr.mtime = timespec64_to_ns(&inode->i_mtime);
+ hdr.ctime = timespec64_to_ns(&inode->i_ctime);
+
+ ret = pkram_write(ps, &hdr, sizeof(hdr));
+ if (ret < 0)
+ return ret;
+ ret = pkram_write(ps, dentry->d_name.name, dentry->d_name.len);
+ if (ret < 0)
+ return ret;
+
+ ps->mapping = inode->i_mapping;
+ return save_file_content(ps);
+}
+
+static int save_tree(struct super_block *sb, struct pkram_stream *ps)
+{
+ struct dentry *dentry, *root = sb->s_root;
+ int err = 0;
+
+ inode_lock(d_inode(root));
+ spin_lock(&root->d_lock);
+ list_for_each_entry(dentry, &root->d_subdirs, d_child) {
+ if (d_unhashed(dentry) || !dentry->d_inode)
+ continue;
+ dget(dentry);
+ spin_unlock(&root->d_lock);
+
+ err = pkram_prepare_save_obj(ps);
+ if (!err)
+ err = save_file(dentry, ps);
+ if (!err)
+ pkram_finish_save_obj(ps);
+ spin_lock(&root->d_lock);
+ dput(dentry);
+ if (err)
+ break;
+ }
+ spin_unlock(&root->d_lock);
+ inode_unlock(d_inode(root));
+
+ return err;
+}
+
+int shmem_save_pkram(struct super_block *sb)
+{
+ struct shmem_sb_info *sbinfo = sb->s_fs_info;
+ struct pkram_stream psobj;
+ char *buf;
+ int err = -ENOMEM;
+
+ if (!sbinfo || !sbinfo->pkram || is_kdump_kernel())
+ return 0;
+
+ buf = (void *)__get_free_page(GFP_KERNEL);
+ if (!buf)
+ goto out;
+
+ err = shmem_pkram_name(buf, PAGE_SIZE, sbinfo);
+ if (!err)
+ err = pkram_prepare_save(&psobj, buf, GFP_KERNEL);
+ if (err)
+ goto out_free_buf;
+
+ err = save_tree(sb, &psobj);
+ if (err)
+ goto out_discard_save;
+
+ pkram_finish_save(&psobj);
+ goto out_free_buf;
+
+out_discard_save:
+ pkram_discard_save(&psobj);
+out_free_buf:
+ free_page((unsigned long)buf);
+out:
+ if (err)
+ pr_err("SHMEM: PKRAM save failed: %d\n", err);
+
+ return err;
+}
+
+static int load_file_content(struct pkram_stream *ps)
+{
+ unsigned long index;
+ struct page *page;
+ int err = 0;
+
+ do {
+ page = pkram_load_page(ps, &index, NULL);
+ if (!page)
+ break;
+
+ err = shmem_insert_page(current->mm, ps->mapping->host, index, page);
+ put_page(page);
+ } while (!err);
+
+ return err;
+}
+
+static int load_file(struct dentry *parent, struct pkram_stream *ps,
+ char *buf, size_t bufsize)
+{
+ struct dentry *dentry;
+ struct inode *inode;
+ struct file_header hdr;
+ size_t ret;
+ umode_t mode;
+ int namelen;
+ int err = 0;
+
+ ret = pkram_read(ps, &hdr, sizeof(hdr));
+ if (ret != sizeof(hdr))
+ return -EINVAL;
+
+ mode = hdr.mode;
+ namelen = hdr.namelen;
+ if (!S_ISREG(mode) || namelen > bufsize)
+ return -EINVAL;
+ if (pkram_read(ps, buf, namelen) != namelen)
+ return -EINVAL;
+
+ inode_lock_nested(d_inode(parent), I_MUTEX_PARENT);
+
+ dentry = lookup_one_len(buf, parent, namelen);
+ if (IS_ERR(dentry)) {
+ err = PTR_ERR(dentry);
+ goto out_unlock;
+ }
+
+ err = vfs_create(parent->d_inode, dentry, mode, NULL);
+ dput(dentry); /* on success shmem pinned it */
+ if (err)
+ goto out_unlock;
+
+ inode = dentry->d_inode;
+ inode->i_mode = mode;
+ inode->i_uid = hdr.uid;
+ inode->i_gid = hdr.gid;
+ inode->i_atime = ns_to_timespec64(hdr.atime);
+ inode->i_mtime = ns_to_timespec64(hdr.mtime);
+ inode->i_ctime = ns_to_timespec64(hdr.ctime);
+ i_size_write(inode, hdr.size);
+
+ ps->mapping = inode->i_mapping;
+ err = load_file_content(ps);
+out_unlock:
+ inode_unlock(d_inode(parent));
+
+ return err;
+}
+
+static int load_tree(struct super_block *sb, struct pkram_stream *ps,
+ char *buf, size_t bufsize)
+{
+ int err;
+
+ do {
+ err = pkram_prepare_load_obj(ps);
+ if (err) {
+ if (err == -ENODATA)
+ err = 0;
+ break;
+ }
+ err = load_file(sb->s_root, ps, buf, PAGE_SIZE);
+ pkram_finish_load_obj(ps);
+ } while (!err);
+
+ return err;
+}
+
+void shmem_load_pkram(struct super_block *sb)
+{
+ struct shmem_sb_info *sbinfo = sb->s_fs_info;
+ struct pkram_stream psobj;
+ char *buf;
+ int err = -ENOMEM;
+
+ if (!sbinfo->pkram)
+ return;
+
+ buf = (void *)__get_free_page(GFP_KERNEL);
+ if (!buf)
+ goto out;
+
+ err = shmem_pkram_name(buf, PAGE_SIZE, sbinfo);
+ if (!err)
+ err = pkram_prepare_load(&psobj, buf);
+ if (err) {
+ if (err == -ENOENT)
+ err = 0;
+ goto out_free_buf;
+ }
+
+ err = load_tree(sb, &psobj, buf, PAGE_SIZE);
+
+ pkram_finish_load(&psobj);
+out_free_buf:
+ free_page((unsigned long)buf);
+out:
+ if (err)
+ pr_err("SHMEM: PKRAM load failed: %d\n", err);
+}
+
+int shmem_release_pkram(struct super_block *sb)
+{
+ struct shmem_sb_info *sbinfo = sb->s_fs_info;
+ struct pkram_stream psobj;
+ char *buf;
+ int err = -ENOMEM;
+
+ if (!sbinfo->pkram)
+ return 0;
+
+ buf = (void *)__get_free_page(GFP_KERNEL);
+ if (!buf)
+ goto out;
+
+ err = shmem_pkram_name(buf, PAGE_SIZE, sbinfo);
+ if (!err)
+ err = pkram_prepare_load(&psobj, buf);
+ if (err) {
+ if (err == -ENOENT)
+ err = 0;
+ goto out_free_buf;
+ }
+
+ pkram_finish_load(&psobj);
+out_free_buf:
+ free_page((unsigned long)buf);
+out:
+ if (err)
+ pr_err("SHMEM: PKRAM load failed: %d\n", err);
+
+ return err;
+}
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 24/43] mm: shmem: prevent swapping of PKRAM-enabled tmpfs pages
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (22 preceding siblings ...)
2020-05-07 0:41 ` [RFC 23/43] mm: shmem: enable saving to PKRAM Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 25/43] mm: shmem: specify the mm to use when inserting pages Anthony Yznaga
` (18 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Workaround the limitation that shmem pages must be in memory in order
to be preserved by preventing them from being swapped out in the first
place. Do this by marking shmem pages associated with a PKRAM node
as unevictable.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
mm/shmem.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/mm/shmem.c b/mm/shmem.c
index 9c28ef657cd1..13475073fb52 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -2360,6 +2360,8 @@ static struct inode *shmem_get_inode(struct super_block *sb, const struct inode
INIT_LIST_HEAD(&info->swaplist);
simple_xattrs_init(&info->xattrs);
cache_no_acl(inode);
+ if (sbinfo->pkram)
+ mapping_set_unevictable(inode->i_mapping);
switch (mode & S_IFMT) {
default:
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 25/43] mm: shmem: specify the mm to use when inserting pages
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (23 preceding siblings ...)
2020-05-07 0:41 ` [RFC 24/43] mm: shmem: prevent swapping of PKRAM-enabled tmpfs pages Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 26/43] mm: shmem: when inserting, handle pages already charged to a memcg Anthony Yznaga
` (17 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Explicitly specify the mm to pass to shmem_insert_page() when
the pkram_stream is initialized rather than use the mm of the
current thread. This will allow for multiple kernel threads to
target the same mm when inserting pages in parallel.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/pkram.h | 1 +
mm/pkram.c | 1 +
mm/shmem_pkram.c | 2 +-
3 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index b47b3aef16e3..cbb79d2803c0 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -18,6 +18,7 @@ struct pkram_stream {
unsigned long next_index;
struct address_space *mapping;
+ struct mm_struct *mm;
/* byte data */
struct page *data_page;
diff --git a/mm/pkram.c b/mm/pkram.c
index 4d4d836fea53..a5e539052af6 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -565,6 +565,7 @@ static void pkram_stream_init(struct pkram_stream *ps,
memset(ps, 0, sizeof(*ps));
ps->gfp_mask = gfp_mask;
ps->node = node;
+ ps->mm = current->mm;
}
static void pkram_stream_init_obj(struct pkram_stream *ps, struct pkram_obj *obj)
diff --git a/mm/shmem_pkram.c b/mm/shmem_pkram.c
index 3fa9cfbe0003..c97d64393822 100644
--- a/mm/shmem_pkram.c
+++ b/mm/shmem_pkram.c
@@ -236,7 +236,7 @@ static int load_file_content(struct pkram_stream *ps)
if (!page)
break;
- err = shmem_insert_page(current->mm, ps->mapping->host, index, page);
+ err = shmem_insert_page(ps->mm, ps->mapping->host, index, page);
put_page(page);
} while (!err);
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 26/43] mm: shmem: when inserting, handle pages already charged to a memcg
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (24 preceding siblings ...)
2020-05-07 0:41 ` [RFC 25/43] mm: shmem: specify the mm to use when inserting pages Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 27/43] x86/mm/numa: add numa_isolate_memblocks() Anthony Yznaga
` (16 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
If shmem_insert_page() is called to insert a page that was preserved
using PKRAM on the current boot (i.e. preserved page is restored without
an intervening kexec boot), the page will still be charged to a memory
cgroup because it is never freed. Don't try to charge it again.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
mm/shmem.c | 21 +++++++++++++--------
1 file changed, 13 insertions(+), 8 deletions(-)
diff --git a/mm/shmem.c b/mm/shmem.c
index 13475073fb52..1f3b43b8fa34 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -693,6 +693,7 @@ int shmem_insert_page(struct mm_struct *mm, struct inode *inode, pgoff_t index,
struct mem_cgroup *memcg;
pgoff_t hindex = index;
bool on_lru = PageLRU(page);
+ bool has_memcg = page->mem_cgroup ? true : false;
if (index > (MAX_LFS_FILESIZE >> PAGE_SHIFT))
return -EFBIG;
@@ -738,20 +739,24 @@ int shmem_insert_page(struct mm_struct *mm, struct inode *inode, pgoff_t index,
__SetPageReferenced(page);
- err = mem_cgroup_try_charge_delay(page, mm, gfp, &memcg,
- PageTransHuge(page));
- if (err)
- goto out_unlock;
+ if (!has_memcg) {
+ err = mem_cgroup_try_charge_delay(page, mm, gfp, &memcg,
+ PageTransHuge(page));
+ if (err)
+ goto out_unlock;
+ }
err = shmem_add_to_page_cache(page, mapping, hindex,
NULL, gfp & GFP_RECLAIM_MASK);
if (err) {
- mem_cgroup_cancel_charge(page, memcg,
- PageTransHuge(page));
+ if (!has_memcg)
+ mem_cgroup_cancel_charge(page, memcg,
+ PageTransHuge(page));
goto out_unlock;
}
- mem_cgroup_commit_charge(page, memcg, on_lru,
- PageTransHuge(page));
+ if (!has_memcg)
+ mem_cgroup_commit_charge(page, memcg, on_lru,
+ PageTransHuge(page));
if (!on_lru)
lru_cache_add_anon(page);
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 27/43] x86/mm/numa: add numa_isolate_memblocks()
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (25 preceding siblings ...)
2020-05-07 0:41 ` [RFC 26/43] mm: shmem: when inserting, handle pages already charged to a memcg Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 28/43] PKRAM: ensure memblocks with preserved pages init'd for numa Anthony Yznaga
` (15 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Provide a way for a caller external to numa to ensure memblocks in the
memblock reserved list do not cross node boundaries and have a node id
assigned to them. This will be used by PKRAM to ensure initialization of
page structs for preserved pages can be deferred and multithreaded
efficiently.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
arch/x86/include/asm/numa.h | 4 ++++
arch/x86/mm/numa.c | 32 ++++++++++++++++++++------------
2 files changed, 24 insertions(+), 12 deletions(-)
diff --git a/arch/x86/include/asm/numa.h b/arch/x86/include/asm/numa.h
index bbfde3d2662f..f9e05f4eb1c6 100644
--- a/arch/x86/include/asm/numa.h
+++ b/arch/x86/include/asm/numa.h
@@ -40,6 +40,7 @@ static inline void set_apicid_to_node(int apicid, s16 node)
}
extern int numa_cpu_node(int cpu);
+extern void __init numa_isolate_memblocks(void);
#else /* CONFIG_NUMA */
static inline void set_apicid_to_node(int apicid, s16 node)
@@ -50,6 +51,9 @@ static inline int numa_cpu_node(int cpu)
{
return NUMA_NO_NODE;
}
+static inline void numa_isolate_memblocks(void)
+{
+}
#endif /* CONFIG_NUMA */
#ifdef CONFIG_X86_32
diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c
index 59ba008504dc..df0065e24ea5 100644
--- a/arch/x86/mm/numa.c
+++ b/arch/x86/mm/numa.c
@@ -475,6 +475,25 @@ static bool __init numa_meminfo_cover_memory(const struct numa_meminfo *mi)
return true;
}
+void __init numa_isolate_memblocks(void)
+{
+ int i;
+
+ /*
+ * Iterate over all memory known to the x86 architecture,
+ * and use those ranges to set the nid in memblock.reserved.
+ * This will split up the memblock regions along node
+ * boundaries and will set the node IDs as well.
+ */
+ for (i = 0; i < numa_meminfo.nr_blks; i++) {
+ struct numa_memblk *mb = numa_meminfo.blk + i;
+ int ret;
+
+ ret = memblock_set_node(mb->start, mb->end - mb->start, &memblock.reserved, mb->nid);
+ WARN_ON_ONCE(ret);
+ }
+}
+
/*
* Mark all currently memblock-reserved physical memory (which covers the
* kernel's own memory ranges) as hot-unswappable.
@@ -493,19 +512,8 @@ static void __init numa_clear_kernel_node_hotplug(void)
* used by the kernel, but those regions are not split up
* along node boundaries yet, and don't necessarily have their
* node ID set yet either.
- *
- * So iterate over all memory known to the x86 architecture,
- * and use those ranges to set the nid in memblock.reserved.
- * This will split up the memblock regions along node
- * boundaries and will set the node IDs as well.
*/
- for (i = 0; i < numa_meminfo.nr_blks; i++) {
- struct numa_memblk *mb = numa_meminfo.blk + i;
- int ret;
-
- ret = memblock_set_node(mb->start, mb->end - mb->start, &memblock.reserved, mb->nid);
- WARN_ON_ONCE(ret);
- }
+ numa_isolate_memblocks();
/*
* Now go over all reserved memblock regions, to construct a
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 28/43] PKRAM: ensure memblocks with preserved pages init'd for numa
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (26 preceding siblings ...)
2020-05-07 0:41 ` [RFC 27/43] x86/mm/numa: add numa_isolate_memblocks() Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 29/43] memblock: PKRAM: mark memblocks that contain preserved pages Anthony Yznaga
` (14 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
In order to facilitate fast initialization of page structs for
preserved pages, memblocks with preserved pages must not cross
numa node boundaries and must have a node id assigned to them.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
mm/pkram.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/mm/pkram.c b/mm/pkram.c
index a5e539052af6..97a7dd0a5b7d 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -21,6 +21,7 @@
#include <linux/sysfs.h>
#include <linux/types.h>
+#include <asm/numa.h>
#include "internal.h"
@@ -242,6 +243,15 @@ void __init pkram_reserve(void)
return;
}
+ /*
+ * Fix up the reserved memblock list to ensure the
+ * memblock regions are split along node boundaries
+ * and have a node ID set. This will allow the page
+ * structs for the preserved pages to be initialized
+ * more efficiently.
+ */
+ numa_isolate_memblocks();
+
done:
pr_info("PKRAM: %lu pages reserved\n", pkram_reserved_pages);
}
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 29/43] memblock: PKRAM: mark memblocks that contain preserved pages
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (27 preceding siblings ...)
2020-05-07 0:41 ` [RFC 28/43] PKRAM: ensure memblocks with preserved pages init'd for numa Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 30/43] memblock: add for_each_reserved_mem_range() Anthony Yznaga
` (13 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
To support deferred initialization of page structs for preserved pages,
separate memblocks containing preserved pages by setting a new flag
when adding them to the memblock reserved list.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/memblock.h | 7 +++++++
mm/memblock.c | 8 +++++++-
mm/pkram.c | 4 ++--
3 files changed, 16 insertions(+), 3 deletions(-)
diff --git a/include/linux/memblock.h b/include/linux/memblock.h
index 6bc37a731d27..27ab2b30ae1d 100644
--- a/include/linux/memblock.h
+++ b/include/linux/memblock.h
@@ -37,6 +37,7 @@ enum memblock_flags {
MEMBLOCK_HOTPLUG = 0x1, /* hotpluggable region */
MEMBLOCK_MIRROR = 0x2, /* mirrored region */
MEMBLOCK_NOMAP = 0x4, /* don't add to kernel direct mapping */
+ MEMBLOCK_PRESERVED = 0x8, /* preserved pages region */
};
/**
@@ -111,6 +112,7 @@ void memblock_allow_resize(void);
int memblock_add_node(phys_addr_t base, phys_addr_t size, int nid);
int memblock_add(phys_addr_t base, phys_addr_t size);
int memblock_remove(phys_addr_t base, phys_addr_t size);
+int __memblock_reserve(phys_addr_t base, phys_addr_t size, enum memblock_flags flags);
int memblock_free(phys_addr_t base, phys_addr_t size);
int memblock_reserve(phys_addr_t base, phys_addr_t size);
#ifdef CONFIG_HAVE_MEMBLOCK_PHYS_MAP
@@ -215,6 +217,11 @@ static inline bool memblock_is_nomap(struct memblock_region *m)
return m->flags & MEMBLOCK_NOMAP;
}
+static inline bool memblock_is_preserved(struct memblock_region *m)
+{
+ return m->flags & MEMBLOCK_PRESERVED;
+}
+
#ifdef CONFIG_HAVE_MEMBLOCK_NODE_MAP
int memblock_search_pfn_nid(unsigned long pfn, unsigned long *start_pfn,
unsigned long *end_pfn);
diff --git a/mm/memblock.c b/mm/memblock.c
index 69ae883b8d21..1a9a2055ed11 100644
--- a/mm/memblock.c
+++ b/mm/memblock.c
@@ -831,6 +831,12 @@ int __init_memblock memblock_free(phys_addr_t base, phys_addr_t size)
return memblock_remove_range(&memblock.reserved, base, size);
}
+int __init_memblock __memblock_reserve(phys_addr_t base, phys_addr_t size,
+ enum memblock_flags flags)
+{
+ return memblock_add_range(&memblock.reserved, base, size, MAX_NUMNODES, flags);
+}
+
int __init_memblock memblock_reserve(phys_addr_t base, phys_addr_t size)
{
phys_addr_t end = base + size - 1;
@@ -838,7 +844,7 @@ int __init_memblock memblock_reserve(phys_addr_t base, phys_addr_t size)
memblock_dbg("%s: [%pa-%pa] %pS\n", __func__,
&base, &end, (void *)_RET_IP_);
- return memblock_add_range(&memblock.reserved, base, size, MAX_NUMNODES, 0);
+ return __memblock_reserve(base, size, 0);
}
#ifdef CONFIG_HAVE_MEMBLOCK_PHYS_MAP
diff --git a/mm/pkram.c b/mm/pkram.c
index 97a7dd0a5b7d..b83d31740619 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -170,7 +170,7 @@ static int __init pkram_reserve_page(unsigned long pfn)
size = PAGE_SIZE;
if (memblock_is_region_reserved(base, size) ||
- memblock_reserve(base, size) < 0)
+ __memblock_reserve(base, size, MEMBLOCK_PRESERVED) < 0)
err = -EBUSY;
if (!err)
@@ -1446,7 +1446,7 @@ static void pkram_remove_identity_map(struct page *page)
static int __init pkram_reserve_range_cb(struct pkram_pg_state *st, unsigned long base, unsigned long size)
{
if (memblock_is_region_reserved(base, size) ||
- memblock_reserve(base, size) < 0) {
+ __memblock_reserve(base, size, MEMBLOCK_PRESERVED) < 0) {
pr_warn("PKRAM: reservations exist in [0x%lx,0x%lx]\n", base, base + size - 1);
/*
* Set a lower bound so another walk can undo the earlier,
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 30/43] memblock: add for_each_reserved_mem_range()
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (28 preceding siblings ...)
2020-05-07 0:41 ` [RFC 29/43] memblock: PKRAM: mark memblocks that contain preserved pages Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 31/43] memblock, mm: defer initialization of preserved pages Anthony Yznaga
` (12 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
To support deferred initialization of page structs for preserved
pages, add an iterator of the memblock reserved list that can select or
exclude ranges based on memblock flags.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/memblock.h | 10 ++++++++++
mm/memblock.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 60 insertions(+), 1 deletion(-)
diff --git a/include/linux/memblock.h b/include/linux/memblock.h
index 27ab2b30ae1d..f348ebb750c9 100644
--- a/include/linux/memblock.h
+++ b/include/linux/memblock.h
@@ -145,6 +145,11 @@ void __next_mem_range_rev(u64 *idx, int nid, enum memblock_flags flags,
void __next_reserved_mem_region(u64 *idx, phys_addr_t *out_start,
phys_addr_t *out_end);
+void __next_reserved_mem_range(u64 *idx, enum memblock_flags flags,
+ enum memblock_flags exclflags,
+ phys_addr_t *out_start, phys_addr_t *out_end,
+ int *out_nid);
+
void __memblock_free_late(phys_addr_t base, phys_addr_t size);
/**
@@ -202,6 +207,11 @@ void __memblock_free_late(phys_addr_t base, phys_addr_t size);
i != (u64)ULLONG_MAX; \
__next_reserved_mem_region(&i, p_start, p_end))
+#define for_each_reserved_mem_range(i, flags, exclflags, p_start, p_end, p_nid)\
+ for (i = 0UL, __next_reserved_mem_range(&i, flags, exclflags, p_start, p_end, p_nid); \
+ i != (u64)ULLONG_MAX; \
+ __next_reserved_mem_range(&i, flags, exclflags, p_start, p_end, p_nid))
+
static inline bool memblock_is_hotpluggable(struct memblock_region *m)
{
return m->flags & MEMBLOCK_HOTPLUG;
diff --git a/mm/memblock.c b/mm/memblock.c
index 1a9a2055ed11..33597f352dc0 100644
--- a/mm/memblock.c
+++ b/mm/memblock.c
@@ -987,6 +987,55 @@ void __init_memblock __next_reserved_mem_region(u64 *idx,
*idx = ULLONG_MAX;
}
+/**
+ * __next_reserved_mem_range - next function for for_each_reserved_range()
+ * @idx: pointer to u64 loop variable
+ * @flags: pick blocks based on memory attributes
+ * @exclflags: exclude blocks based on memory attributes
+ * @out_start: ptr to phys_addr_t for start address of the range, can be %NULL
+ * @out_end: ptr to phys_addr_t for end address of the range, can be %NULL
+ * @out_nid: ptr to int for nid of the range, can be %NULL
+ *
+ * Iterate over all reserved memory ranges.
+ */
+void __init_memblock __next_reserved_mem_range(u64 *idx,
+ enum memblock_flags flags,
+ enum memblock_flags exclflags,
+ phys_addr_t *out_start,
+ phys_addr_t *out_end, int *out_nid)
+{
+ struct memblock_type *type = &memblock.reserved;
+ int _idx = *idx;
+
+ for (; _idx < type->cnt; _idx++) {
+ struct memblock_region *r = &type->regions[_idx];
+ phys_addr_t base = r->base;
+ phys_addr_t size = r->size;
+
+ /* skip preserved pages */
+ if ((exclflags & MEMBLOCK_PRESERVED) && memblock_is_preserved(r))
+ continue;
+
+ /* skip non-preserved pages */
+ if ((flags & MEMBLOCK_PRESERVED) && !memblock_is_preserved(r))
+ continue;
+
+ if (out_start)
+ *out_start = base;
+ if (out_end)
+ *out_end = base + size - 1;
+ if (out_nid)
+ *out_nid = r->nid;
+
+ _idx++;
+ *idx = (u64)_idx;
+ return;
+ }
+
+ /* signal end of iteration */
+ *idx = ULLONG_MAX;
+}
+
static bool should_skip_region(struct memblock_region *m, int nid, int flags)
{
int m_nid = memblock_get_region_node(m);
@@ -1011,7 +1060,7 @@ static bool should_skip_region(struct memblock_region *m, int nid, int flags)
}
/**
- * __next_mem_range - next function for for_each_free_mem_range() etc.
+ * __next__mem_range - next function for for_each_free_mem_range() etc.
* @idx: pointer to u64 loop variable
* @nid: node selector, %NUMA_NO_NODE for all nodes
* @flags: pick from blocks based on memory attributes
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 31/43] memblock, mm: defer initialization of preserved pages
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (29 preceding siblings ...)
2020-05-07 0:41 ` [RFC 30/43] memblock: add for_each_reserved_mem_range() Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 32/43] shmem: PKRAM: preserve shmem files a chunk at a time Anthony Yznaga
` (11 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Preserved pages are represented in the memblock reserved list, but page
structs for pages in the reserved list are initialized early while boot
is single threaded which means that a large number of preserved pages
can impact boot time. To mitigate, defer initialization of preserved
pages by skipping them when other reserved pages are initialized and
initializing them later with a separate kernel thread.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
arch/x86/mm/init_64.c | 1 -
include/linux/mm.h | 2 +-
mm/memblock.c | 10 ++++++++--
mm/page_alloc.c | 52 +++++++++++++++++++++++++++++++++++++++++++--------
4 files changed, 53 insertions(+), 12 deletions(-)
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 72662615977b..ae569ef6bd7d 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -1245,7 +1245,6 @@ void __init mem_init(void)
after_bootmem = 1;
x86_init.hyper.init_after_bootmem();
- pkram_free_pgt();
totalram_pages_add(pkram_reserved_pages);
/*
* Must be done after boot memory is put on freelist, because here we
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 5a323422d783..69b9cd08c721 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2297,7 +2297,7 @@ extern void free_highmem_page(struct page *page);
extern void adjust_managed_page_count(struct page *page, long count);
extern void mem_init_print_info(const char *str);
-extern void reserve_bootmem_region(phys_addr_t start, phys_addr_t end);
+extern void reserve_bootmem_region(phys_addr_t start, phys_addr_t end, int nid);
/* Free the reserved page into the buddy system, so it gets managed. */
static inline void __free_reserved_page(struct page *page)
diff --git a/mm/memblock.c b/mm/memblock.c
index 33597f352dc0..5524edbaf691 100644
--- a/mm/memblock.c
+++ b/mm/memblock.c
@@ -2042,11 +2042,17 @@ static unsigned long __init free_low_memory_core_early(void)
unsigned long count = 0;
phys_addr_t start, end;
u64 i;
+ enum memblock_flags exclude;
memblock_clear_hotplug(0, -1);
- for_each_reserved_mem_region(i, &start, &end)
- reserve_bootmem_region(start, end);
+ if (IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT))
+ exclude = MEMBLOCK_PRESERVED;
+ else
+ exclude = MEMBLOCK_NONE;
+
+ for_each_reserved_mem_range(i, 0, exclude, &start, &end, NULL)
+ reserve_bootmem_region(start, end, -1);
/*
* We need to use NUMA_NO_NODE instead of NODE_DATA(0)->node_id
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 69827d4fa052..afd97b31725e 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -68,6 +68,7 @@
#include <linux/lockdep.h>
#include <linux/nmi.h>
#include <linux/psi.h>
+#include <linux/pkram.h>
#include <asm/sections.h>
#include <asm/tlbflush.h>
@@ -1408,15 +1409,18 @@ static void __meminit __init_single_page(struct page *page, unsigned long pfn,
}
#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
-static void __meminit init_reserved_page(unsigned long pfn)
+static void __meminit init_reserved_page(unsigned long pfn, int nid)
{
pg_data_t *pgdat;
- int nid, zid;
+ int zid;
- if (!early_page_uninitialised(pfn))
- return;
+ if (nid == -1) {
+ if (!early_page_uninitialised(pfn))
+ return;
+
+ nid = early_pfn_to_nid(pfn);
+ }
- nid = early_pfn_to_nid(pfn);
pgdat = NODE_DATA(nid);
for (zid = 0; zid < MAX_NR_ZONES; zid++) {
@@ -1428,7 +1432,7 @@ static void __meminit init_reserved_page(unsigned long pfn)
__init_single_page(pfn_to_page(pfn), pfn, zid, nid);
}
#else
-static inline void init_reserved_page(unsigned long pfn)
+static inline void init_reserved_page(unsigned long pfn, int nid)
{
}
#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */
@@ -1439,7 +1443,7 @@ static inline void init_reserved_page(unsigned long pfn)
* marks the pages PageReserved. The remaining valid pages are later
* sent to the buddy page allocator.
*/
-void __meminit reserve_bootmem_region(phys_addr_t start, phys_addr_t end)
+void __meminit reserve_bootmem_region(phys_addr_t start, phys_addr_t end, int nid)
{
unsigned long start_pfn = PFN_DOWN(start);
unsigned long end_pfn = PFN_UP(end);
@@ -1448,7 +1452,7 @@ void __meminit reserve_bootmem_region(phys_addr_t start, phys_addr_t end)
if (pfn_valid(start_pfn)) {
struct page *page = pfn_to_page(start_pfn);
- init_reserved_page(start_pfn);
+ init_reserved_page(start_pfn, nid);
/* Avoid false-positive PageTail() */
INIT_LIST_HEAD(&page->lru);
@@ -1876,6 +1880,34 @@ static int __init deferred_init_memmap(void *data)
return 0;
}
+#ifdef CONFIG_PKRAM
+static int __init deferred_init_preserved(void *dummy)
+{
+ unsigned long start = jiffies;
+ unsigned long nr_pages = 0;
+ phys_addr_t spa, epa;
+ int nid;
+ u64 i;
+
+ for_each_reserved_mem_range(i, MEMBLOCK_PRESERVED, 0, &spa, &epa, &nid) {
+ reserve_bootmem_region(spa, epa, nid);
+ nr_pages += ((epa - spa) >> PAGE_SHIFT);
+ }
+
+ pr_info("initialised %lu preserved pages in %ums\n", nr_pages,
+ jiffies_to_msecs(jiffies - start));
+
+ /*
+ * Free the preserved pages pagetable now that page structs are
+ * initialized.
+ */
+ pkram_free_pgt();
+
+ pgdat_init_report_one_done();
+ return 0;
+}
+#endif /* CONFIG_PKRAM */
+
/*
* If this zone has deferred pages, try to grow it by initializing enough
* deferred pages to satisfy the allocation specified by order, rounded up to
@@ -1985,6 +2017,10 @@ void __init page_alloc_init_late(void)
/* There will be num_node_state(N_MEMORY) threads */
atomic_set(&pgdat_init_n_undone, num_node_state(N_MEMORY));
+#ifdef CONFIG_PKRAM
+ atomic_inc(&pgdat_init_n_undone);
+ kthread_run(deferred_init_preserved, NULL, "pgdatainit_preserved");
+#endif
for_each_node_state(nid, N_MEMORY) {
kthread_run(deferred_init_memmap, NODE_DATA(nid), "pgdatinit%d", nid);
}
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 32/43] shmem: PKRAM: preserve shmem files a chunk at a time
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (30 preceding siblings ...)
2020-05-07 0:41 ` [RFC 31/43] memblock, mm: defer initialization of preserved pages Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:41 ` [RFC 33/43] PKRAM: atomically add and remove link pages Anthony Yznaga
` (10 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
To prepare for multithreading the work done to a preserve a file,
divide the work into subranges of the total index range of the file.
The chunk size is a rather arbitrary 256k indices.
A new API call, pkram_prepare_save_chunk(), is added. It is called
after calling pkram_prepare_save_obj(), and it initializes pkram_stream
with the index range of the next available range of pages to save.
find_get_pages_range() can then be used to get the pages in the range.
When no more index ranges are available, pkram_prepare_save_chunk()
returns -ENODATA.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/pkram.h | 6 +++++
mm/pkram.c | 26 +++++++++++++++++++++
mm/shmem_pkram.c | 63 +++++++++++++++++++++++++++++++++++----------------
3 files changed, 75 insertions(+), 20 deletions(-)
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index cbb79d2803c0..e71ccb91d6a6 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -20,6 +20,11 @@ struct pkram_stream {
struct address_space *mapping;
struct mm_struct *mm;
+ unsigned long start_idx; /* first index in range to save */
+ unsigned long end_idx; /* last index in range to save */
+ unsigned long max_idx; /* maximum index to save */
+ atomic64_t *next_idx; /* first index of next range to save */
+
/* byte data */
struct page *data_page;
unsigned int data_offset;
@@ -46,6 +51,7 @@ void pkram_free_pgt_walk_pgd(pgd_t *pgd);
int pkram_prepare_save(struct pkram_stream *ps, const char *name,
gfp_t gfp_mask);
int pkram_prepare_save_obj(struct pkram_stream *ps);
+int pkram_prepare_save_chunk(struct pkram_stream *ps);
void pkram_finish_save(struct pkram_stream *ps);
void pkram_finish_save_obj(struct pkram_stream *ps);
void pkram_discard_save(struct pkram_stream *ps);
diff --git a/mm/pkram.c b/mm/pkram.c
index b83d31740619..5f4e4d12865f 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -638,6 +638,25 @@ int pkram_prepare_save(struct pkram_stream *ps, const char *name, gfp_t gfp_mask
return 0;
}
+unsigned long max_pages_per_chunk = 512 * 512;
+
+/*
+ * Initialize the stream @ps for the next index range to save.
+ *
+ * Returns 0 on success, -ENODATA if no index range is available
+ *
+ */
+int pkram_prepare_save_chunk(struct pkram_stream *ps)
+{
+ ps->start_idx = atomic64_fetch_add(max_pages_per_chunk, ps->next_idx);
+ if (ps->start_idx >= ps->max_idx)
+ return -ENODATA;
+
+ ps->end_idx = ps->start_idx + max_pages_per_chunk - 1;
+
+ return 0;
+}
+
/**
* Create a preserved memory object and initialize stream @ps for saving data
* to it.
@@ -667,6 +686,11 @@ int pkram_prepare_save_obj(struct pkram_stream *ps)
obj->obj_pfn = node->obj_pfn;
node->obj_pfn = page_to_pfn(page);
+ ps->next_idx = kmalloc(sizeof(atomic64_t), GFP_KERNEL);
+ if (!ps->next_idx)
+ return -ENOMEM;
+ atomic64_set(ps->next_idx, 0);
+
pkram_stream_init_obj(ps, obj);
return 0;
}
@@ -679,6 +703,8 @@ void pkram_finish_save_obj(struct pkram_stream *ps)
struct pkram_node *node = ps->node;
BUG_ON((node->flags & PKRAM_ACCMODE_MASK) != PKRAM_SAVE);
+
+ kfree(ps->next_idx);
}
/**
diff --git a/mm/shmem_pkram.c b/mm/shmem_pkram.c
index c97d64393822..2f4d0bdf3e05 100644
--- a/mm/shmem_pkram.c
+++ b/mm/shmem_pkram.c
@@ -74,58 +74,81 @@ static int save_page(struct page *page, struct pkram_stream *ps)
return err;
}
-static int save_file_content(struct pkram_stream *ps)
+static int save_file_content_range(struct address_space *mapping,
+ struct pkram_stream *ps)
{
+ unsigned long index, end;
struct pagevec pvec;
- pgoff_t indices[PAGEVEC_SIZE];
- pgoff_t index = 0;
struct page *page;
- int i, err = 0;
+ int err = 0;
+ int i;
+
+ index = ps->start_idx;
+ end = ps->end_idx;
pagevec_init(&pvec);
for ( ; ; ) {
- pvec.nr = find_get_entries(ps->mapping, index, PAGEVEC_SIZE,
- pvec.pages, indices);
+ pvec.nr = find_get_pages_range(mapping, &index, end,
+ PAGEVEC_SIZE, pvec.pages);
if (!pvec.nr)
break;
- for (i = 0; i < pagevec_count(&pvec); i++) {
+ for (i = 0; i < pagevec_count(&pvec); ) {
page = pvec.pages[i];
- index = indices[i];
-
- if (WARN_ON_ONCE(xa_is_value(page))) {
- err = -EINVAL;
- break;
- }
-
lock_page(page);
if (PageTransTail(page)) {
WARN_ONCE(1, "PageTransTail returned true\n");
unlock_page(page);
+ i++;
continue;
}
- BUG_ON(page->mapping != ps->mapping);
+ BUG_ON(page->mapping != mapping);
err = save_page(page, ps);
- i += compound_nr(page) - 1;
- index += compound_nr(page) - 1;
+ if (PageCompound(page)) {
+ index = page->index + compound_nr(page);
+ i += compound_nr(page);
+ } else {
+ i++;
+ }
unlock_page(page);
if (err)
break;
}
- pagevec_remove_exceptionals(&pvec);
pagevec_release(&pvec);
- if (err)
+ if (err || (index > end))
break;
cond_resched();
- index++;
}
return err;
}
+static int do_save_file_content(struct pkram_stream *ps)
+{
+ int ret;
+
+ do {
+ ret = pkram_prepare_save_chunk(ps);
+ if (!ret)
+ ret = save_file_content_range(ps->mapping, ps);
+ } while (!ret);
+
+ if (ret == -ENODATA)
+ ret = 0;
+
+ return ret;
+}
+
+static int save_file_content(struct pkram_stream *ps)
+{
+ ps->max_idx = DIV_ROUND_UP(i_size_read(ps->mapping->host), PAGE_SIZE);
+
+ return do_save_file_content(ps);
+}
+
static int save_file(struct dentry *dentry, struct pkram_stream *ps)
{
struct inode *inode = dentry->d_inode;
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 33/43] PKRAM: atomically add and remove link pages
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (31 preceding siblings ...)
2020-05-07 0:41 ` [RFC 32/43] shmem: PKRAM: preserve shmem files a chunk at a time Anthony Yznaga
@ 2020-05-07 0:41 ` Anthony Yznaga
2020-05-07 0:42 ` [RFC 34/43] shmem: PKRAM: multithread preserving and restoring shmem pages Anthony Yznaga
` (9 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:41 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Add and remove pkram_link pages from a pkram_obj atomically to prepare
for multithreading.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
mm/pkram.c | 27 ++++++++++++++++++---------
1 file changed, 18 insertions(+), 9 deletions(-)
diff --git a/mm/pkram.c b/mm/pkram.c
index 5f4e4d12865f..042c14dedc25 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -551,22 +551,31 @@ static void pkram_truncate(void)
static void pkram_add_link(struct pkram_link *link, struct pkram_obj *obj)
{
- link->link_pfn = obj->link_pfn;
- obj->link_pfn = page_to_pfn(virt_to_page(link));
+ __u64 link_pfn = page_to_pfn(virt_to_page(link));
+ __u64 *head = &obj->link_pfn;
+
+ do {
+ link->link_pfn = *head;
+ } while (cmpxchg64(head, link->link_pfn, link_pfn) != link->link_pfn);
}
static struct pkram_link *pkram_remove_link(struct pkram_obj *obj)
{
struct pkram_link *current_link;
+ __u64 *head = &obj->link_pfn;
+ __u64 head_pfn = *head;
+
+ while (head_pfn) {
+ current_link = pfn_to_kaddr(head_pfn);
+ if (cmpxchg64(head, head_pfn, current_link->link_pfn) == head_pfn) {
+ current_link->link_pfn = 0;
+ return current_link;
+ }
- if (!obj->link_pfn)
- return NULL;
-
- current_link = pfn_to_kaddr(obj->link_pfn);
- obj->link_pfn = current_link->link_pfn;
- current_link->link_pfn = 0;
+ head_pfn = *head;
+ }
- return current_link;
+ return NULL;
}
static void pkram_stream_init(struct pkram_stream *ps,
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 34/43] shmem: PKRAM: multithread preserving and restoring shmem pages
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (32 preceding siblings ...)
2020-05-07 0:41 ` [RFC 33/43] PKRAM: atomically add and remove link pages Anthony Yznaga
@ 2020-05-07 0:42 ` Anthony Yznaga
2020-05-07 16:30 ` Randy Dunlap
2020-05-07 0:42 ` [RFC 35/43] shmem: introduce shmem_insert_pages() Anthony Yznaga
` (8 subsequent siblings)
42 siblings, 1 reply; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:42 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Improve performance by multithreading the work to preserve and restore
shmem pages.
Add 'pkram_max_threads=' kernel option to specify the maximum number
of threads to use to preserve or restore the pages of a shmem file.
The default is 16.
When preserving pages each thread saves chunks of a file to a pkram_obj
until no more no more chunks are available.
When restoring pages each thread loads pages using a copy of a
pkram_stream initialized by pkram_prepare_load_obj(). Under the hood
each thread ends up fetching and operating on pkram_link pages.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/pkram.h | 2 +
mm/shmem_pkram.c | 101 +++++++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 101 insertions(+), 2 deletions(-)
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index e71ccb91d6a6..bf2e138b044e 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -13,6 +13,8 @@ struct pkram_stream {
struct pkram_node *node;
struct pkram_obj *obj;
+ int error;
+
struct pkram_link *link; /* current link */
unsigned int entry_idx; /* next entry in link */
diff --git a/mm/shmem_pkram.c b/mm/shmem_pkram.c
index 2f4d0bdf3e05..4992b6c3e54e 100644
--- a/mm/shmem_pkram.c
+++ b/mm/shmem_pkram.c
@@ -126,6 +126,16 @@ static int save_file_content_range(struct address_space *mapping,
return err;
}
+/* Completion tracking for do_save_file_content_thr() threads */
+static atomic_t pkram_save_n_undone;
+static DECLARE_COMPLETION(pkram_save_all_done_comp);
+
+static inline void pkram_save_report_one_done(void)
+{
+ if (atomic_dec_and_test(&pkram_save_n_undone))
+ complete(&pkram_save_all_done_comp);
+}
+
static int do_save_file_content(struct pkram_stream *ps)
{
int ret;
@@ -142,11 +152,55 @@ static int do_save_file_content(struct pkram_stream *ps)
return ret;
}
+static int do_save_file_content_thr(void *data)
+{
+ struct pkram_stream *ps = data;
+ struct pkram_stream pslocal = *ps;
+ int ret;
+
+ ret = do_save_file_content(&pslocal);
+ if (ret && !ps->error)
+ ps->error = ret;
+
+ pkram_save_report_one_done();
+ return 0;
+}
+#define PKRAM_DEFAULT_MAX_THREADS 16
+
+static int pkram_max_threads = PKRAM_DEFAULT_MAX_THREADS;
+
+static int __init set_pkram_max_threads(char *str)
+{
+ unsigned int val;
+
+ if (kstrtouint(str, 0, &val))
+ return 1;
+
+ pkram_max_threads = val;
+
+ return 1;
+}
+__setup("pkram_max_threads=", set_pkram_max_threads);
+
static int save_file_content(struct pkram_stream *ps)
{
+ unsigned int thr, nr_threads;
+
+ nr_threads = num_online_cpus() - 1;
+ nr_threads = clamp_val(pkram_max_threads, 1, nr_threads);
+
ps->max_idx = DIV_ROUND_UP(i_size_read(ps->mapping->host), PAGE_SIZE);
- return do_save_file_content(ps);
+ if (nr_threads == 1)
+ return do_save_file_content(ps);
+
+ atomic_set(&pkram_save_n_undone, nr_threads);
+ for (thr = 0; thr < nr_threads; thr++)
+ kthread_run(do_save_file_content_thr, ps, "pkram_save%d", thr);
+
+ wait_for_completion(&pkram_save_all_done_comp);
+
+ return ps->error;
}
static int save_file(struct dentry *dentry, struct pkram_stream *ps)
@@ -248,7 +302,17 @@ int shmem_save_pkram(struct super_block *sb)
return err;
}
-static int load_file_content(struct pkram_stream *ps)
+/* Completion tracking for do_load_file_content_thr() threads */
+static atomic_t pkram_load_n_undone;
+static DECLARE_COMPLETION(pkram_load_all_done_comp);
+
+static inline void pkram_load_report_one_done(void)
+{
+ if (atomic_dec_and_test(&pkram_load_n_undone))
+ complete(&pkram_load_all_done_comp);
+}
+
+static int do_load_file_content(struct pkram_stream *ps)
{
unsigned long index;
struct page *page;
@@ -266,6 +330,39 @@ static int load_file_content(struct pkram_stream *ps)
return err;
}
+static int do_load_file_content_thr(void *data)
+{
+ struct pkram_stream *ps = data;
+ struct pkram_stream pslocal = *ps;
+ int ret;
+
+ ret = do_load_file_content(&pslocal);
+ if (ret && !ps->error)
+ ps->error = ret;
+
+ pkram_load_report_one_done();
+ return 0;
+}
+
+static int load_file_content(struct pkram_stream *ps)
+{
+ unsigned int thr, nr_threads;
+
+ nr_threads = num_online_cpus() - 1;
+ nr_threads = clamp_val(pkram_max_threads, 1, nr_threads);
+
+ if (nr_threads == 1)
+ return do_load_file_content(ps);
+
+ atomic_set(&pkram_load_n_undone, nr_threads);
+ for (thr = 0; thr < nr_threads; thr++)
+ kthread_run(do_load_file_content_thr, ps, "pkram_load%d", thr);
+
+ wait_for_completion(&pkram_load_all_done_comp);
+
+ return ps->error;
+}
+
static int load_file(struct dentry *parent, struct pkram_stream *ps,
char *buf, size_t bufsize)
{
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* Re: [RFC 34/43] shmem: PKRAM: multithread preserving and restoring shmem pages
2020-05-07 0:42 ` [RFC 34/43] shmem: PKRAM: multithread preserving and restoring shmem pages Anthony Yznaga
@ 2020-05-07 16:30 ` Randy Dunlap
2020-05-07 17:59 ` Anthony Yznaga
0 siblings, 1 reply; 50+ messages in thread
From: Randy Dunlap @ 2020-05-07 16:30 UTC (permalink / raw)
To: Anthony Yznaga, linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
On 5/6/20 5:42 PM, Anthony Yznaga wrote:
> Improve performance by multithreading the work to preserve and restore
> shmem pages.
>
> Add 'pkram_max_threads=' kernel option to specify the maximum number
> of threads to use to preserve or restore the pages of a shmem file.
> The default is 16.
Hi,
Please document kernel boot options in Documentation/admin-guide/kernel-parameters.txt.
> When preserving pages each thread saves chunks of a file to a pkram_obj
> until no more no more chunks are available.
>
> When restoring pages each thread loads pages using a copy of a
> pkram_stream initialized by pkram_prepare_load_obj(). Under the hood
> each thread ends up fetching and operating on pkram_link pages.
>
> Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
> ---
> include/linux/pkram.h | 2 +
> mm/shmem_pkram.c | 101 +++++++++++++++++++++++++++++++++++++++++++++++++-
> 2 files changed, 101 insertions(+), 2 deletions(-)
thanks.
--
~Randy
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [RFC 34/43] shmem: PKRAM: multithread preserving and restoring shmem pages
2020-05-07 16:30 ` Randy Dunlap
@ 2020-05-07 17:59 ` Anthony Yznaga
0 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 17:59 UTC (permalink / raw)
To: Randy Dunlap, linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
On 5/7/20 9:30 AM, Randy Dunlap wrote:
> On 5/6/20 5:42 PM, Anthony Yznaga wrote:
>> Improve performance by multithreading the work to preserve and restore
>> shmem pages.
>>
>> Add 'pkram_max_threads=' kernel option to specify the maximum number
>> of threads to use to preserve or restore the pages of a shmem file.
>> The default is 16.
> Hi,
> Please document kernel boot options in Documentation/admin-guide/kernel-parameters.txt.
I'll do that. Thanks!
Anthony
>
>> When preserving pages each thread saves chunks of a file to a pkram_obj
>> until no more no more chunks are available.
>>
>> When restoring pages each thread loads pages using a copy of a
>> pkram_stream initialized by pkram_prepare_load_obj(). Under the hood
>> each thread ends up fetching and operating on pkram_link pages.
>>
>> Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
>> ---
>> include/linux/pkram.h | 2 +
>> mm/shmem_pkram.c | 101 +++++++++++++++++++++++++++++++++++++++++++++++++-
>> 2 files changed, 101 insertions(+), 2 deletions(-)
> thanks.
^ permalink raw reply [flat|nested] 50+ messages in thread
* [RFC 35/43] shmem: introduce shmem_insert_pages()
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (33 preceding siblings ...)
2020-05-07 0:42 ` [RFC 34/43] shmem: PKRAM: multithread preserving and restoring shmem pages Anthony Yznaga
@ 2020-05-07 0:42 ` Anthony Yznaga
2020-05-07 0:42 ` [RFC 36/43] PKRAM: add support for loading pages in bulk Anthony Yznaga
` (7 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:42 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Calling shmem_insert_page() to insert one page at a time does
not scale well when multiple threads are inserting pages into
the same shmem segment. This is primarily due to the locking needed
when adding to the pagecache and LRU but also due to contention
on the shmem_inode_info lock. To address the shmem_inode_info lock
and prepare for future optimizations, introduce shmem_insert_pages()
which allows a caller to pass an array of pages to be inserted into a
shmem segment.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/shmem_fs.h | 3 +-
mm/shmem.c | 114 +++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 116 insertions(+), 1 deletion(-)
diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h
index f2ce9937a8f2..d308a6a154b6 100644
--- a/include/linux/shmem_fs.h
+++ b/include/linux/shmem_fs.h
@@ -105,7 +105,8 @@ extern int shmem_getpage(struct inode *inode, pgoff_t index,
extern int shmem_insert_page(struct mm_struct *mm, struct inode *inode,
pgoff_t index, struct page *page);
-
+extern int shmem_insert_pages(struct mm_struct *mm, struct inode *inode,
+ pgoff_t index, struct page *pages[], int npages);
#ifdef CONFIG_PKRAM
extern int shmem_parse_pkram(const char *str, struct shmem_pkram_info **pkram);
extern void shmem_show_pkram(struct seq_file *seq, struct shmem_pkram_info *pkram,
diff --git a/mm/shmem.c b/mm/shmem.c
index 1f3b43b8fa34..ca5edf580f24 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -781,6 +781,120 @@ int shmem_insert_page(struct mm_struct *mm, struct inode *inode, pgoff_t index,
return err;
}
+int shmem_insert_pages(struct mm_struct *mm, struct inode *inode, pgoff_t index,
+ struct page *pages[], int npages)
+{
+ struct address_space *mapping = inode->i_mapping;
+ struct shmem_inode_info *info = SHMEM_I(inode);
+ struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
+ gfp_t gfp = mapping_gfp_mask(mapping);
+ struct mem_cgroup *memcg;
+ int i, err;
+ int nr = 0;
+
+ for (i = 0; i < npages; i++)
+ nr += compound_nr(pages[i]);
+
+ if (index + nr - 1 > (MAX_LFS_FILESIZE >> PAGE_SHIFT))
+ return -EFBIG;
+
+retry:
+ err = 0;
+ if (!shmem_inode_acct_block(inode, nr))
+ err = -ENOSPC;
+ if (err) {
+ int retry = 5;
+
+ /*
+ * Try to reclaim some space by splitting a huge page
+ * beyond i_size on the filesystem.
+ */
+ while (retry--) {
+ int ret;
+
+ ret = shmem_unused_huge_shrink(sbinfo, NULL, 1);
+ if (ret == SHRINK_STOP)
+ break;
+ if (ret)
+ goto retry;
+ }
+ goto failed;
+ }
+
+ for (i = 0; i < npages; i++) {
+ if (!PageLRU(pages[i])) {
+ __SetPageLocked(pages[i]);
+ __SetPageSwapBacked(pages[i]);
+ } else {
+ lock_page(pages[i]);
+ }
+
+ __SetPageReferenced(pages[i]);
+
+ if (pages[i]->mem_cgroup)
+ continue;
+
+ err = mem_cgroup_try_charge_delay(pages[i], mm, gfp,
+ &memcg, PageTransHuge(pages[i]));
+ if (err)
+ goto out_unlock;
+
+ }
+
+ for (i = 0; i < npages; i++) {
+ err = shmem_add_to_page_cache(pages[i], mapping, index,
+ NULL, gfp & GFP_RECLAIM_MASK);
+ if (err)
+ goto out_truncate;
+
+ if (PageTransHuge(pages[i]))
+ index += HPAGE_PMD_NR;
+ else
+ index++;
+ }
+
+ spin_lock(&info->lock);
+ info->alloced += nr;
+ inode->i_blocks += BLOCKS_PER_PAGE * nr;
+ shmem_recalc_inode(inode);
+ spin_unlock(&info->lock);
+
+ for (i = 0; i < npages; i++) {
+ if (!pages[i]->mem_cgroup) {
+ mem_cgroup_commit_charge(pages[i], memcg,
+ PageLRU(pages[i]), PageTransHuge(pages[i]));
+ }
+
+ if (!PageLRU(pages[i]))
+ lru_cache_add_anon(pages[i]);
+
+ flush_dcache_page(pages[i]);
+ SetPageUptodate(pages[i]);
+ set_page_dirty(pages[i]);
+
+ unlock_page(pages[i]);
+ }
+
+ return 0;
+
+out_truncate:
+ while (--i >= 0)
+ truncate_inode_page(mapping, pages[i]);
+ i = npages;
+out_unlock:
+ while (--i >= 0) {
+ if (!pages[i]->mem_cgroup) {
+ mem_cgroup_cancel_charge(pages[i], memcg,
+ PageTransHuge(pages[i]));
+ }
+
+ unlock_page(pages[i]);
+ }
+ shmem_inode_unacct_blocks(inode, nr);
+failed:
+ return err;
+}
+
/*
* Remove swap entry from page cache, free the swap and its page cache.
*/
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 36/43] PKRAM: add support for loading pages in bulk
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (34 preceding siblings ...)
2020-05-07 0:42 ` [RFC 35/43] shmem: introduce shmem_insert_pages() Anthony Yznaga
@ 2020-05-07 0:42 ` Anthony Yznaga
2020-05-07 0:42 ` [RFC 37/43] shmem: PKRAM: enable bulk loading of preserved pages into shmem Anthony Yznaga
` (6 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:42 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
This patch adds three functions:
pkram_prepare_load_pages()
Called after calling pkram_prepare_load_obj()
pkram_load_pages()
Loads some number of pages that are contiguous by their original
file index values. The index of the first page, an array of the
page pointers, and the number of pages in the array are provided
to the caller.
pkram_finish_load_pages()
Called when no more pages will be loaded from the pkram_obj.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/pkram.h | 6 +++
mm/pkram.c | 106 ++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 112 insertions(+)
diff --git a/include/linux/pkram.h b/include/linux/pkram.h
index bf2e138b044e..3f059791f88c 100644
--- a/include/linux/pkram.h
+++ b/include/linux/pkram.h
@@ -18,6 +18,9 @@ struct pkram_stream {
struct pkram_link *link; /* current link */
unsigned int entry_idx; /* next entry in link */
+ struct page **pages;
+ unsigned int nr_pages;
+
unsigned long next_index;
struct address_space *mapping;
struct mm_struct *mm;
@@ -60,14 +63,17 @@ void pkram_discard_save(struct pkram_stream *ps);
int pkram_prepare_load(struct pkram_stream *ps, const char *name);
int pkram_prepare_load_obj(struct pkram_stream *ps);
+int pkram_prepare_load_pages(struct pkram_stream *ps);
void pkram_finish_load(struct pkram_stream *ps);
void pkram_finish_load_obj(struct pkram_stream *ps);
+void pkram_finish_load_pages(struct pkram_stream *ps);
#define PKRAM_PAGE_TRANS_HUGE 0x1 /* page is a transparent hugepage */
int pkram_save_page(struct pkram_stream *ps, struct page *page, short flags);
struct page *pkram_load_page(struct pkram_stream *ps, unsigned long *index,
short *flags);
+int pkram_load_pages(struct pkram_stream *ps, unsigned long *index);
ssize_t pkram_write(struct pkram_stream *ps, const void *buf, size_t count);
size_t pkram_read(struct pkram_stream *ps, void *buf, size_t count);
diff --git a/mm/pkram.c b/mm/pkram.c
index 042c14dedc25..ef092aa5ce7a 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -820,6 +820,37 @@ int pkram_prepare_load_obj(struct pkram_stream *ps)
}
/**
+ * Initialize stream @ps for loading preserved pages from it.
+ *
+ * Returns 0 on success, -errno on failure.
+ *
+ * Error values:
+ * %ENOMEM: insufficient memory available
+ *
+ * After the load has finished, pkram_finish_load_pages() is to be called.
+ */
+int pkram_prepare_load_pages(struct pkram_stream *ps)
+{
+ BUG_ON((ps->node->flags & PKRAM_ACCMODE_MASK) != PKRAM_LOAD);
+
+ ps->pages = kzalloc(PAGE_SIZE, GFP_KERNEL);
+ if (!ps->pages)
+ return -ENOMEM;
+
+ return 0;
+}
+
+/**
+ * Finish the load of preserved pages started with pkram_prepare_load_pages()
+ */
+void pkram_finish_load_pages(struct pkram_stream *ps)
+{
+ BUG_ON((ps->node->flags & PKRAM_ACCMODE_MASK) != PKRAM_LOAD);
+
+ kfree(ps->pages);
+}
+
+/**
* Finish the load of a preserved memory object started with
* pkram_prepare_load_obj() freeing the object and any data that has not
* been loaded from it.
@@ -1066,6 +1097,81 @@ struct page *pkram_load_page(struct pkram_stream *ps, unsigned long *index, shor
}
/**
+ * Load pages from the preserved memory object associated with stream
+ * @ps. The stream must have been initialized with pkram_prepare_load(),
+ * pkram_prepare_load_obj(), and pkram_prepare_load_pages().
+ * The page entries of a single pkram_link are processed, and the stream
+ * 'pages' buffer is populated with the page pointers.
+ *
+ * Returns 0 if one or more pages are loaded or -ENODATA if there are no
+ * pages to load.
+ *
+ * The pages loaded have an incremented refcount either because the page
+ * was initialized with a refcount of 1 at boot or because the page was
+ * subsequently preserved which increased the refcount.
+ */
+int pkram_load_pages(struct pkram_stream *ps, unsigned long *index)
+{
+ struct pkram_link *link = ps->link;
+ int nr_entries = 0;
+ int i;
+
+ if (!link) {
+ link = pkram_remove_link(ps->obj);
+ if (!link)
+ return -ENODATA;
+ }
+
+ *index = link->index;
+
+ for (i = 0; i < PKRAM_LINK_ENTRIES_MAX; i++) {
+ unsigned long p = link->entry[i];
+ struct page *page;
+ short flags;
+
+ if (!p)
+ break;
+
+ flags = (p >> PKRAM_ENTRY_FLAGS_SHIFT) & PKRAM_ENTRY_FLAGS_MASK;
+ nr_entries++;
+
+ page = pfn_to_page(PHYS_PFN(p));
+ ps->pages[i] = page;
+
+ if (flags & PKRAM_PAGE_TRANS_HUGE) {
+ int order = p & PKRAM_ENTRY_ORDER_MASK;
+ int nr_pages = 1 << order;
+ int j;
+
+ for (j = 0; j < nr_pages; j++) {
+ struct page *p = page + j;
+
+ ClearPageReserved(p);
+ }
+
+ prep_compound_page(page, order);
+ prep_transhuge_page(page);
+ } else {
+ ClearPageReserved(page);
+ }
+
+ pkram_remove_identity_map(page);
+ }
+
+ ps->nr_pages = nr_entries;
+
+ /* Advance to next pkram_link page and free this one */
+ if (link->link_pfn)
+ ps->link = pfn_to_kaddr(link->link_pfn);
+ else
+ ps->link = NULL;
+
+ pkram_free_page(link);
+
+ return 0;
+}
+
+/**
* Copy @count bytes from @buf to the preserved memory node and object
* associated with stream @ps. The stream must have been initialized with
* pkram_prepare_save() and pkram_prepare_save_obj().
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 37/43] shmem: PKRAM: enable bulk loading of preserved pages into shmem
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (35 preceding siblings ...)
2020-05-07 0:42 ` [RFC 36/43] PKRAM: add support for loading pages in bulk Anthony Yznaga
@ 2020-05-07 0:42 ` Anthony Yznaga
2020-05-07 0:42 ` [RFC 38/43] mm: implement splicing a list of pages to the LRU Anthony Yznaga
` (5 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:42 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Make use of new interfaces for loading and inserting preserved pages
into a shmem file in bulk.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
mm/shmem_pkram.c | 23 +++++++++++++++++------
1 file changed, 17 insertions(+), 6 deletions(-)
diff --git a/mm/shmem_pkram.c b/mm/shmem_pkram.c
index 4992b6c3e54e..435488368104 100644
--- a/mm/shmem_pkram.c
+++ b/mm/shmem_pkram.c
@@ -315,18 +315,29 @@ static inline void pkram_load_report_one_done(void)
static int do_load_file_content(struct pkram_stream *ps)
{
unsigned long index;
- struct page *page;
- int err = 0;
+ int i, err;
+
+ err = pkram_prepare_load_pages(ps);
+ if (err)
+ return err;
do {
- page = pkram_load_page(ps, &index, NULL);
- if (!page)
+ err = pkram_load_pages(ps, &index);
+ if (err) {
+ if (err == -ENODATA)
+ err = 0;
break;
+ }
- err = shmem_insert_page(ps->mm, ps->mapping->host, index, page);
- put_page(page);
+ err = shmem_insert_pages(ps->mm, ps->mapping->host, index,
+ ps->pages, ps->nr_pages);
+
+ for (i = 0; i < ps->nr_pages; i++)
+ put_page(ps->pages[i]);
} while (!err);
+ pkram_finish_load_pages(ps);
+
return err;
}
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 38/43] mm: implement splicing a list of pages to the LRU
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (36 preceding siblings ...)
2020-05-07 0:42 ` [RFC 37/43] shmem: PKRAM: enable bulk loading of preserved pages into shmem Anthony Yznaga
@ 2020-05-07 0:42 ` Anthony Yznaga
2020-05-07 0:42 ` [RFC 39/43] shmem: optimize adding pages to the LRU in shmem_insert_pages() Anthony Yznaga
` (4 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:42 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Considerable contention on the LRU lock happens when multiple threads
are used to insert pages into a shmem file in parallel. To alleviate this
provide a way for pages to be added to the same LRU to be staged so that
they can be added by splicing lists and updating stats once with the lock
held. For now only unevictable pages are supported.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
include/linux/swap.h | 11 ++++++
mm/swap.c | 101 +++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 112 insertions(+)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index e1bbf7a16b27..462045f536a8 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -346,6 +346,17 @@ extern void swap_setup(void);
extern void lru_cache_add_active_or_unevictable(struct page *page,
struct vm_area_struct *vma);
+struct lru_splice {
+ struct list_head splice;
+ struct list_head *lru_head;
+ struct pglist_data *pgdat;
+ struct lruvec *lruvec;
+ enum lru_list lru;
+ unsigned long nr_pages[MAX_NR_ZONES];
+ unsigned long pgculled;
+};
+extern void lru_splice_add_anon(struct page *page, struct lru_splice *splice);
+extern void add_splice_to_lru_list(struct lru_splice *splice);
/* linux/mm/vmscan.c */
extern unsigned long zone_reclaimable_pages(struct zone *zone);
diff --git a/mm/swap.c b/mm/swap.c
index bf9a79fed62d..848f8b516471 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -187,6 +187,107 @@ int get_kernel_page(unsigned long start, int write, struct page **pages)
}
EXPORT_SYMBOL_GPL(get_kernel_page);
+/*
+ * Update stats and move accumulated pages from an lru_splice to the lru.
+ */
+void add_splice_to_lru_list(struct lru_splice *splice)
+{
+ struct pglist_data *pgdat = splice->pgdat;
+ struct lruvec *lruvec = splice->lruvec;
+ enum lru_list lru = splice->lru;
+ unsigned long flags = 0;
+ int zid;
+
+ spin_lock_irqsave(&pgdat->lru_lock, flags);
+ for (zid = 0; zid < MAX_NR_ZONES; zid++) {
+ if (splice->nr_pages[zid])
+ update_lru_size(lruvec, lru, zid, splice->nr_pages[zid]);
+ }
+ count_vm_events(UNEVICTABLE_PGCULLED, splice->pgculled);
+ list_splice(&splice->splice, splice->lru_head);
+ spin_unlock_irqrestore(&pgdat->lru_lock, flags);
+
+ splice->lru_head = NULL;
+ splice->pgculled = 0;
+}
+
+static void add_page_to_lru_splice(struct page *page,
+ struct lru_splice *splice,
+ struct lruvec *lruvec, enum lru_list lru)
+{
+ int zid;
+
+ if (splice->lru_head == &lruvec->lists[lru]) {
+ list_add(&page->lru, &splice->splice);
+ splice->nr_pages[page_zonenum(page)] += hpage_nr_pages(page);
+ return;
+ }
+
+ INIT_LIST_HEAD(&splice->splice);
+ splice->lruvec = lruvec;
+ splice->lru_head = &lruvec->lists[lru];
+ splice->lru = lru;
+ list_add(&page->lru, &splice->splice);
+ for (zid = 0; zid < MAX_NR_ZONES; zid++)
+ splice->nr_pages[zid] = 0;
+ splice->nr_pages[page_zonenum(page)] = hpage_nr_pages(page);
+
+}
+
+/*
+ * Similar in functionality to __pagevec_lru_add_fn() but here the page is
+ * being added to an lru_splice and the LRU lock is not held.
+ */
+static void page_lru_splice_add(struct page *page, struct lruvec *lruvec, struct lru_splice *splice)
+{
+ enum lru_list lru;
+ int was_unevictable = TestClearPageUnevictable(page);
+
+ VM_BUG_ON_PAGE(PageLRU(page), page);
+ /* XXX only supports unevictable pages at the moment */
+ VM_BUG_ON_PAGE(was_unevictable, page);
+
+ SetPageLRU(page);
+ smp_mb();
+
+ lru = LRU_UNEVICTABLE;
+ ClearPageActive(page);
+ SetPageUnevictable(page);
+ if (!was_unevictable)
+ splice->pgculled++;
+
+ add_page_to_lru_splice(page, splice, lruvec, lru);
+ trace_mm_lru_insertion(page, lru);
+}
+
+static void lru_splice_add(struct page *page, struct lru_splice *splice)
+{
+ struct pglist_data *pagepgdat, *pgdat = splice->pgdat;
+ struct lruvec *lruvec;
+
+ pagepgdat = page_pgdat(page);
+
+ if (pagepgdat != pgdat) {
+ if (pgdat)
+ add_splice_to_lru_list(splice);
+ splice->pgdat = pagepgdat;
+ }
+
+ lruvec = mem_cgroup_page_lruvec(page, pagepgdat);
+ page_lru_splice_add(page, lruvec, splice);
+}
+
+void lru_splice_add_anon(struct page *page, struct lru_splice *splice)
+{
+ if (PageActive(page))
+ ClearPageActive(page);
+ get_page(page);
+
+ lru_splice_add(page, splice);
+
+ put_page(page);
+}
+
static void pagevec_lru_move_fn(struct pagevec *pvec,
void (*move_fn)(struct page *page, struct lruvec *lruvec, void *arg),
void *arg)
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 39/43] shmem: optimize adding pages to the LRU in shmem_insert_pages()
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (37 preceding siblings ...)
2020-05-07 0:42 ` [RFC 38/43] mm: implement splicing a list of pages to the LRU Anthony Yznaga
@ 2020-05-07 0:42 ` Anthony Yznaga
2020-05-07 0:42 ` [RFC 40/43] shmem: initial support for adding multiple pages to pagecache Anthony Yznaga
` (3 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:42 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Reduce LRU lock contention when inserting shmem pages by staging pages
to be added to the same LRU and adding them en masse.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
mm/shmem.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/mm/shmem.c b/mm/shmem.c
index ca5edf580f24..678a396ba8d3 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -789,9 +789,12 @@ int shmem_insert_pages(struct mm_struct *mm, struct inode *inode, pgoff_t index,
struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
gfp_t gfp = mapping_gfp_mask(mapping);
struct mem_cgroup *memcg;
+ struct lru_splice splice;
int i, err;
int nr = 0;
+ memset(&splice, 0, sizeof(splice));
+
for (i = 0; i < npages; i++)
nr += compound_nr(pages[i]);
@@ -866,7 +869,7 @@ int shmem_insert_pages(struct mm_struct *mm, struct inode *inode, pgoff_t index,
}
if (!PageLRU(pages[i]))
- lru_cache_add_anon(pages[i]);
+ lru_splice_add_anon(pages[i], &splice);
flush_dcache_page(pages[i]);
SetPageUptodate(pages[i]);
@@ -875,6 +878,9 @@ int shmem_insert_pages(struct mm_struct *mm, struct inode *inode, pgoff_t index,
unlock_page(pages[i]);
}
+ if (splice.pgdat)
+ add_splice_to_lru_list(&splice);
+
return 0;
out_truncate:
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 40/43] shmem: initial support for adding multiple pages to pagecache
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (38 preceding siblings ...)
2020-05-07 0:42 ` [RFC 39/43] shmem: optimize adding pages to the LRU in shmem_insert_pages() Anthony Yznaga
@ 2020-05-07 0:42 ` Anthony Yznaga
2020-05-07 0:42 ` [RFC 41/43] XArray: add xas_export_node() and xas_import_node() Anthony Yznaga
` (2 subsequent siblings)
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:42 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
shmem_insert_pages() currently loops over the array of pages passed
to it and calls shmem_add_to_page_cache() for each one. Prepare
for adding pages to the pagecache in bulk by adding and using a
shmem_add_pages_to_cache() call. For now it just iterates over
an array and adds pages individually, but improvements in performance
when multiple threads are adding to the same pagecache are achieved
by calling a new shmem_add_to_page_cache_fast() function that does
not check for conflicts and drops the xarray lock before updating stats.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
mm/shmem.c | 95 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--------
1 file changed, 84 insertions(+), 11 deletions(-)
diff --git a/mm/shmem.c b/mm/shmem.c
index 678a396ba8d3..f621d863e362 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -660,6 +660,57 @@ static int shmem_add_to_page_cache(struct page *page,
return 0;
}
+static int shmem_add_to_page_cache_fast(struct page *page,
+ struct address_space *mapping,
+ pgoff_t index, gfp_t gfp)
+{
+ XA_STATE_ORDER(xas, &mapping->i_pages, index, compound_order(page));
+ unsigned long nr = compound_nr(page);
+ unsigned long i = 0;
+
+ VM_BUG_ON_PAGE(PageTail(page), page);
+ VM_BUG_ON_PAGE(index != round_down(index, nr), page);
+ VM_BUG_ON_PAGE(!PageLocked(page), page);
+ VM_BUG_ON_PAGE(!PageSwapBacked(page), page);
+
+ page_ref_add(page, nr);
+ page->mapping = mapping;
+ page->index = index;
+
+ do {
+ xas_lock_irq(&xas);
+ xas_create_range(&xas);
+ if (xas_error(&xas))
+ goto unlock;
+next:
+ xas_store(&xas, page);
+ if (++i < nr) {
+ xas_next(&xas);
+ goto next;
+ }
+ mapping->nrpages += nr;
+ xas_unlock(&xas);
+ if (PageTransHuge(page)) {
+ count_vm_event(THP_FILE_ALLOC);
+ __inc_node_page_state(page, NR_SHMEM_THPS);
+ }
+ __mod_node_page_state(page_pgdat(page), NR_FILE_PAGES, nr);
+ __mod_node_page_state(page_pgdat(page), NR_SHMEM, nr);
+ local_irq_enable();
+ break;
+unlock:
+ xas_unlock_irq(&xas);
+ } while (xas_nomem(&xas, gfp));
+
+ if (xas_error(&xas)) {
+ page->mapping = NULL;
+ page_ref_sub(page, nr);
+ return xas_error(&xas);
+ }
+
+ return 0;
+}
+
/*
* Like delete_from_page_cache, but substitutes swap for page.
*/
@@ -681,6 +732,35 @@ static void shmem_delete_from_page_cache(struct page *page, void *radswap)
BUG_ON(error);
}
+static int shmem_add_pages_to_cache(struct page *pages[], int npages,
+ struct address_space *mapping,
+ pgoff_t start, gfp_t gfp)
+{
+ pgoff_t index = start;
+ int err = 0;
+ int i;
+
+ i = 0;
+ while (i < npages) {
+ if (PageTransHuge(pages[i])) {
+ err = shmem_add_to_page_cache_fast(pages[i], mapping, index, gfp);
+ if (err)
+ break;
+ index += HPAGE_PMD_NR;
+ i++;
+ continue;
+ }
+
+ err = shmem_add_to_page_cache_fast(pages[i], mapping, index, gfp);
+ if (err)
+ break;
+ index++;
+ i++;
+ }
+
+ return err;
+}
+
int shmem_insert_page(struct mm_struct *mm, struct inode *inode, pgoff_t index,
struct page *page)
{
@@ -844,17 +924,10 @@ int shmem_insert_pages(struct mm_struct *mm, struct inode *inode, pgoff_t index,
}
- for (i = 0; i < npages; i++) {
- err = shmem_add_to_page_cache(pages[i], mapping, index,
- NULL, gfp & GFP_RECLAIM_MASK);
- if (err)
- goto out_truncate;
-
- if (PageTransHuge(pages[i]))
- index += HPAGE_PMD_NR;
- else
- index++;
- }
+ err = shmem_add_pages_to_cache(pages, npages, mapping, index,
+ gfp & GFP_RECLAIM_MASK);
+ if (err)
+ goto out_truncate;
spin_lock(&info->lock);
info->alloced += nr;
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 41/43] XArray: add xas_export_node() and xas_import_node()
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (39 preceding siblings ...)
2020-05-07 0:42 ` [RFC 40/43] shmem: initial support for adding multiple pages to pagecache Anthony Yznaga
@ 2020-05-07 0:42 ` Anthony Yznaga
2020-05-07 0:42 ` [RFC 42/43] shmem: reduce time holding xa_lock when inserting pages Anthony Yznaga
2020-05-07 0:42 ` [RFC 43/43] PKRAM: improve index alignment of pkram_link entries Anthony Yznaga
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:42 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Contention on the xarray lock when multiple threads are adding to the
same xarray can be mitigated by providing a way to add entries in
bulk.
Allow a caller to allocate and populate an xarray node outside of
the target xarray and then only take the xarray lock long enough to
import the node into it.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
Documentation/core-api/xarray.rst | 8 +++
include/linux/xarray.h | 2 +
lib/test_xarray.c | 45 +++++++++++++++++
lib/xarray.c | 100 ++++++++++++++++++++++++++++++++++++++
4 files changed, 155 insertions(+)
diff --git a/Documentation/core-api/xarray.rst b/Documentation/core-api/xarray.rst
index 640934b6f7b4..659e10df8901 100644
--- a/Documentation/core-api/xarray.rst
+++ b/Documentation/core-api/xarray.rst
@@ -444,6 +444,14 @@ called each time the XArray updates a node. This is used by the page
cache workingset code to maintain its list of nodes which contain only
shadow entries.
+xas_export_node() is used to remove and return a node from an XArray
+while xas_import_node() is used to add a node to an XArray. Together
+these can be used, for example, to reduce lock contention when multiple
+threads are updating an XArray by allowing a caller to allocate and
+populate a node outside of the target XArray in a local XArray, export
+the node, and then take the target XArray lock just long enough to import
+the node.
+
Multi-Index Entries
-------------------
diff --git a/include/linux/xarray.h b/include/linux/xarray.h
index 14c893433139..73bd8ccc4424 100644
--- a/include/linux/xarray.h
+++ b/include/linux/xarray.h
@@ -1504,6 +1504,8 @@ bool xas_nomem(struct xa_state *, gfp_t);
void xas_pause(struct xa_state *);
void xas_create_range(struct xa_state *);
+struct xa_node *xas_export_node(struct xa_state *xas);
+void xas_import_node(struct xa_state *xas, struct xa_node *node);
/**
* xas_reload() - Refetch an entry from the xarray.
diff --git a/lib/test_xarray.c b/lib/test_xarray.c
index d4f97925dbd8..5cfaa1720cc1 100644
--- a/lib/test_xarray.c
+++ b/lib/test_xarray.c
@@ -1682,6 +1682,50 @@ static noinline void check_destroy(struct xarray *xa)
#endif
}
+static noinline void check_export_import_1(struct xarray *xa,
+ unsigned long index, unsigned int order)
+{
+ int xa_shift = order + XA_CHUNK_SHIFT - (order % XA_CHUNK_SHIFT);
+ XA_STATE(xas, xa, index);
+ struct xa_node *node;
+ unsigned long i;
+
+ xa_store_many_order(xa, index, xa_shift);
+
+ xas_lock(&xas);
+ xas_set_order(&xas, index, xa_shift);
+ node = xas_export_node(&xas);
+ xas_unlock(&xas);
+
+ XA_BUG_ON(xa, !xa_empty(xa));
+
+ do {
+ xas_lock(&xas);
+ xas_set_order(&xas, index, xa_shift);
+ xas_import_node(&xas, node);
+ xas_unlock(&xas);
+ } while (xas_nomem(&xas, GFP_KERNEL));
+
+ for (i = index; i < index + (1UL << xa_shift); i++)
+ xa_erase_index(xa, i);
+
+ XA_BUG_ON(xa, !xa_empty(xa));
+}
+
+static noinline void check_export_import(struct xarray *xa)
+{
+ unsigned int order;
+ unsigned int max_order = IS_ENABLED(CONFIG_XARRAY_MULTI) ? 12 : 1;
+
+ for (order = 0; order < max_order; order += XA_CHUNK_SHIFT) {
+ int xa_shift = order + XA_CHUNK_SHIFT;
+ unsigned long j;
+
+ for (j = 0; j < XA_CHUNK_SIZE; j++)
+ check_export_import_1(xa, j << xa_shift, order);
+ }
+}
+
static DEFINE_XARRAY(array);
static int xarray_checks(void)
@@ -1712,6 +1756,7 @@ static int xarray_checks(void)
check_workingset(&array, 0);
check_workingset(&array, 64);
check_workingset(&array, 4096);
+ check_export_import(&array);
printk("XArray: %u of %u tests passed\n", tests_passed, tests_run);
return (tests_run == tests_passed) ? 0 : -EINVAL;
diff --git a/lib/xarray.c b/lib/xarray.c
index e9e641d3c0c3..478925780e87 100644
--- a/lib/xarray.c
+++ b/lib/xarray.c
@@ -507,6 +507,30 @@ static void xas_delete_node(struct xa_state *xas)
xas_shrink(xas);
}
+static void xas_unlink_node(struct xa_state *xas)
+{
+ struct xa_node *node = xas->xa_node;
+ struct xa_node *parent;
+
+ parent = xa_parent_locked(xas->xa, node);
+ xas->xa_node = parent;
+ xas->xa_offset = node->offset;
+
+ if (!parent) {
+ xas->xa->xa_head = NULL;
+ xas->xa_node = XAS_BOUNDS;
+ return;
+ }
+
+ parent->slots[xas->xa_offset] = NULL;
+ parent->count--;
+ XA_NODE_BUG_ON(parent, parent->count > XA_CHUNK_SIZE);
+
+ xas_update(xas, parent);
+
+ xas_delete_node(xas);
+}
+
/**
* xas_free_nodes() - Free this node and all nodes that it references
* @xas: Array operation state.
@@ -1540,6 +1564,82 @@ static void xas_set_range(struct xa_state *xas, unsigned long first,
}
/**
+ * xas_export_node() - remove and return a node from an XArray
+ * @xas: XArray operation state
+ *
+ * The range covered by @xas must be aligned to and cover a single node
+ * at any level of the tree.
+ *
+ * Return: On success, returns the removed node. If the range is invalid,
+ * returns %NULL and sets -EINVAL in @xas. Otherwise returns %NULL if the
+ * node does not exist.
+ */
+struct xa_node *xas_export_node(struct xa_state *xas)
+{
+ struct xa_node *node;
+
+ if (!xas->xa_shift || xas->xa_sibs) {
+ xas_set_err(xas, -EINVAL);
+ return NULL;
+ }
+
+ xas->xa_shift -= XA_CHUNK_SHIFT;
+
+ if (!xas_find(xas, xas->xa_index))
+ return NULL;
+ node = xas->xa_node;
+ xas_unlink_node(xas);
+ node->parent = NULL;
+
+ return node;
+}
+
+/**
+ * xas_import_node() - add a node to an XArray
+ * @xas: XArray operation state
+ * @node: The node to add
+ *
+ * The range covered by @xas must be aligned to and cover a single node
+ * at any level of the tree. No nodes should already exist within the
+ * range.
+ * Sets an error in @xas if the range is invalid or xas_create() fails
+ */
+void xas_import_node(struct xa_state *xas, struct xa_node *node)
+{
+ struct xa_node *parent = NULL;
+ void __rcu **slot = &xas->xa->xa_head;
+ int count = 0;
+
+ if (!xas->xa_shift || xas->xa_sibs) {
+ xas_set_err(xas, -EINVAL);
+ return;
+ }
+
+ if (xas->xa_index || xa_head_locked(xas->xa)) {
+ xas_set_order(xas, xas->xa_index, node->shift + XA_CHUNK_SHIFT);
+ xas_create(xas, true);
+
+ if (xas_invalid(xas))
+ return;
+
+ parent = xas->xa_node;
+ }
+
+ if (parent) {
+ slot = &parent->slots[xas->xa_offset];
+ node->offset = xas->xa_offset;
+ count++;
+ }
+
+ RCU_INIT_POINTER(node->parent, parent);
+ node->array = xas->xa;
+
+ rcu_assign_pointer(*slot, xa_mk_node(node));
+
+ update_node(xas, parent, count, 0);
+}
+
+/**
* xa_store_range() - Store this entry at a range of indices in the XArray.
* @xa: XArray.
* @first: First index to affect.
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 42/43] shmem: reduce time holding xa_lock when inserting pages
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (40 preceding siblings ...)
2020-05-07 0:42 ` [RFC 41/43] XArray: add xas_export_node() and xas_import_node() Anthony Yznaga
@ 2020-05-07 0:42 ` Anthony Yznaga
2020-05-07 0:42 ` [RFC 43/43] PKRAM: improve index alignment of pkram_link entries Anthony Yznaga
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:42 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
Rather than adding one page at a time to the page cache and taking the
page cache xarray lock each time, where possible add pages in bulk by
first populating an xarray node outside of the page cache before taking
the lock to insert it.
When a group of pages to be inserted will fill an xarray node, add them
to a local xarray, export the xarray node, and then take the lock on the
page cache xarray and insert the node.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
mm/shmem.c | 145 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 138 insertions(+), 7 deletions(-)
diff --git a/mm/shmem.c b/mm/shmem.c
index f621d863e362..9d3c4e1f2dc1 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -732,17 +732,130 @@ static void shmem_delete_from_page_cache(struct page *page, void *radswap)
BUG_ON(error);
}
+static int shmem_add_aligned_to_page_cache(struct page *pages[], int npages,
+ struct address_space *mapping,
+ pgoff_t index, gfp_t gfp, int order)
+{
+ int xa_shift = order + XA_CHUNK_SHIFT - (order % XA_CHUNK_SHIFT);
+ XA_STATE_ORDER(xas, &mapping->i_pages, index, xa_shift);
+ struct xa_state *xas_ptr = &xas;
+ struct xarray xa_tmp;
+ /*
+ * Specify order so xas_create_range() only needs to be called once
+ * to allocate the entire range. This guarantees that xas_store()
+ * will not fail due to lack of memory.
+ * Specify index == 0 so the minimum necessary nodes are allocated.
+ */
+ XA_STATE_ORDER(xas_tmp, &xa_tmp, 0, xa_shift);
+ unsigned long nr = 1UL << order;
+ struct xa_node *node;
+ int i;
+
+ if (npages * nr != 1 << xa_shift) {
+ WARN_ONCE(1, "npages (%d) not aligned to xa_shift\n", npages);
+ return -EINVAL;
+ }
+ if (!IS_ALIGNED(index, 1 << xa_shift)) {
+ WARN_ONCE(1, "index (%lu) not aligned to xa_shift\n", index);
+ return -EINVAL;
+ }
+
+ for (i = 0; i < npages; i++) {
+ VM_BUG_ON_PAGE(PageTail(pages[i]), pages[i]);
+ VM_BUG_ON_PAGE(!PageLocked(pages[i]), pages[i]);
+ VM_BUG_ON_PAGE(!PageSwapBacked(pages[i]), pages[i]);
+
+ page_ref_add(pages[i], nr);
+ pages[i]->mapping = mapping;
+ pages[i]->index = index + (i * nr);
+ }
+
+ xa_init(&xa_tmp);
+ do {
+ xas_lock(&xas_tmp);
+ xas_create_range(&xas_tmp);
+ if (xas_error(&xas_tmp))
+ goto unlock;
+ for (i = 0; i < npages; i++) {
+ int j = 0;
+next:
+ xas_store(&xas_tmp, pages[i]);
+ if (++j < nr) {
+ xas_next(&xas_tmp);
+ goto next;
+ }
+ if (i < npages - 1)
+ xas_next(&xas_tmp);
+ }
+ xas_set_order(&xas_tmp, 0, xa_shift);
+ node = xas_export_node(&xas_tmp);
+unlock:
+ xas_unlock(&xas_tmp);
+ } while (xas_nomem(&xas_tmp, gfp));
+
+ if (xas_error(&xas_tmp)) {
+ xas_ptr = &xas_tmp;
+ goto error;
+ }
+
+ do {
+ xas_lock_irq(&xas);
+ xas_import_node(&xas, node);
+ if (xas_error(&xas))
+ goto unlock1;
+ mapping->nrpages += nr * npages;
+ xas_unlock(&xas);
+ for (i = 0; i < npages; i++) {
+ __mod_node_page_state(page_pgdat(pages[i]), NR_FILE_PAGES, nr);
+ __mod_node_page_state(page_pgdat(pages[i]), NR_SHMEM, nr);
+ if (PageTransHuge(pages[i])) {
+ count_vm_event(THP_FILE_ALLOC);
+ __inc_node_page_state(pages[i], NR_SHMEM_THPS);
+ }
+ }
+ local_irq_enable();
+ break;
+unlock1:
+ xas_unlock_irq(&xas);
+ } while (xas_nomem(&xas, gfp));
+
+ if (!xas_error(&xas))
+ return 0;
+
+error:
+ for (i = 0; i < npages; i++) {
+ pages[i]->mapping = NULL;
+ page_ref_sub(pages[i], nr);
+ }
+ return xas_error(xas_ptr);
+}
+
static int shmem_add_pages_to_cache(struct page *pages[], int npages,
struct address_space *mapping,
pgoff_t start, gfp_t gfp)
{
pgoff_t index = start;
int err = 0;
- int i;
+ int i, j;
i = 0;
while (i < npages) {
if (PageTransHuge(pages[i])) {
+ if (IS_ALIGNED(index, 4096) && i+8 <= npages) {
+ for (j = 1; j < 8; j++) {
+ if (!PageTransHuge(pages[i+j]))
+ break;
+ }
+ if (j == 8) {
+ err = shmem_add_aligned_to_page_cache(&pages[i], 8, mapping, index, gfp, HPAGE_PMD_ORDER);
+ if (err)
+ goto done;
+ index += HPAGE_PMD_NR * 8;
+ i += 8;
+ continue;
+ }
+ }
+
err = shmem_add_to_page_cache_fast(pages[i], mapping, index, gfp);
if (err)
break;
@@ -751,13 +864,31 @@ static int shmem_add_pages_to_cache(struct page *pages[], int npages,
continue;
}
- err = shmem_add_to_page_cache_fast(pages[i], mapping, index, gfp);
- if (err)
- break;
- index++;
- i++;
- }
+ for (j = 1; i + j < npages; j++) {
+ if (PageTransHuge(pages[i + j]))
+ break;
+ }
+
+ while (j > 0) {
+ if (IS_ALIGNED(index, 64) && j >= 64) {
+ err = shmem_add_aligned_to_page_cache(&pages[i], 64, mapping, index, gfp, 0);
+ if (err)
+ goto done;
+ index += 64;
+ i += 64;
+ j -= 64;
+ continue;
+ }
+ err = shmem_add_to_page_cache_fast(pages[i], mapping, index, gfp);
+ if (err)
+ goto done;
+ index++;
+ i++;
+ j--;
+ }
+ }
+done:
return err;
}
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread* [RFC 43/43] PKRAM: improve index alignment of pkram_link entries
2020-05-07 0:41 [RFC 00/43] PKRAM: Preserved-over-Kexec RAM Anthony Yznaga
` (41 preceding siblings ...)
2020-05-07 0:42 ` [RFC 42/43] shmem: reduce time holding xa_lock when inserting pages Anthony Yznaga
@ 2020-05-07 0:42 ` Anthony Yznaga
42 siblings, 0 replies; 50+ messages in thread
From: Anthony Yznaga @ 2020-05-07 0:42 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: willy, corbet, tglx, mingo, bp, x86, hpa, dave.hansen, luto,
peterz, rppt, akpm, hughd, ebiederm, masahiroy, ardb,
ndesaulniers, dima, daniel.kiper, nivedita, rafael.j.wysocki,
dan.j.williams, zhenzhong.duan, jroedel, bhe, guro,
Thomas.Lendacky, andriy.shevchenko, keescook, hannes, minchan,
mhocko, ying.huang, yang.shi, gustavo, ziqian.lzq, vdavydov.dev,
jason.zeng, kevin.tian, zhiyuan.lv, lei.l.li, paul.c.lai,
ashok.raj, linux-fsdevel, linux-doc, kexec
To take advantage of optimizations when adding pages to the page cache
via shmem_insert_pages(), improve the likelihood that the pages array
passed to shmem_insert_pages() starts on an aligned index. Do this
when preserving pages by starting a new pkram_link page when the current
page is aligned and the next aligned page will not fit on the pkram_link
page.
Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com>
---
mm/pkram.c | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/mm/pkram.c b/mm/pkram.c
index ef092aa5ce7a..416c3ca4411b 100644
--- a/mm/pkram.c
+++ b/mm/pkram.c
@@ -913,11 +913,21 @@ static int __pkram_save_page(struct pkram_stream *ps,
{
struct pkram_link *link = ps->link;
struct pkram_obj *obj = ps->obj;
+ int order, align, align_cnt;
pkram_entry_t p;
- int order;
+
+ if (PageTransHuge(page)) {
+ align = 1 << (HPAGE_PMD_ORDER + XA_CHUNK_SHIFT - (HPAGE_PMD_ORDER % XA_CHUNK_SHIFT));
+ align_cnt = align >> HPAGE_PMD_ORDER;
+ } else {
+ align = XA_CHUNK_SIZE;
+ align_cnt = XA_CHUNK_SIZE;
+ }
if (!link || ps->entry_idx >= PKRAM_LINK_ENTRIES_MAX ||
- index != ps->next_index) {
+ index != ps->next_index ||
+ (IS_ALIGNED(index, align) &&
+ (ps->entry_idx + align_cnt > PKRAM_LINK_ENTRIES_MAX))) {
struct page *link_page;
link_page = pkram_alloc_page((ps->gfp_mask & GFP_RECLAIM_MASK) |
--
2.13.3
^ permalink raw reply related [flat|nested] 50+ messages in thread