[PATCH 0/23 v2] mm: Speedup page cache truncation

[PATCH 0/23 v2] mm: Speedup page cache truncation

[Jan Kara]

Hello,

this is a second version of my patches to avoid clearing marks from xas_store()
and thus fix regression in page cache truncation.

Changes since v1
- rebased on 5.7-rc2
- drop xas_for_each_marked() fix as it was already merged
- reworked the whole series based on Matthew's feedback - we now create new
  function xas_store_noinit() and use it instead of changing xas_store()
  behavior. Note that for xas_store_range() and __xa_cmpxchg() I didn't bother
  to change names although they stop clearing marks as well. This is because
  there are only very few callers so it's easy to verify them, also chances of
  a clash with other patch introducing new callers are very small.

Original motivation:

Conversion of page cache to xarray (commit 69b6c1319b6 "mm: Convert truncate to
XArray" in particular) has regressed performance of page cache truncation
by about 10% (see my original report here [1]). This patch series aims at
improving the truncation to get some of that regression back.

The first patch fixes a long standing bug with xas_for_each_marked() that I've
uncovered when debugging my patches. The remaining patches then work towards
the ability to stop clearing marks in xas_store() which improves truncation
performance by about 6%.

The patches have passed radix_tree tests in tools/testing and also fstests runs
for ext4 & xfs.

								Honza

[1] https://lore.kernel.org/linux-mm/20190226165628.GB24711@quack2.suse.cz

[PATCH 01/23] xarray: Remove stale comment

[Jan Kara]

Since commit 7e934cf5ace1 "xarray: Fix early termination of
xas_for_each_marked" the comment is no longer relevant since
xas_for_each_marked() can cope with marked NULL entries. Remove the
comment.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 lib/xarray.c | 7 -------
 1 file changed, 7 deletions(-)

diff --git a/lib/xarray.c b/lib/xarray.c
index e9e641d3c0c3..dae68dd13a02 100644
--- a/lib/xarray.c
+++ b/lib/xarray.c
@@ -803,13 +803,6 @@ void *xas_store(struct xa_state *xas, void *entry)
 		xas_init_marks(xas);
 
 	for (;;) {
-		/*
-		 * Must clear the marks before setting the entry to NULL,
-		 * otherwise xas_for_each_marked may find a NULL entry and
-		 * stop early.  rcu_assign_pointer contains a release barrier
-		 * so the mark clearing will appear to happen before the
-		 * entry is set to NULL.
-		 */
 		rcu_assign_pointer(*slot, entry);
 		if (xa_is_node(next) && (!node || node->shift))
 			xas_free_nodes(xas, xa_to_node(next));
-- 
2.16.4

[PATCH 02/23] xarray: Provide xas_erase() and xas_store_noinit() helpers

[Jan Kara]

Currently xas_store() clears marks when stored value is NULL. This is
somewhat counter-intuitive and also causes measurable performance impact
when mark clearing is not needed (e.g. because marks are already clear).
So provide xas_erase() helper (similarly to existing xa_erase()) which
stores NULL at given index and also takes care of clearing marks. We
also introduce xas_store_noinit() helper that works like xas_store()
except that it does not initialize marks (and thus has better
performance).  In the following patches, all callers of xas_store() will
be converted either to xas_erase() or xas_store_noinit().

Signed-off-by: Jan Kara <jack@suse.cz>
---
 include/linux/xarray.h |  2 ++
 lib/xarray.c           | 59 +++++++++++++++++++++++++++++++++++++-------------
 2 files changed, 46 insertions(+), 15 deletions(-)

diff --git a/include/linux/xarray.h b/include/linux/xarray.h
index 14c893433139..06acef49ec95 100644
--- a/include/linux/xarray.h
+++ b/include/linux/xarray.h
@@ -1491,6 +1491,8 @@ static inline bool xas_retry(struct xa_state *xas, const void *entry)
 
 void *xas_load(struct xa_state *);
 void *xas_store(struct xa_state *, void *entry);
+void *xas_store_noinit(struct xa_state *, void *entry);
+void *xas_erase(struct xa_state *);
 void *xas_find(struct xa_state *, unsigned long max);
 void *xas_find_conflict(struct xa_state *);
 
diff --git a/lib/xarray.c b/lib/xarray.c
index dae68dd13a02..ed98fc152b17 100644
--- a/lib/xarray.c
+++ b/lib/xarray.c
@@ -753,20 +753,7 @@ static void update_node(struct xa_state *xas, struct xa_node *node,
 		xas_delete_node(xas);
 }
 
-/**
- * xas_store() - Store this entry in the XArray.
- * @xas: XArray operation state.
- * @entry: New entry.
- *
- * If @xas is operating on a multi-index entry, the entry returned by this
- * function is essentially meaningless (it may be an internal entry or it
- * may be %NULL, even if there are non-NULL entries at some of the indices
- * covered by the range).  This is not a problem for any current users,
- * and can be changed if needed.
- *
- * Return: The old entry at this index.
- */
-void *xas_store(struct xa_state *xas, void *entry)
+static void *__xas_store(struct xa_state *xas, void *entry, bool init_marks)
 {
 	struct xa_node *node;
 	void __rcu **slot = &xas->xa->xa_head;
@@ -799,7 +786,7 @@ void *xas_store(struct xa_state *xas, void *entry)
 		if (xas->xa_sibs)
 			xas_squash_marks(xas);
 	}
-	if (!entry)
+	if (init_marks)
 		xas_init_marks(xas);
 
 	for (;;) {
@@ -831,8 +818,33 @@ void *xas_store(struct xa_state *xas, void *entry)
 	update_node(xas, node, count, values);
 	return first;
 }
+
+/**
+ * xas_store() - Store this entry in the XArray.
+ * @xas: XArray operation state.
+ * @entry: New entry.
+ *
+ * If @xas is operating on a multi-index entry, the entry returned by this
+ * function is essentially meaningless (it may be an internal entry or it
+ * may be %NULL, even if there are non-NULL entries at some of the indices
+ * covered by the range).  This is not a problem for any current users,
+ * and can be changed if needed.
+ *
+ * Return: The old entry at this index.
+ */
+void *xas_store(struct xa_state *xas, void *entry)
+{
+	return __xas_store(xas, entry, true);
+}
 EXPORT_SYMBOL_GPL(xas_store);
 
+/* This is like xas_store() but does not initialize marks on stored entry */
+void *xas_store_noinit(struct xa_state *xas, void *entry)
+{
+	return __xas_store(xas, entry, false);
+}
+EXPORT_SYMBOL_GPL(xas_store_noinit);
+
 /**
  * xas_get_mark() - Returns the state of this mark.
  * @xas: XArray operation state.
@@ -1314,6 +1326,23 @@ static void *xas_result(struct xa_state *xas, void *curr)
 	return curr;
 }
 
+/**
+ * xas_erase() - Erase this entry from the XArray
+ * @xas: XArray operation state.
+ *
+ * After this function returns, loading from @index will return %NULL. The
+ * function also clears all marks associated with the @index.  If the index is
+ * part of a multi-index entry, all indices will be erased and none of the
+ * entries will be part of a multi-index entry.
+ *
+ * Return: The entry which used to be at this index.
+ */
+void *xas_erase(struct xa_state *xas)
+{
+	return __xas_store(xas, NULL, true);
+}
+EXPORT_SYMBOL(xas_erase);
+
 /**
  * __xa_erase() - Erase this entry from the XArray while locked.
  * @xa: XArray.
-- 
2.16.4

[PATCH 03/23] xarray: Use xas_store_noinit() in __xa_store, __xa_insert, __xa_alloc

[Jan Kara]

These three functions never store NULL in the array so xas_store() is
equivalent to xas_store_noinit(). Replace it.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 lib/xarray.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/lib/xarray.c b/lib/xarray.c
index ed98fc152b17..2eb634e8bf15 100644
--- a/lib/xarray.c
+++ b/lib/xarray.c
@@ -1079,7 +1079,7 @@ EXPORT_SYMBOL_GPL(__xas_next);
  *
  * If no entry is found and the array is smaller than @max, the iterator
  * is set to the smallest index not yet in the array.  This allows @xas
- * to be immediately passed to xas_store().
+ * to be immediately passed to xas_store_noinit().
  *
  * Return: The entry, if found, otherwise %NULL.
  */
@@ -1145,7 +1145,7 @@ EXPORT_SYMBOL_GPL(xas_find);
  * If no marked entry is found and the array is smaller than @max, @xas is
  * set to the bounds state and xas->xa_index is set to the smallest index
  * not yet in the array.  This allows @xas to be immediately passed to
- * xas_store().
+ * xas_store_noinit().
  *
  * If no entry is found before @max is reached, @xas is set to the restart
  * state.
@@ -1412,7 +1412,7 @@ void *__xa_store(struct xarray *xa, unsigned long index, void *entry, gfp_t gfp)
 		entry = XA_ZERO_ENTRY;
 
 	do {
-		curr = xas_store(&xas, entry);
+		curr = xas_store_noinit(&xas, entry);
 		if (xa_track_free(xa))
 			xas_clear_mark(&xas, XA_FREE_MARK);
 	} while (__xas_nomem(&xas, gfp));
@@ -1517,7 +1517,7 @@ int __xa_insert(struct xarray *xa, unsigned long index, void *entry, gfp_t gfp)
 	do {
 		curr = xas_load(&xas);
 		if (!curr) {
-			xas_store(&xas, entry);
+			xas_store_noinit(&xas, entry);
 			if (xa_track_free(xa))
 				xas_clear_mark(&xas, XA_FREE_MARK);
 		} else {
@@ -1653,7 +1653,7 @@ int __xa_alloc(struct xarray *xa, u32 *id, void *entry,
 			xas_set_err(&xas, -EBUSY);
 		else
 			*id = xas.xa_index;
-		xas_store(&xas, entry);
+		xas_store_noinit(&xas, entry);
 		xas_clear_mark(&xas, XA_FREE_MARK);
 	} while (__xas_nomem(&xas, gfp));
 
-- 
2.16.4

[PATCH 04/23] xarray: Switch xa_store_range() to use xas_store_noinit()

[Jan Kara]

There is only a single user of xa_store_range() and that is
mm/memremap.c which does not use marks at all. Just switch
xa_store_range() to use xas_store_noinit() to avoid implicit (and
unnecessary for all users) initialization of xarray marks.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 lib/xarray.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/xarray.c b/lib/xarray.c
index 2eb634e8bf15..49fafcee1c8e 100644
--- a/lib/xarray.c
+++ b/lib/xarray.c
@@ -1602,7 +1602,7 @@ void *xa_store_range(struct xarray *xa, unsigned long first,
 		}
 		do {
 			xas_set_range(&xas, first, last);
-			xas_store(&xas, entry);
+			xas_store_noinit(&xas, entry);
 			if (xas_error(&xas))
 				goto unlock;
 			first += xas_size(&xas);
-- 
2.16.4

[PATCH 05/23] xarray: Use xas_erase() in __xa_erase()

[Jan Kara]

In this case we want to erase element from the array. Use xas_erase()
for it.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 lib/xarray.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/xarray.c b/lib/xarray.c
index 49fafcee1c8e..96be029412b2 100644
--- a/lib/xarray.c
+++ b/lib/xarray.c
@@ -1358,7 +1358,7 @@ EXPORT_SYMBOL(xas_erase);
 void *__xa_erase(struct xarray *xa, unsigned long index)
 {
 	XA_STATE(xas, xa, index);
-	return xas_result(&xas, xas_store(&xas, NULL));
+	return xas_result(&xas, xas_erase(&xas));
 }
 EXPORT_SYMBOL(__xa_erase);
 
-- 
2.16.4

[PATCH 06/23] xarray: Explicitely set XA_FREE_MARK in __xa_cmpxchg()

[Jan Kara]

__xa_cmpxchg() relies on xas_store() to set XA_FREE_MARK when storing
NULL into xarray that has free tracking enabled. Make the setting of
XA_FREE_MARK explicit similarly as its clearing currently is.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 lib/xarray.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/lib/xarray.c b/lib/xarray.c
index 96be029412b2..a372c59e3914 100644
--- a/lib/xarray.c
+++ b/lib/xarray.c
@@ -1479,8 +1479,12 @@ void *__xa_cmpxchg(struct xarray *xa, unsigned long index,
 		curr = xas_load(&xas);
 		if (curr == old) {
 			xas_store(&xas, entry);
-			if (xa_track_free(xa) && entry && !curr)
-				xas_clear_mark(&xas, XA_FREE_MARK);
+			if (xa_track_free(xa)) {
+				if (entry && !curr)
+					xas_clear_mark(&xas, XA_FREE_MARK);
+				else if (!entry && curr)
+					xas_set_mark(&xas, XA_FREE_MARK);
+			}
 		}
 	} while (__xas_nomem(&xas, gfp));
 
-- 
2.16.4

[PATCH 07/23] xarray: Switch __xa_cmpxchg() to use xas_store_noinit()

[Jan Kara]

Currently there are four places that end up calling into __xa_cmpxchg().
Two in drivers/infiniband/hw/mlx5/odp.c, one in fs/erofs/utils.c and one
in mm/shmem.c. The xarray in the first three places do not contain any
marks ever, the fourth place originally used radix_tree_delete_item()
which didn't touch marks either. So we are safe to switch __xa_cmpxchg()
to use xas_store_noinit(). Also document that __xa_cmpxchg() now does
not touch marks at all - i.e., it has ordinary store semantics without
specialcasing NULL value. If someone ever needs xa_cmpxchg() equivalent
that would really cause mark clearing on storing NULL, we can create
xa_erase_item() function.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 lib/xarray.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lib/xarray.c b/lib/xarray.c
index a372c59e3914..d87045d120ad 100644
--- a/lib/xarray.c
+++ b/lib/xarray.c
@@ -1460,7 +1460,8 @@ EXPORT_SYMBOL(xa_store);
  *
  * You must already be holding the xa_lock when calling this function.
  * It will drop the lock if needed to allocate memory, and then reacquire
- * it afterwards.
+ * it afterwards. The call does not change any xarray marks except for
+ * XA_FREE_MARK if free tracking is enabled.
  *
  * Context: Any context.  Expects xa_lock to be held on entry.  May
  * release and reacquire xa_lock if @gfp flags permit.
@@ -1478,7 +1479,7 @@ void *__xa_cmpxchg(struct xarray *xa, unsigned long index,
 	do {
 		curr = xas_load(&xas);
 		if (curr == old) {
-			xas_store(&xas, entry);
+			xas_store_noinit(&xas, entry);
 			if (xa_track_free(xa)) {
 				if (entry && !curr)
 					xas_clear_mark(&xas, XA_FREE_MARK);
-- 
2.16.4

[PATCH 08/23] dax: Use xas_erase() in __dax_invalidate_entry()

[Jan Kara]

When truncating DAX entry, we need to clear all the outstanding marks
for the entry. Use dax_erase() instead of dax_store().

Signed-off-by: Jan Kara <jack@suse.cz>
---
 fs/dax.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/dax.c b/fs/dax.c
index 11b16729b86f..f8358928549c 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -643,7 +643,7 @@ static int __dax_invalidate_entry(struct address_space *mapping,
 	     xas_get_mark(&xas, PAGECACHE_TAG_TOWRITE)))
 		goto out;
 	dax_disassociate_entry(entry, mapping, trunc);
-	xas_store(&xas, NULL);
+	xas_erase(&xas);
 	mapping->nrexceptional--;
 	ret = 1;
 out:
-- 
2.16.4

[PATCH 09/23] dax: Use dax_store_noinit() in grab_mapping_entry()

[Jan Kara]

Although we store NULL here, there aren't any marks set on the entry
so use xas_store_noinit() to save some time when trying to reinitialize
marks. Furthermore we actually overwrite the entry shortly afterwards
with new PTE entry anyway.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 fs/dax.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/fs/dax.c b/fs/dax.c
index f8358928549c..03c6ca693f3c 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -523,7 +523,11 @@ static void *grab_mapping_entry(struct xa_state *xas,
 		}
 
 		dax_disassociate_entry(entry, mapping, false);
-		xas_store(xas, NULL);	/* undo the PMD join */
+		/*
+		 * Remove PMD entry. There should be no marks set for the entry
+ 		 * so don't bother reinitializing them with xas_erase().
+		 */
+		xas_store_noinit(xas, NULL);
 		dax_wake_entry(xas, entry, true);
 		mapping->nrexceptional--;
 		entry = NULL;
-- 
2.16.4

[PATCH 10/23] dax: Convert xas_store() to xas_store_noinit()

[Jan Kara]

All remaining users of xas_store() store non-NULL entries so xas_store()
and xas_store_noinit() are equivalent. Replace xas_store() with
xas_store_noinit().

Signed-off-by: Jan Kara <jack@suse.cz>
---
 fs/dax.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/fs/dax.c b/fs/dax.c
index 03c6ca693f3c..1c905830ee10 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -283,7 +283,7 @@ static void dax_unlock_entry(struct xa_state *xas, void *entry)
 	BUG_ON(dax_is_locked(entry));
 	xas_reset(xas);
 	xas_lock_irq(xas);
-	old = xas_store(xas, entry);
+	old = xas_store_noinit(xas, entry);
 	xas_unlock_irq(xas);
 	BUG_ON(!dax_is_locked(old));
 	dax_wake_entry(xas, entry, false);
@@ -295,7 +295,7 @@ static void dax_unlock_entry(struct xa_state *xas, void *entry)
 static void *dax_lock_entry(struct xa_state *xas, void *entry)
 {
 	unsigned long v = xa_to_value(entry);
-	return xas_store(xas, xa_mk_value(v | DAX_LOCKED));
+	return xas_store_noinit(xas, xa_mk_value(v | DAX_LOCKED));
 }
 
 static unsigned long dax_entry_size(void *entry)
@@ -923,7 +923,7 @@ static int dax_writeback_one(struct xa_state *xas, struct dax_device *dax_dev,
 	 */
 	xas_reset(xas);
 	xas_lock_irq(xas);
-	xas_store(xas, entry);
+	xas_store_noinit(xas, entry);
 	xas_clear_mark(xas, PAGECACHE_TAG_DIRTY);
 	dax_wake_entry(xas, entry, false);
 
-- 
2.16.4

[PATCH 11/23] mm: Use xas_erase() in page_cache_delete_batch()

[Jan Kara]

We need to clear marks when removing a page from xarray since there
could be DIRTY or TOWRITE tags left for the page. Use xas_erase() to
explicitely request mark clearing.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 mm/filemap.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index 23a051a7ef0f..48c488b505ad 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -333,7 +333,7 @@ static void page_cache_delete_batch(struct address_space *mapping,
 		 */
 		if (page->index + compound_nr(page) - 1 == xas.xa_index)
 			i++;
-		xas_store(&xas, NULL);
+		xas_erase(&xas);
 		total_pages++;
 	}
 	mapping->nrpages -= total_pages;
-- 
2.16.4

[PATCH 12/23] mm: Use xas_erase() in collapse_file()

[Jan Kara]

When undoing failed collapse of ordinary pages into a huge page, use
xas_erase() to explicitly clear any xarray marks that may have been
added to entries.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 mm/khugepaged.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 99d77ffb79c2..8da820c02de7 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1819,7 +1819,7 @@ static void collapse_file(struct mm_struct *mm,
 					break;
 				nr_none--;
 				/* Put holes back where they were */
-				xas_store(&xas, NULL);
+				xas_erase(&xas);
 				continue;
 			}
 
-- 
2.16.4

[PATCH 13/23] mm: Use xas_store_noinit() when storing non-NULL

[Jan Kara]

When we store value different from NULL, xas_store_noinit() is
equivalent to xas_store(). Transition these places to
xas_store_noinit().

Signed-off-by: Jan Kara <jack@suse.cz>
---
 mm/filemap.c    | 6 +++---
 mm/khugepaged.c | 6 +++---
 mm/migrate.c    | 6 +++---
 mm/shmem.c      | 4 ++--
 mm/swap_state.c | 2 +-
 5 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/mm/filemap.c b/mm/filemap.c
index 48c488b505ad..4fb515d8c242 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -134,7 +134,7 @@ static void page_cache_delete(struct address_space *mapping,
 	VM_BUG_ON_PAGE(PageTail(page), page);
 	VM_BUG_ON_PAGE(nr != 1 && shadow, page);
 
-	xas_store(&xas, shadow);
+	xas_store_noinit(&xas, shadow);
 	xas_init_marks(&xas);
 
 	page->mapping = NULL;
@@ -803,7 +803,7 @@ int replace_page_cache_page(struct page *old, struct page *new, gfp_t gfp_mask)
 	new->index = offset;
 
 	xas_lock_irqsave(&xas, flags);
-	xas_store(&xas, new);
+	xas_store_noinit(&xas, new);
 
 	old->mapping = NULL;
 	/* hugetlb pages do not participate in page cache accounting. */
@@ -856,7 +856,7 @@ static int __add_to_page_cache_locked(struct page *page,
 		old = xas_load(&xas);
 		if (old && !xa_is_value(old))
 			xas_set_err(&xas, -EEXIST);
-		xas_store(&xas, page);
+		xas_store_noinit(&xas, page);
 		if (xas_error(&xas))
 			goto unlock;
 
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 8da820c02de7..79e5f0d12517 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1588,7 +1588,7 @@ static void collapse_file(struct mm_struct *mm,
 					result = SCAN_FAIL;
 					goto xa_locked;
 				}
-				xas_store(&xas, new_page);
+				xas_store_noinit(&xas, new_page);
 				nr_none++;
 				continue;
 			}
@@ -1724,7 +1724,7 @@ static void collapse_file(struct mm_struct *mm,
 		list_add_tail(&page->lru, &pagelist);
 
 		/* Finally, replace with the new page. */
-		xas_store(&xas, new_page);
+		xas_store_noinit(&xas, new_page);
 		continue;
 out_unlock:
 		unlock_page(page);
@@ -1828,7 +1828,7 @@ static void collapse_file(struct mm_struct *mm,
 			/* Unfreeze the page. */
 			list_del(&page->lru);
 			page_ref_unfreeze(page, 2);
-			xas_store(&xas, page);
+			xas_store_noinit(&xas, page);
 			xas_pause(&xas);
 			xas_unlock_irq(&xas);
 			unlock_page(page);
diff --git a/mm/migrate.c b/mm/migrate.c
index 7160c1556f79..1610b7336af8 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -459,13 +459,13 @@ int migrate_page_move_mapping(struct address_space *mapping,
 		SetPageDirty(newpage);
 	}
 
-	xas_store(&xas, newpage);
+	xas_store_noinit(&xas, newpage);
 	if (PageTransHuge(page)) {
 		int i;
 
 		for (i = 1; i < HPAGE_PMD_NR; i++) {
 			xas_next(&xas);
-			xas_store(&xas, newpage);
+			xas_store_noinit(&xas, newpage);
 		}
 	}
 
@@ -536,7 +536,7 @@ int migrate_huge_page_move_mapping(struct address_space *mapping,
 
 	get_page(newpage);
 
-	xas_store(&xas, newpage);
+	xas_store_noinit(&xas, newpage);
 
 	page_ref_unfreeze(page, expected_count - 1);
 
diff --git a/mm/shmem.c b/mm/shmem.c
index bd8840082c94..35a4abd9b417 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -361,7 +361,7 @@ static int shmem_replace_entry(struct address_space *mapping,
 	item = xas_load(&xas);
 	if (item != expected)
 		return -ENOENT;
-	xas_store(&xas, replacement);
+	xas_store_noinit(&xas, replacement);
 	return 0;
 }
 
@@ -631,7 +631,7 @@ static int shmem_add_to_page_cache(struct page *page,
 		if (xas_error(&xas))
 			goto unlock;
 next:
-		xas_store(&xas, page);
+		xas_store_noinit(&xas, page);
 		if (++i < nr) {
 			xas_next(&xas);
 			goto next;
diff --git a/mm/swap_state.c b/mm/swap_state.c
index ebed37bbf7a3..1afbf68f1724 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -133,7 +133,7 @@ int add_to_swap_cache(struct page *page, swp_entry_t entry, gfp_t gfp)
 		for (i = 0; i < nr; i++) {
 			VM_BUG_ON_PAGE(xas.xa_index != idx + i, page);
 			set_page_private(page + i, entry.val + i);
-			xas_store(&xas, page);
+			xas_store_noinit(&xas, page);
 			xas_next(&xas);
 		}
 		address_space->nrpages += nr;
-- 
2.16.4

[PATCH 14/23] workingset: Use xas_store_noinit() to clear shadow entry

[Jan Kara]

Shadow entries have no marks. Use xas_store_noinit() to clear the entry
so avoid unneeded initialization of the xarray marks. This provides a
nice boost to truncate numbers. Sample benchmark showing time to
truncate 128 files 1GB each on machine with 64GB of RAM (so about half
of entries are shadow entries):

         AVG      STDDEV
Vanilla  4.825s   0.036
Patched  4.516s   0.014

So we can see about 6% reduction in overall truncate time.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 mm/truncate.c   | 2 +-
 mm/workingset.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/mm/truncate.c b/mm/truncate.c
index dd9ebc1da356..baef636564cc 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -39,7 +39,7 @@ static inline void __clear_shadow_entry(struct address_space *mapping,
 	xas_set_update(&xas, workingset_update_node);
 	if (xas_load(&xas) != entry)
 		return;
-	xas_store(&xas, NULL);
+	xas_store_noinit(&xas, NULL);
 	mapping->nrexceptional--;
 }
 
diff --git a/mm/workingset.c b/mm/workingset.c
index 474186b76ced..6a492140057f 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -537,7 +537,7 @@ static enum lru_status shadow_lru_isolate(struct list_head *item,
 	 * We could store a shadow entry here which was the minimum of the
 	 * shadow entries we were tracking ...
 	 */
-	xas_store(&xas, NULL);
+	xas_store_noinit(&xas, NULL);
 	__inc_lruvec_slab_state(node, WORKINGSET_NODERECLAIM);
 
 out_invalid:
-- 
2.16.4

[PATCH 15/23] swap: Use xas_erase() when removing page from swap cache

[Jan Kara]

Use xas_erase() to explicitely clear xarray marks when removing swap
cache pages from the i_mapping xarray.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 mm/swap_state.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/swap_state.c b/mm/swap_state.c
index 1afbf68f1724..b5c8cbdcf8f0 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -167,7 +167,7 @@ void __delete_from_swap_cache(struct page *page, swp_entry_t entry)
 	VM_BUG_ON_PAGE(PageWriteback(page), page);
 
 	for (i = 0; i < nr; i++) {
-		void *entry = xas_store(&xas, NULL);
+		void *entry = xas_erase(&xas);
 		VM_BUG_ON_PAGE(entry != page, entry);
 		set_page_private(page + i, 0);
 		xas_next(&xas);
-- 
2.16.4

[PATCH 16/23] idr: Use xas_erase() in ida_destroy()

[Jan Kara]

Explicitely clear marks (and set XA_MARK_FREE) in ida_destroy() by
calling xas_erase() instead of relying on xas_store() on implicitely
doing this.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 lib/idr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/idr.c b/lib/idr.c
index c2cf2c52bbde..fd4877fef06d 100644
--- a/lib/idr.c
+++ b/lib/idr.c
@@ -543,7 +543,7 @@ void ida_destroy(struct ida *ida)
 	xas_for_each(&xas, bitmap, ULONG_MAX) {
 		if (!xa_is_value(bitmap))
 			kfree(bitmap);
-		xas_store(&xas, NULL);
+		xas_erase(&xas);
 	}
 	xas_unlock_irqrestore(&xas, flags);
 }
-- 
2.16.4

[PATCH 17/23] idr: Use xas_erase() in ida_free()

[Jan Kara]

Explicitely clear marks in ida_free() by calling xas_erase() instead of
relying on xas_store() on implicitely doing this.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 lib/idr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/idr.c b/lib/idr.c
index fd4877fef06d..4ee06bc7208a 100644
--- a/lib/idr.c
+++ b/lib/idr.c
@@ -511,7 +511,7 @@ void ida_free(struct ida *ida, unsigned int id)
 		if (bitmap_empty(bitmap->bitmap, IDA_BITMAP_BITS)) {
 			kfree(bitmap);
 delete:
-			xas_store(&xas, NULL);
+			xas_erase(&xas);
 		}
 	}
 	xas_unlock_irqrestore(&xas, flags);
-- 
2.16.4

[PATCH 18/23] idr: Convert xas_store() to xas_store_noinit()

[Jan Kara]

All remaining users of xas_store() store non-NULL entries so xas_store()
and xas_store_noinit() are equivalent. Replace xas_store() with
xas_store_noinit().

Signed-off-by: Jan Kara <jack@suse.cz>
---
 lib/idr.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/lib/idr.c b/lib/idr.c
index 4ee06bc7208a..afd171077901 100644
--- a/lib/idr.c
+++ b/lib/idr.c
@@ -408,7 +408,7 @@ int ida_alloc_range(struct ida *ida, unsigned int min, unsigned int max,
 				goto nospc;
 			if (bit < BITS_PER_XA_VALUE) {
 				tmp |= 1UL << bit;
-				xas_store(&xas, xa_mk_value(tmp));
+				xas_store_noinit(&xas, xa_mk_value(tmp));
 				goto out;
 			}
 		}
@@ -418,7 +418,7 @@ int ida_alloc_range(struct ida *ida, unsigned int min, unsigned int max,
 		if (!bitmap)
 			goto alloc;
 		bitmap->bitmap[0] = tmp;
-		xas_store(&xas, bitmap);
+		xas_store_noinit(&xas, bitmap);
 		if (xas_error(&xas)) {
 			bitmap->bitmap[0] = 0;
 			goto out;
@@ -446,7 +446,7 @@ int ida_alloc_range(struct ida *ida, unsigned int min, unsigned int max,
 				goto alloc;
 			__set_bit(bit, bitmap->bitmap);
 		}
-		xas_store(&xas, bitmap);
+		xas_store_noinit(&xas, bitmap);
 	}
 out:
 	xas_unlock_irqrestore(&xas, flags);
@@ -502,7 +502,7 @@ void ida_free(struct ida *ida, unsigned int id)
 		v &= ~(1UL << bit);
 		if (!v)
 			goto delete;
-		xas_store(&xas, xa_mk_value(v));
+		xas_store_noinit(&xas, xa_mk_value(v));
 	} else {
 		if (!test_bit(bit, bitmap->bitmap))
 			goto err;
-- 
2.16.4

[PATCH 19/23] testing: Use xas_erase() to remove entries from xarray

[Jan Kara]

Use xas_erase() instead of xas_store(... NULL) for removing entries from
xarray.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 lib/test_xarray.c               | 10 +++++-----
 tools/testing/radix-tree/test.c |  2 +-
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/lib/test_xarray.c b/lib/test_xarray.c
index d4f97925dbd8..434031fe8d54 100644
--- a/lib/test_xarray.c
+++ b/lib/test_xarray.c
@@ -318,7 +318,7 @@ static noinline void check_xa_shrink(struct xarray *xa)
 	XA_BUG_ON(xa, xas_load(&xas) != xa_mk_value(1));
 	node = xas.xa_node;
 	XA_BUG_ON(xa, xa_entry_locked(xa, node, 0) != xa_mk_value(0));
-	XA_BUG_ON(xa, xas_store(&xas, NULL) != xa_mk_value(1));
+	XA_BUG_ON(xa, xas_erase(&xas) != xa_mk_value(1));
 	XA_BUG_ON(xa, xa_load(xa, 1) != NULL);
 	XA_BUG_ON(xa, xas.xa_node != XAS_BOUNDS);
 	XA_BUG_ON(xa, xa_entry_locked(xa, node, 0) != XA_RETRY_ENTRY);
@@ -488,13 +488,13 @@ static noinline void check_xas_erase(struct xarray *xa)
 		} while (xas_nomem(&xas, GFP_KERNEL));
 
 		xas_lock(&xas);
-		xas_store(&xas, NULL);
+		xas_erase(&xas);
 
 		xas_set(&xas, 0);
 		j = i;
 		xas_for_each(&xas, entry, ULONG_MAX) {
 			XA_BUG_ON(xa, entry != xa_mk_index(j));
-			xas_store(&xas, NULL);
+			xas_erase(&xas);
 			j++;
 		}
 		xas_unlock(&xas);
@@ -537,7 +537,7 @@ static noinline void check_multi_store_2(struct xarray *xa, unsigned long index,
 	xas_lock(&xas);
 	XA_BUG_ON(xa, xas_store(&xas, xa_mk_value(1)) != xa_mk_value(0));
 	XA_BUG_ON(xa, xas.xa_index != index);
-	XA_BUG_ON(xa, xas_store(&xas, NULL) != xa_mk_value(1));
+	XA_BUG_ON(xa, xas_erase(&xas) != xa_mk_value(1));
 	xas_unlock(&xas);
 	XA_BUG_ON(xa, !xa_empty(xa));
 }
@@ -1582,7 +1582,7 @@ static noinline void shadow_remove(struct xarray *xa)
 		xas.xa_offset = node->offset;
 		xas.xa_shift = node->shift + XA_CHUNK_SHIFT;
 		xas_set_update(&xas, test_update_node);
-		xas_store(&xas, NULL);
+		xas_erase(&xas);
 	}
 	xa_unlock(xa);
 }
diff --git a/tools/testing/radix-tree/test.c b/tools/testing/radix-tree/test.c
index a15d0512e633..07dc2b4dc587 100644
--- a/tools/testing/radix-tree/test.c
+++ b/tools/testing/radix-tree/test.c
@@ -261,7 +261,7 @@ void item_kill_tree(struct xarray *xa)
 		if (!xa_is_value(entry)) {
 			item_free(entry, xas.xa_index);
 		}
-		xas_store(&xas, NULL);
+		xas_erase(&xas);
 	}
 
 	assert(xa_empty(xa));
-- 
2.16.4

[PATCH 20/23] testing: Use xas_store_noinit() for non-NULL entries

[Jan Kara]

When we store value different from NULL, xas_store_noinit() is
equivalent to xas_store(). Transition these places to
xas_store_noinit().

Signed-off-by: Jan Kara <jack@suse.cz>
---
 lib/test_xarray.c                          | 28 ++++++++++++++--------------
 tools/testing/radix-tree/iteration_check.c |  2 +-
 tools/testing/radix-tree/multiorder.c      |  2 +-
 3 files changed, 16 insertions(+), 16 deletions(-)

diff --git a/lib/test_xarray.c b/lib/test_xarray.c
index 434031fe8d54..2cf3ef5d5014 100644
--- a/lib/test_xarray.c
+++ b/lib/test_xarray.c
@@ -130,13 +130,13 @@ static noinline void check_xas_retry(struct xarray *xa)
 	/* Make sure we can iterate through retry entries */
 	xas_lock(&xas);
 	xas_set(&xas, 0);
-	xas_store(&xas, XA_RETRY_ENTRY);
+	xas_store_noinit(&xas, XA_RETRY_ENTRY);
 	xas_set(&xas, 1);
-	xas_store(&xas, XA_RETRY_ENTRY);
+	xas_store_noinit(&xas, XA_RETRY_ENTRY);
 
 	xas_set(&xas, 0);
 	xas_for_each(&xas, entry, ULONG_MAX) {
-		xas_store(&xas, xa_mk_index(xas.xa_index));
+		xas_store_noinit(&xas, xa_mk_index(xas.xa_index));
 	}
 	xas_unlock(&xas);
 
@@ -475,7 +475,7 @@ static noinline void check_xas_erase(struct xarray *xa)
 			xas_set(&xas, j);
 			do {
 				xas_lock(&xas);
-				xas_store(&xas, xa_mk_index(j));
+				xas_store_noinit(&xas, xa_mk_index(j));
 				xas_unlock(&xas);
 			} while (xas_nomem(&xas, GFP_KERNEL));
 		}
@@ -483,7 +483,7 @@ static noinline void check_xas_erase(struct xarray *xa)
 		xas_set(&xas, ULONG_MAX);
 		do {
 			xas_lock(&xas);
-			xas_store(&xas, xa_mk_value(0));
+			xas_store_noinit(&xas, xa_mk_value(0));
 			xas_unlock(&xas);
 		} while (xas_nomem(&xas, GFP_KERNEL));
 
@@ -517,7 +517,7 @@ static noinline void check_multi_store_1(struct xarray *xa, unsigned long index,
 	XA_BUG_ON(xa, xa_load(xa, min - 1) != NULL);
 
 	xas_lock(&xas);
-	XA_BUG_ON(xa, xas_store(&xas, xa_mk_index(min)) != xa_mk_index(index));
+	XA_BUG_ON(xa, xas_store_noinit(&xas, xa_mk_index(min)) != xa_mk_index(index));
 	xas_unlock(&xas);
 	XA_BUG_ON(xa, xa_load(xa, min) != xa_mk_index(min));
 	XA_BUG_ON(xa, xa_load(xa, max - 1) != xa_mk_index(min));
@@ -535,7 +535,7 @@ static noinline void check_multi_store_2(struct xarray *xa, unsigned long index,
 	xa_store_order(xa, index, order, xa_mk_value(0), GFP_KERNEL);
 
 	xas_lock(&xas);
-	XA_BUG_ON(xa, xas_store(&xas, xa_mk_value(1)) != xa_mk_value(0));
+	XA_BUG_ON(xa, xas_store_noinit(&xas, xa_mk_value(1)) != xa_mk_value(0));
 	XA_BUG_ON(xa, xas.xa_index != index);
 	XA_BUG_ON(xa, xas_erase(&xas) != xa_mk_value(1));
 	xas_unlock(&xas);
@@ -854,7 +854,7 @@ static noinline void __check_store_iter(struct xarray *xa, unsigned long start,
 		XA_BUG_ON(xa, entry > xa_mk_index(start + (1UL << order) - 1));
 		count++;
 	}
-	xas_store(&xas, xa_mk_index(start));
+	xas_store_noinit(&xas, xa_mk_index(start));
 	xas_unlock(&xas);
 	if (xas_nomem(&xas, GFP_KERNEL)) {
 		count = 0;
@@ -1365,7 +1365,7 @@ static noinline void xa_store_many_order(struct xarray *xa,
 		if (xas_error(&xas))
 			goto unlock;
 		for (i = 0; i < (1U << order); i++) {
-			XA_BUG_ON(xa, xas_store(&xas, xa_mk_index(index + i)));
+			XA_BUG_ON(xa, xas_store_noinit(&xas, xa_mk_index(index + i)));
 			xas_next(&xas);
 		}
 unlock:
@@ -1420,7 +1420,7 @@ static noinline void check_create_range_4(struct xarray *xa,
 		if (xas_error(&xas))
 			goto unlock;
 		for (i = 0; i < (1UL << order); i++) {
-			void *old = xas_store(&xas, xa_mk_index(base + i));
+			void *old = xas_store_noinit(&xas, xa_mk_index(base + i));
 			if (xas.xa_index == index)
 				XA_BUG_ON(xa, old != xa_mk_index(base + i));
 			else
@@ -1594,9 +1594,9 @@ static noinline void check_workingset(struct xarray *xa, unsigned long index)
 
 	do {
 		xas_lock(&xas);
-		xas_store(&xas, xa_mk_value(0));
+		xas_store_noinit(&xas, xa_mk_value(0));
 		xas_next(&xas);
-		xas_store(&xas, xa_mk_value(1));
+		xas_store_noinit(&xas, xa_mk_value(1));
 		xas_unlock(&xas);
 	} while (xas_nomem(&xas, GFP_KERNEL));
 
@@ -1604,10 +1604,10 @@ static noinline void check_workingset(struct xarray *xa, unsigned long index)
 
 	xas_lock(&xas);
 	xas_next(&xas);
-	xas_store(&xas, &xas);
+	xas_store_noinit(&xas, &xas);
 	XA_BUG_ON(xa, !list_empty(&shadow_nodes));
 
-	xas_store(&xas, xa_mk_value(2));
+	xas_store_noinit(&xas, xa_mk_value(2));
 	xas_unlock(&xas);
 	XA_BUG_ON(xa, list_empty(&shadow_nodes));
 
diff --git a/tools/testing/radix-tree/iteration_check.c b/tools/testing/radix-tree/iteration_check.c
index e9908bcb06dd..ce110e96757c 100644
--- a/tools/testing/radix-tree/iteration_check.c
+++ b/tools/testing/radix-tree/iteration_check.c
@@ -31,7 +31,7 @@ void my_item_insert(struct xarray *xa, unsigned long index)
 		item->order = order;
 		if (xas_find_conflict(&xas))
 			continue;
-		xas_store(&xas, item);
+		xas_store_noinit(&xas, item);
 		xas_set_mark(&xas, TAG);
 		break;
 	}
diff --git a/tools/testing/radix-tree/multiorder.c b/tools/testing/radix-tree/multiorder.c
index 9eae0fb5a67d..695a562d5ff1 100644
--- a/tools/testing/radix-tree/multiorder.c
+++ b/tools/testing/radix-tree/multiorder.c
@@ -20,7 +20,7 @@ static int item_insert_order(struct xarray *xa, unsigned long index,
 
 	do {
 		xas_lock(&xas);
-		xas_store(&xas, item);
+		xas_store_noinit(&xas, item);
 		xas_unlock(&xas);
 	} while (xas_nomem(&xas, GFP_KERNEL));
 
-- 
2.16.4

[PATCH 21/23] testing: Introduce xa_erase_order() and use it

[Jan Kara]

Introduce helper xa_erase_order() and call it from places that want to
erase entries from xarray instead of using xa_store_order(). This also
explicitely takes care of clearing possibly existing xarray marks
(although no current test seems to need it).

Signed-off-by: Jan Kara <jack@suse.cz>
---
 lib/test_xarray.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/lib/test_xarray.c b/lib/test_xarray.c
index 2cf3ef5d5014..fc16eac1cbb9 100644
--- a/lib/test_xarray.c
+++ b/lib/test_xarray.c
@@ -83,6 +83,16 @@ static void *xa_store_order(struct xarray *xa, unsigned long index,
 	return curr;
 }
 
+static void xa_erase_order(struct xarray *xa, unsigned long index,
+		unsigned order)
+{
+	XA_STATE_ORDER(xas, xa, index, order);
+
+	xas_lock(&xas);
+	xas_erase(&xas);
+	xas_unlock(&xas);
+}
+
 static noinline void check_xa_err(struct xarray *xa)
 {
 	XA_BUG_ON(xa, xa_err(xa_store_index(xa, 0, GFP_NOWAIT)) != 0);
@@ -608,13 +618,13 @@ static noinline void check_multi_store(struct xarray *xa)
 	rcu_read_unlock();
 
 	/* We can erase multiple values with a single store */
-	xa_store_order(xa, 0, BITS_PER_LONG - 1, NULL, GFP_KERNEL);
+	xa_erase_order(xa, 0, BITS_PER_LONG - 1);
 	XA_BUG_ON(xa, !xa_empty(xa));
 
 	/* Even when the first slot is empty but the others aren't */
 	xa_store_index(xa, 1, GFP_KERNEL);
 	xa_store_index(xa, 2, GFP_KERNEL);
-	xa_store_order(xa, 0, 2, NULL, GFP_KERNEL);
+	xa_erase_order(xa, 0, 2);
 	XA_BUG_ON(xa, !xa_empty(xa));
 
 	for (i = 0; i < max_order; i++) {
-- 
2.16.4

[PATCH 22/23] testing: Switch xa_store_order() to xas_store_noinit()

[Jan Kara]

xa_store_order() is currently only used for non-NULL entries. So
xas_store_noinit() is equivalent to xas_store() and we can safely use it
in xa_store_order.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 lib/test_xarray.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/test_xarray.c b/lib/test_xarray.c
index fc16eac1cbb9..3adc29819b0d 100644
--- a/lib/test_xarray.c
+++ b/lib/test_xarray.c
@@ -76,7 +76,7 @@ static void *xa_store_order(struct xarray *xa, unsigned long index,
 
 	do {
 		xas_lock(&xas);
-		curr = xas_store(&xas, entry);
+		curr = xas_store_noinit(&xas, entry);
 		xas_unlock(&xas);
 	} while (xas_nomem(&xas, gfp));
 
-- 
2.16.4

[PATCH 23/23] xarray: Remove xas_store()

[Jan Kara]

xas_store() now has no users as every call site has been transitioned
either to xas_erase() or to xas_store_noinit(). Remove xas_store() and
all remaining traces of it.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 Documentation/core-api/xarray.rst |  4 ++--
 include/linux/xarray.h            |  1 -
 lib/xarray.c                      | 14 ++++----------
 3 files changed, 6 insertions(+), 13 deletions(-)

diff --git a/Documentation/core-api/xarray.rst b/Documentation/core-api/xarray.rst
index 640934b6f7b4..1c201628a15a 100644
--- a/Documentation/core-api/xarray.rst
+++ b/Documentation/core-api/xarray.rst
@@ -394,7 +394,7 @@ You can use xas_init_marks() to reset the marks on an entry
 to their default state.  This is usually all marks clear, unless the
 XArray is marked with ``XA_FLAGS_TRACK_FREE``, in which case mark 0 is set
 and all other marks are clear.  Replacing one entry with another using
-xas_store() will not reset the marks on that entry; if you want
+xas_store_noinit() will not reset the marks on that entry; if you want
 the marks reset, you should do that explicitly.
 
 The xas_load() will walk the xa_state as close to the entry
@@ -458,7 +458,7 @@ save substantial quantities of memory; for example tying 512 entries
 together will save over 4kB.
 
 You can create a multi-index entry by using XA_STATE_ORDER()
-or xas_set_order() followed by a call to xas_store().
+or xas_set_order() followed by a call to xas_store_noinit().
 Calling xas_load() with a multi-index xa_state will walk the
 xa_state to the right location in the tree, but the return value is not
 meaningful, potentially being an internal entry or ``NULL`` even when there
diff --git a/include/linux/xarray.h b/include/linux/xarray.h
index 06acef49ec95..680c0dcaeb12 100644
--- a/include/linux/xarray.h
+++ b/include/linux/xarray.h
@@ -1490,7 +1490,6 @@ static inline bool xas_retry(struct xa_state *xas, const void *entry)
 }
 
 void *xas_load(struct xa_state *);
-void *xas_store(struct xa_state *, void *entry);
 void *xas_store_noinit(struct xa_state *, void *entry);
 void *xas_erase(struct xa_state *);
 void *xas_find(struct xa_state *, unsigned long max);
diff --git a/lib/xarray.c b/lib/xarray.c
index d87045d120ad..2912d706dc01 100644
--- a/lib/xarray.c
+++ b/lib/xarray.c
@@ -625,7 +625,7 @@ static int xas_expand(struct xa_state *xas, void *head)
  * @allow_root: %true if we can store the entry in the root directly
  *
  * Most users will not need to call this function directly, as it is called
- * by xas_store().  It is useful for doing conditional store operations
+ * by xas_store_noinit().  It is useful for doing conditional store operations
  * (see the xa_cmpxchg() implementation for an example).
  *
  * Return: If the slot already existed, returns the contents of this slot.
@@ -820,7 +820,7 @@ static void *__xas_store(struct xa_state *xas, void *entry, bool init_marks)
 }
 
 /**
- * xas_store() - Store this entry in the XArray.
+ * xas_store_noinit() - Store this entry in the XArray.
  * @xas: XArray operation state.
  * @entry: New entry.
  *
@@ -828,17 +828,11 @@ static void *__xas_store(struct xa_state *xas, void *entry, bool init_marks)
  * function is essentially meaningless (it may be an internal entry or it
  * may be %NULL, even if there are non-NULL entries at some of the indices
  * covered by the range).  This is not a problem for any current users,
- * and can be changed if needed.
+ * and can be changed if needed. Note that this function does not affect
+ * xarray marks.
  *
  * Return: The old entry at this index.
  */
-void *xas_store(struct xa_state *xas, void *entry)
-{
-	return __xas_store(xas, entry, true);
-}
-EXPORT_SYMBOL_GPL(xas_store);
-
-/* This is like xas_store() but does not initialize marks on stored entry */
 void *xas_store_noinit(struct xa_state *xas, void *entry)
 {
 	return __xas_store(xas, entry, false);
-- 
2.16.4