* [PATCH v2 01/12] mm: Make pagecache tagged lookups return only head pages
2020-09-14 13:00 [PATCH v2 00/12] Overhaul multi-page lookups for THP Matthew Wilcox (Oracle)
@ 2020-09-14 13:00 ` Matthew Wilcox (Oracle)
2020-09-29 9:13 ` Jan Kara
2020-09-14 13:00 ` [PATCH v2 02/12] mm/shmem: Use pagevec_lookup in shmem_unlock_mapping Matthew Wilcox (Oracle)
` (12 subsequent siblings)
13 siblings, 1 reply; 38+ messages in thread
From: Matthew Wilcox (Oracle) @ 2020-09-14 13:00 UTC (permalink / raw)
To: linux-mm
Cc: Matthew Wilcox (Oracle),
Andrew Morton, Hugh Dickins, William Kucharski, Johannes Weiner,
Jan Kara, Yang Shi, Dave Chinner, linux-kernel
Pagecache tags are used for dirty page writeback. Since dirtiness is
tracked on a per-THP basis, we only want to return the head page rather
than each subpage of a tagged page. All the filesystems which use huge
pages today are in-memory, so there are no tagged huge pages today.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
mm/filemap.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/mm/filemap.c b/mm/filemap.c
index f5fda3038782..7d8cf1a25628 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1968,7 +1968,7 @@ unsigned find_get_pages_contig(struct address_space *mapping, pgoff_t index,
EXPORT_SYMBOL(find_get_pages_contig);
/**
- * find_get_pages_range_tag - find and return pages in given range matching @tag
+ * find_get_pages_range_tag - Find and return head pages matching @tag.
* @mapping: the address_space to search
* @index: the starting page index
* @end: The final page index (inclusive)
@@ -1976,8 +1976,8 @@ EXPORT_SYMBOL(find_get_pages_contig);
* @nr_pages: the maximum number of pages
* @pages: where the resulting pages are placed
*
- * Like find_get_pages, except we only return pages which are tagged with
- * @tag. We update @index to index the next page for the traversal.
+ * Like find_get_pages(), except we only return head pages which are tagged
+ * with @tag. We update @index to index the next page for the traversal.
*
* Return: the number of pages which were found.
*/
@@ -2011,9 +2011,9 @@ unsigned find_get_pages_range_tag(struct address_space *mapping, pgoff_t *index,
if (unlikely(page != xas_reload(&xas)))
goto put_page;
- pages[ret] = find_subpage(page, xas.xa_index);
+ pages[ret] = page;
if (++ret == nr_pages) {
- *index = xas.xa_index + 1;
+ *index = page->index + thp_nr_pages(page);
goto out;
}
continue;
--
2.28.0
^ permalink raw reply related [flat|nested] 38+ messages in thread* Re: [PATCH v2 01/12] mm: Make pagecache tagged lookups return only head pages
2020-09-14 13:00 ` [PATCH v2 01/12] mm: Make pagecache tagged lookups return only head pages Matthew Wilcox (Oracle)
@ 2020-09-29 9:13 ` Jan Kara
0 siblings, 0 replies; 38+ messages in thread
From: Jan Kara @ 2020-09-29 9:13 UTC (permalink / raw)
To: Matthew Wilcox (Oracle)
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Jan Kara, Yang Shi, Dave Chinner, linux-kernel
On Mon 14-09-20 14:00:31, Matthew Wilcox (Oracle) wrote:
> Pagecache tags are used for dirty page writeback. Since dirtiness is
> tracked on a per-THP basis, we only want to return the head page rather
> than each subpage of a tagged page. All the filesystems which use huge
> pages today are in-memory, so there are no tagged huge pages today.
>
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Looks good to me. You can add:
Reviewed-by: Jan Kara <jack@suse.cz>
Honza
> ---
> mm/filemap.c | 10 +++++-----
> 1 file changed, 5 insertions(+), 5 deletions(-)
>
> diff --git a/mm/filemap.c b/mm/filemap.c
> index f5fda3038782..7d8cf1a25628 100644
> --- a/mm/filemap.c
> +++ b/mm/filemap.c
> @@ -1968,7 +1968,7 @@ unsigned find_get_pages_contig(struct address_space *mapping, pgoff_t index,
> EXPORT_SYMBOL(find_get_pages_contig);
>
> /**
> - * find_get_pages_range_tag - find and return pages in given range matching @tag
> + * find_get_pages_range_tag - Find and return head pages matching @tag.
> * @mapping: the address_space to search
> * @index: the starting page index
> * @end: The final page index (inclusive)
> @@ -1976,8 +1976,8 @@ EXPORT_SYMBOL(find_get_pages_contig);
> * @nr_pages: the maximum number of pages
> * @pages: where the resulting pages are placed
> *
> - * Like find_get_pages, except we only return pages which are tagged with
> - * @tag. We update @index to index the next page for the traversal.
> + * Like find_get_pages(), except we only return head pages which are tagged
> + * with @tag. We update @index to index the next page for the traversal.
> *
> * Return: the number of pages which were found.
> */
> @@ -2011,9 +2011,9 @@ unsigned find_get_pages_range_tag(struct address_space *mapping, pgoff_t *index,
> if (unlikely(page != xas_reload(&xas)))
> goto put_page;
>
> - pages[ret] = find_subpage(page, xas.xa_index);
> + pages[ret] = page;
> if (++ret == nr_pages) {
> - *index = xas.xa_index + 1;
> + *index = page->index + thp_nr_pages(page);
> goto out;
> }
> continue;
> --
> 2.28.0
>
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 38+ messages in thread
* [PATCH v2 02/12] mm/shmem: Use pagevec_lookup in shmem_unlock_mapping
2020-09-14 13:00 [PATCH v2 00/12] Overhaul multi-page lookups for THP Matthew Wilcox (Oracle)
2020-09-14 13:00 ` [PATCH v2 01/12] mm: Make pagecache tagged lookups return only head pages Matthew Wilcox (Oracle)
@ 2020-09-14 13:00 ` Matthew Wilcox (Oracle)
2020-09-29 8:28 ` Jan Kara
2020-09-14 13:00 ` [PATCH v2 03/12] mm/filemap: Add helper for finding pages Matthew Wilcox (Oracle)
` (11 subsequent siblings)
13 siblings, 1 reply; 38+ messages in thread
From: Matthew Wilcox (Oracle) @ 2020-09-14 13:00 UTC (permalink / raw)
To: linux-mm
Cc: Matthew Wilcox (Oracle),
Andrew Morton, Hugh Dickins, William Kucharski, Johannes Weiner,
Jan Kara, Yang Shi, Dave Chinner, linux-kernel
The comment shows that the reason for using find_get_entries() is now
stale; find_get_pages() will not return 0 if it hits a consecutive run
of swap entries, and I don't believe it has since 2011. pagevec_lookup()
is a simpler function to use than find_get_pages(), so use it instead.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
mm/shmem.c | 11 +----------
1 file changed, 1 insertion(+), 10 deletions(-)
diff --git a/mm/shmem.c b/mm/shmem.c
index 58bc9e326d0d..108931a6cc43 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -840,7 +840,6 @@ unsigned long shmem_swap_usage(struct vm_area_struct *vma)
void shmem_unlock_mapping(struct address_space *mapping)
{
struct pagevec pvec;
- pgoff_t indices[PAGEVEC_SIZE];
pgoff_t index = 0;
pagevec_init(&pvec);
@@ -848,16 +847,8 @@ void shmem_unlock_mapping(struct address_space *mapping)
* Minor point, but we might as well stop if someone else SHM_LOCKs it.
*/
while (!mapping_unevictable(mapping)) {
- /*
- * Avoid pagevec_lookup(): find_get_pages() returns 0 as if it
- * has finished, if it hits a row of PAGEVEC_SIZE swap entries.
- */
- pvec.nr = find_get_entries(mapping, index,
- PAGEVEC_SIZE, pvec.pages, indices);
- if (!pvec.nr)
+ if (!pagevec_lookup(&pvec, mapping, &index))
break;
- index = indices[pvec.nr - 1] + 1;
- pagevec_remove_exceptionals(&pvec);
check_move_unevictable_pages(&pvec);
pagevec_release(&pvec);
cond_resched();
--
2.28.0
^ permalink raw reply related [flat|nested] 38+ messages in thread* Re: [PATCH v2 02/12] mm/shmem: Use pagevec_lookup in shmem_unlock_mapping
2020-09-14 13:00 ` [PATCH v2 02/12] mm/shmem: Use pagevec_lookup in shmem_unlock_mapping Matthew Wilcox (Oracle)
@ 2020-09-29 8:28 ` Jan Kara
2020-09-29 18:36 ` Matthew Wilcox
0 siblings, 1 reply; 38+ messages in thread
From: Jan Kara @ 2020-09-29 8:28 UTC (permalink / raw)
To: Matthew Wilcox (Oracle)
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Jan Kara, Yang Shi, Dave Chinner, linux-kernel
On Mon 14-09-20 14:00:32, Matthew Wilcox (Oracle) wrote:
> The comment shows that the reason for using find_get_entries() is now
> stale; find_get_pages() will not return 0 if it hits a consecutive run
> of swap entries, and I don't believe it has since 2011. pagevec_lookup()
> is a simpler function to use than find_get_pages(), so use it instead.
>
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Looks good to me. BTW, I think I've already reviewed this... You can add:
Reviewed-by: Jan Kara <jack@suse.cz>
Honza
> ---
> mm/shmem.c | 11 +----------
> 1 file changed, 1 insertion(+), 10 deletions(-)
>
> diff --git a/mm/shmem.c b/mm/shmem.c
> index 58bc9e326d0d..108931a6cc43 100644
> --- a/mm/shmem.c
> +++ b/mm/shmem.c
> @@ -840,7 +840,6 @@ unsigned long shmem_swap_usage(struct vm_area_struct *vma)
> void shmem_unlock_mapping(struct address_space *mapping)
> {
> struct pagevec pvec;
> - pgoff_t indices[PAGEVEC_SIZE];
> pgoff_t index = 0;
>
> pagevec_init(&pvec);
> @@ -848,16 +847,8 @@ void shmem_unlock_mapping(struct address_space *mapping)
> * Minor point, but we might as well stop if someone else SHM_LOCKs it.
> */
> while (!mapping_unevictable(mapping)) {
> - /*
> - * Avoid pagevec_lookup(): find_get_pages() returns 0 as if it
> - * has finished, if it hits a row of PAGEVEC_SIZE swap entries.
> - */
> - pvec.nr = find_get_entries(mapping, index,
> - PAGEVEC_SIZE, pvec.pages, indices);
> - if (!pvec.nr)
> + if (!pagevec_lookup(&pvec, mapping, &index))
> break;
> - index = indices[pvec.nr - 1] + 1;
> - pagevec_remove_exceptionals(&pvec);
> check_move_unevictable_pages(&pvec);
> pagevec_release(&pvec);
> cond_resched();
> --
> 2.28.0
>
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: [PATCH v2 02/12] mm/shmem: Use pagevec_lookup in shmem_unlock_mapping
2020-09-29 8:28 ` Jan Kara
@ 2020-09-29 18:36 ` Matthew Wilcox
0 siblings, 0 replies; 38+ messages in thread
From: Matthew Wilcox @ 2020-09-29 18:36 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Yang Shi, Dave Chinner, linux-kernel
On Tue, Sep 29, 2020 at 10:28:28AM +0200, Jan Kara wrote:
> On Mon 14-09-20 14:00:32, Matthew Wilcox (Oracle) wrote:
> > The comment shows that the reason for using find_get_entries() is now
> > stale; find_get_pages() will not return 0 if it hits a consecutive run
> > of swap entries, and I don't believe it has since 2011. pagevec_lookup()
> > is a simpler function to use than find_get_pages(), so use it instead.
> >
> > Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
>
> Looks good to me. BTW, I think I've already reviewed this... You can add:
>
> Reviewed-by: Jan Kara <jack@suse.cz>
So you did! My apologies for missing that.
^ permalink raw reply [flat|nested] 38+ messages in thread
* [PATCH v2 03/12] mm/filemap: Add helper for finding pages
2020-09-14 13:00 [PATCH v2 00/12] Overhaul multi-page lookups for THP Matthew Wilcox (Oracle)
2020-09-14 13:00 ` [PATCH v2 01/12] mm: Make pagecache tagged lookups return only head pages Matthew Wilcox (Oracle)
2020-09-14 13:00 ` [PATCH v2 02/12] mm/shmem: Use pagevec_lookup in shmem_unlock_mapping Matthew Wilcox (Oracle)
@ 2020-09-14 13:00 ` Matthew Wilcox (Oracle)
2020-09-29 8:27 ` Jan Kara
2020-09-14 13:00 ` [PATCH v2 04/12] mm/filemap: Add mapping_seek_hole_data Matthew Wilcox (Oracle)
` (10 subsequent siblings)
13 siblings, 1 reply; 38+ messages in thread
From: Matthew Wilcox (Oracle) @ 2020-09-14 13:00 UTC (permalink / raw)
To: linux-mm
Cc: Matthew Wilcox (Oracle),
Andrew Morton, Hugh Dickins, William Kucharski, Johannes Weiner,
Jan Kara, Yang Shi, Dave Chinner, linux-kernel
There is a lot of common code in find_get_entries(),
find_get_pages_range() and find_get_pages_range_tag(). Factor out
xas_find_get_entry() which simplifies all three functions.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
mm/filemap.c | 98 +++++++++++++++++++++++-----------------------------
1 file changed, 43 insertions(+), 55 deletions(-)
diff --git a/mm/filemap.c b/mm/filemap.c
index 7d8cf1a25628..d8f5ff07eb9c 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1758,6 +1758,43 @@ struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index,
}
EXPORT_SYMBOL(pagecache_get_page);
+static inline struct page *xas_find_get_entry(struct xa_state *xas,
+ pgoff_t max, xa_mark_t mark)
+{
+ struct page *page;
+
+retry:
+ if (mark == XA_PRESENT)
+ page = xas_find(xas, max);
+ else
+ page = xas_find_marked(xas, max, mark);
+
+ if (xas_retry(xas, page))
+ goto retry;
+ /*
+ * A shadow entry of a recently evicted page, a swap
+ * entry from shmem/tmpfs or a DAX entry. Return it
+ * without attempting to raise page count.
+ */
+ if (!page || xa_is_value(page))
+ return page;
+
+ if (!page_cache_get_speculative(page))
+ goto reset;
+
+ /* Has the page moved or been split? */
+ if (unlikely(page != xas_reload(xas))) {
+ put_page(page);
+ goto reset;
+ }
+ VM_BUG_ON_PAGE(!thp_contains(page, xas->xa_index), page);
+
+ return page;
+reset:
+ xas_reset(xas);
+ goto retry;
+}
+
/**
* find_get_entries - gang pagecache lookup
* @mapping: The address_space to search
@@ -1797,42 +1834,21 @@ unsigned find_get_entries(struct address_space *mapping,
return 0;
rcu_read_lock();
- xas_for_each(&xas, page, ULONG_MAX) {
- if (xas_retry(&xas, page))
- continue;
- /*
- * A shadow entry of a recently evicted page, a swap
- * entry from shmem/tmpfs or a DAX entry. Return it
- * without attempting to raise page count.
- */
- if (xa_is_value(page))
- goto export;
-
- if (!page_cache_get_speculative(page))
- goto retry;
-
- /* Has the page moved or been split? */
- if (unlikely(page != xas_reload(&xas)))
- goto put_page;
-
+ while ((page = xas_find_get_entry(&xas, ULONG_MAX, XA_PRESENT))) {
/*
* Terminate early on finding a THP, to allow the caller to
* handle it all at once; but continue if this is hugetlbfs.
*/
- if (PageTransHuge(page) && !PageHuge(page)) {
+ if (!xa_is_value(page) && PageTransHuge(page) &&
+ !PageHuge(page)) {
page = find_subpage(page, xas.xa_index);
nr_entries = ret + 1;
}
-export:
+
indices[ret] = xas.xa_index;
entries[ret] = page;
if (++ret == nr_entries)
break;
- continue;
-put_page:
- put_page(page);
-retry:
- xas_reset(&xas);
}
rcu_read_unlock();
return ret;
@@ -1871,30 +1887,16 @@ unsigned find_get_pages_range(struct address_space *mapping, pgoff_t *start,
return 0;
rcu_read_lock();
- xas_for_each(&xas, page, end) {
- if (xas_retry(&xas, page))
- continue;
+ while ((page = xas_find_get_entry(&xas, end, XA_PRESENT))) {
/* Skip over shadow, swap and DAX entries */
if (xa_is_value(page))
continue;
- if (!page_cache_get_speculative(page))
- goto retry;
-
- /* Has the page moved or been split? */
- if (unlikely(page != xas_reload(&xas)))
- goto put_page;
-
pages[ret] = find_subpage(page, xas.xa_index);
if (++ret == nr_pages) {
*start = xas.xa_index + 1;
goto out;
}
- continue;
-put_page:
- put_page(page);
-retry:
- xas_reset(&xas);
}
/*
@@ -1993,9 +1995,7 @@ unsigned find_get_pages_range_tag(struct address_space *mapping, pgoff_t *index,
return 0;
rcu_read_lock();
- xas_for_each_marked(&xas, page, end, tag) {
- if (xas_retry(&xas, page))
- continue;
+ while ((page = xas_find_get_entry(&xas, end, tag))) {
/*
* Shadow entries should never be tagged, but this iteration
* is lockless so there is a window for page reclaim to evict
@@ -2004,23 +2004,11 @@ unsigned find_get_pages_range_tag(struct address_space *mapping, pgoff_t *index,
if (xa_is_value(page))
continue;
- if (!page_cache_get_speculative(page))
- goto retry;
-
- /* Has the page moved or been split? */
- if (unlikely(page != xas_reload(&xas)))
- goto put_page;
-
pages[ret] = page;
if (++ret == nr_pages) {
*index = page->index + thp_nr_pages(page);
goto out;
}
- continue;
-put_page:
- put_page(page);
-retry:
- xas_reset(&xas);
}
/*
--
2.28.0
^ permalink raw reply related [flat|nested] 38+ messages in thread* Re: [PATCH v2 03/12] mm/filemap: Add helper for finding pages
2020-09-14 13:00 ` [PATCH v2 03/12] mm/filemap: Add helper for finding pages Matthew Wilcox (Oracle)
@ 2020-09-29 8:27 ` Jan Kara
0 siblings, 0 replies; 38+ messages in thread
From: Jan Kara @ 2020-09-29 8:27 UTC (permalink / raw)
To: Matthew Wilcox (Oracle)
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Jan Kara, Yang Shi, Dave Chinner, linux-kernel
On Mon 14-09-20 14:00:33, Matthew Wilcox (Oracle) wrote:
> There is a lot of common code in find_get_entries(),
> find_get_pages_range() and find_get_pages_range_tag(). Factor out
> xas_find_get_entry() which simplifies all three functions.
>
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Looks good. You can add:
Reviewed-by: Jan Kara <jack@suse.cz>
Honza
> ---
> mm/filemap.c | 98 +++++++++++++++++++++++-----------------------------
> 1 file changed, 43 insertions(+), 55 deletions(-)
>
> diff --git a/mm/filemap.c b/mm/filemap.c
> index 7d8cf1a25628..d8f5ff07eb9c 100644
> --- a/mm/filemap.c
> +++ b/mm/filemap.c
> @@ -1758,6 +1758,43 @@ struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index,
> }
> EXPORT_SYMBOL(pagecache_get_page);
>
> +static inline struct page *xas_find_get_entry(struct xa_state *xas,
> + pgoff_t max, xa_mark_t mark)
> +{
> + struct page *page;
> +
> +retry:
> + if (mark == XA_PRESENT)
> + page = xas_find(xas, max);
> + else
> + page = xas_find_marked(xas, max, mark);
> +
> + if (xas_retry(xas, page))
> + goto retry;
> + /*
> + * A shadow entry of a recently evicted page, a swap
> + * entry from shmem/tmpfs or a DAX entry. Return it
> + * without attempting to raise page count.
> + */
> + if (!page || xa_is_value(page))
> + return page;
> +
> + if (!page_cache_get_speculative(page))
> + goto reset;
> +
> + /* Has the page moved or been split? */
> + if (unlikely(page != xas_reload(xas))) {
> + put_page(page);
> + goto reset;
> + }
> + VM_BUG_ON_PAGE(!thp_contains(page, xas->xa_index), page);
> +
> + return page;
> +reset:
> + xas_reset(xas);
> + goto retry;
> +}
> +
> /**
> * find_get_entries - gang pagecache lookup
> * @mapping: The address_space to search
> @@ -1797,42 +1834,21 @@ unsigned find_get_entries(struct address_space *mapping,
> return 0;
>
> rcu_read_lock();
> - xas_for_each(&xas, page, ULONG_MAX) {
> - if (xas_retry(&xas, page))
> - continue;
> - /*
> - * A shadow entry of a recently evicted page, a swap
> - * entry from shmem/tmpfs or a DAX entry. Return it
> - * without attempting to raise page count.
> - */
> - if (xa_is_value(page))
> - goto export;
> -
> - if (!page_cache_get_speculative(page))
> - goto retry;
> -
> - /* Has the page moved or been split? */
> - if (unlikely(page != xas_reload(&xas)))
> - goto put_page;
> -
> + while ((page = xas_find_get_entry(&xas, ULONG_MAX, XA_PRESENT))) {
> /*
> * Terminate early on finding a THP, to allow the caller to
> * handle it all at once; but continue if this is hugetlbfs.
> */
> - if (PageTransHuge(page) && !PageHuge(page)) {
> + if (!xa_is_value(page) && PageTransHuge(page) &&
> + !PageHuge(page)) {
> page = find_subpage(page, xas.xa_index);
> nr_entries = ret + 1;
> }
> -export:
> +
> indices[ret] = xas.xa_index;
> entries[ret] = page;
> if (++ret == nr_entries)
> break;
> - continue;
> -put_page:
> - put_page(page);
> -retry:
> - xas_reset(&xas);
> }
> rcu_read_unlock();
> return ret;
> @@ -1871,30 +1887,16 @@ unsigned find_get_pages_range(struct address_space *mapping, pgoff_t *start,
> return 0;
>
> rcu_read_lock();
> - xas_for_each(&xas, page, end) {
> - if (xas_retry(&xas, page))
> - continue;
> + while ((page = xas_find_get_entry(&xas, end, XA_PRESENT))) {
> /* Skip over shadow, swap and DAX entries */
> if (xa_is_value(page))
> continue;
>
> - if (!page_cache_get_speculative(page))
> - goto retry;
> -
> - /* Has the page moved or been split? */
> - if (unlikely(page != xas_reload(&xas)))
> - goto put_page;
> -
> pages[ret] = find_subpage(page, xas.xa_index);
> if (++ret == nr_pages) {
> *start = xas.xa_index + 1;
> goto out;
> }
> - continue;
> -put_page:
> - put_page(page);
> -retry:
> - xas_reset(&xas);
> }
>
> /*
> @@ -1993,9 +1995,7 @@ unsigned find_get_pages_range_tag(struct address_space *mapping, pgoff_t *index,
> return 0;
>
> rcu_read_lock();
> - xas_for_each_marked(&xas, page, end, tag) {
> - if (xas_retry(&xas, page))
> - continue;
> + while ((page = xas_find_get_entry(&xas, end, tag))) {
> /*
> * Shadow entries should never be tagged, but this iteration
> * is lockless so there is a window for page reclaim to evict
> @@ -2004,23 +2004,11 @@ unsigned find_get_pages_range_tag(struct address_space *mapping, pgoff_t *index,
> if (xa_is_value(page))
> continue;
>
> - if (!page_cache_get_speculative(page))
> - goto retry;
> -
> - /* Has the page moved or been split? */
> - if (unlikely(page != xas_reload(&xas)))
> - goto put_page;
> -
> pages[ret] = page;
> if (++ret == nr_pages) {
> *index = page->index + thp_nr_pages(page);
> goto out;
> }
> - continue;
> -put_page:
> - put_page(page);
> -retry:
> - xas_reset(&xas);
> }
>
> /*
> --
> 2.28.0
>
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 38+ messages in thread
* [PATCH v2 04/12] mm/filemap: Add mapping_seek_hole_data
2020-09-14 13:00 [PATCH v2 00/12] Overhaul multi-page lookups for THP Matthew Wilcox (Oracle)
` (2 preceding siblings ...)
2020-09-14 13:00 ` [PATCH v2 03/12] mm/filemap: Add helper for finding pages Matthew Wilcox (Oracle)
@ 2020-09-14 13:00 ` Matthew Wilcox (Oracle)
2020-09-29 8:46 ` Jan Kara
2020-09-14 13:00 ` [PATCH v2 05/12] mm: Add and use find_lock_entries Matthew Wilcox (Oracle)
` (9 subsequent siblings)
13 siblings, 1 reply; 38+ messages in thread
From: Matthew Wilcox (Oracle) @ 2020-09-14 13:00 UTC (permalink / raw)
To: linux-mm
Cc: Matthew Wilcox (Oracle),
Andrew Morton, Hugh Dickins, William Kucharski, Johannes Weiner,
Jan Kara, Yang Shi, Dave Chinner, linux-kernel
Rewrite shmem_seek_hole_data() and move it to filemap.c.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
include/linux/pagemap.h | 2 ++
mm/filemap.c | 68 ++++++++++++++++++++++++++++++++++++++
mm/shmem.c | 72 +++--------------------------------------
3 files changed, 74 insertions(+), 68 deletions(-)
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 4e52a3ff92fb..869dc371b800 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -724,6 +724,8 @@ extern void __delete_from_page_cache(struct page *page, void *shadow);
int replace_page_cache_page(struct page *old, struct page *new, gfp_t gfp_mask);
void delete_from_page_cache_batch(struct address_space *mapping,
struct pagevec *pvec);
+loff_t mapping_seek_hole_data(struct address_space *, loff_t start, loff_t end,
+ int whence);
/*
* Like add_to_page_cache_locked, but used to add newly allocated pages:
diff --git a/mm/filemap.c b/mm/filemap.c
index d8f5ff07eb9c..5662f932b85b 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2416,6 +2416,74 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
}
EXPORT_SYMBOL(generic_file_read_iter);
+static inline
+unsigned int seek_page_size(struct xa_state *xas, struct page *page)
+{
+ if (xa_is_value(page))
+ return PAGE_SIZE << xa_get_order(xas->xa, xas->xa_index);
+ return thp_size(page);
+}
+
+/**
+ * mapping_seek_hole_data - Seek for SEEK_DATA / SEEK_HOLE in the page cache.
+ * @mapping: Address space to search.
+ * @start: First byte to consider.
+ * @end: Limit of search (exclusive).
+ * @whence: Either SEEK_HOLE or SEEK_DATA.
+ *
+ * If the page cache knows which blocks contain holes and which blocks
+ * contain data, your filesystem can use this function to implement
+ * SEEK_HOLE and SEEK_DATA. This is useful for filesystems which are
+ * entirely memory-based such as tmpfs, and filesystems which support
+ * unwritten extents.
+ *
+ * Return: The requested offset on successs, or -ENXIO if @whence specifies
+ * SEEK_DATA and there is no data after @start. There is an implicit hole
+ * after @end - 1, so SEEK_HOLE returns @end if all the bytes between @start
+ * and @end contain data.
+ */
+loff_t mapping_seek_hole_data(struct address_space *mapping, loff_t start,
+ loff_t end, int whence)
+{
+ XA_STATE(xas, &mapping->i_pages, start >> PAGE_SHIFT);
+ pgoff_t max = (end - 1) / PAGE_SIZE;
+ bool seek_data = (whence == SEEK_DATA);
+ struct page *page;
+
+ if (end <= start)
+ return -ENXIO;
+
+ rcu_read_lock();
+ while ((page = xas_find_get_entry(&xas, max, XA_PRESENT))) {
+ loff_t pos = xas.xa_index * PAGE_SIZE;
+
+ if (start < pos) {
+ if (!seek_data)
+ goto unlock;
+ start = pos;
+ }
+
+ if (seek_data)
+ goto unlock;
+
+ start = pos + seek_page_size(&xas, page);
+ }
+ rcu_read_unlock();
+
+ if (seek_data)
+ return -ENXIO;
+ goto out;
+
+unlock:
+ rcu_read_unlock();
+ if (!xa_is_value(page))
+ put_page(page);
+out:
+ if (start > end)
+ return end;
+ return start;
+}
+
#ifdef CONFIG_MMU
#define MMAP_LOTSAMISS (100)
/*
diff --git a/mm/shmem.c b/mm/shmem.c
index 108931a6cc43..b65263d9bb67 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -2659,85 +2659,21 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
return retval ? retval : error;
}
-/*
- * llseek SEEK_DATA or SEEK_HOLE through the page cache.
- */
-static pgoff_t shmem_seek_hole_data(struct address_space *mapping,
- pgoff_t index, pgoff_t end, int whence)
-{
- struct page *page;
- struct pagevec pvec;
- pgoff_t indices[PAGEVEC_SIZE];
- bool done = false;
- int i;
-
- pagevec_init(&pvec);
- pvec.nr = 1; /* start small: we may be there already */
- while (!done) {
- pvec.nr = find_get_entries(mapping, index,
- pvec.nr, pvec.pages, indices);
- if (!pvec.nr) {
- if (whence == SEEK_DATA)
- index = end;
- break;
- }
- for (i = 0; i < pvec.nr; i++, index++) {
- if (index < indices[i]) {
- if (whence == SEEK_HOLE) {
- done = true;
- break;
- }
- index = indices[i];
- }
- page = pvec.pages[i];
- if (page && !xa_is_value(page)) {
- if (!PageUptodate(page))
- page = NULL;
- }
- if (index >= end ||
- (page && whence == SEEK_DATA) ||
- (!page && whence == SEEK_HOLE)) {
- done = true;
- break;
- }
- }
- pagevec_remove_exceptionals(&pvec);
- pagevec_release(&pvec);
- pvec.nr = PAGEVEC_SIZE;
- cond_resched();
- }
- return index;
-}
-
static loff_t shmem_file_llseek(struct file *file, loff_t offset, int whence)
{
struct address_space *mapping = file->f_mapping;
struct inode *inode = mapping->host;
- pgoff_t start, end;
- loff_t new_offset;
if (whence != SEEK_DATA && whence != SEEK_HOLE)
return generic_file_llseek_size(file, offset, whence,
MAX_LFS_FILESIZE, i_size_read(inode));
+ if (offset < 0)
+ return -ENXIO;
+
inode_lock(inode);
/* We're holding i_mutex so we can access i_size directly */
- if (offset < 0 || offset >= inode->i_size)
- offset = -ENXIO;
- else {
- start = offset >> PAGE_SHIFT;
- end = (inode->i_size + PAGE_SIZE - 1) >> PAGE_SHIFT;
- new_offset = shmem_seek_hole_data(mapping, start, end, whence);
- new_offset <<= PAGE_SHIFT;
- if (new_offset > offset) {
- if (new_offset < inode->i_size)
- offset = new_offset;
- else if (whence == SEEK_DATA)
- offset = -ENXIO;
- else
- offset = inode->i_size;
- }
- }
+ offset = mapping_seek_hole_data(mapping, offset, inode->i_size, whence);
if (offset >= 0)
offset = vfs_setpos(file, offset, MAX_LFS_FILESIZE);
--
2.28.0
^ permalink raw reply related [flat|nested] 38+ messages in thread* Re: [PATCH v2 04/12] mm/filemap: Add mapping_seek_hole_data
2020-09-14 13:00 ` [PATCH v2 04/12] mm/filemap: Add mapping_seek_hole_data Matthew Wilcox (Oracle)
@ 2020-09-29 8:46 ` Jan Kara
2020-09-29 12:42 ` Matthew Wilcox
0 siblings, 1 reply; 38+ messages in thread
From: Jan Kara @ 2020-09-29 8:46 UTC (permalink / raw)
To: Matthew Wilcox (Oracle)
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Jan Kara, Yang Shi, Dave Chinner, linux-kernel
On Mon 14-09-20 14:00:34, Matthew Wilcox (Oracle) wrote:
> Rewrite shmem_seek_hole_data() and move it to filemap.c.
>
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
...
> +/**
> + * mapping_seek_hole_data - Seek for SEEK_DATA / SEEK_HOLE in the page cache.
> + * @mapping: Address space to search.
> + * @start: First byte to consider.
> + * @end: Limit of search (exclusive).
> + * @whence: Either SEEK_HOLE or SEEK_DATA.
> + *
> + * If the page cache knows which blocks contain holes and which blocks
> + * contain data, your filesystem can use this function to implement
> + * SEEK_HOLE and SEEK_DATA. This is useful for filesystems which are
> + * entirely memory-based such as tmpfs, and filesystems which support
> + * unwritten extents.
> + *
> + * Return: The requested offset on successs, or -ENXIO if @whence specifies
> + * SEEK_DATA and there is no data after @start. There is an implicit hole
> + * after @end - 1, so SEEK_HOLE returns @end if all the bytes between @start
> + * and @end contain data.
> + */
> +loff_t mapping_seek_hole_data(struct address_space *mapping, loff_t start,
> + loff_t end, int whence)
> +{
> + XA_STATE(xas, &mapping->i_pages, start >> PAGE_SHIFT);
> + pgoff_t max = (end - 1) / PAGE_SIZE;
> + bool seek_data = (whence == SEEK_DATA);
> + struct page *page;
> +
> + if (end <= start)
> + return -ENXIO;
> +
> + rcu_read_lock();
> + while ((page = xas_find_get_entry(&xas, max, XA_PRESENT))) {
> + loff_t pos = xas.xa_index * PAGE_SIZE;
OK, but for ordinary filesystems this could be problematic because of
exceptional entries?
Also for shmem you've dropped the PageUptodate check which I'm not sure is
safe?
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: [PATCH v2 04/12] mm/filemap: Add mapping_seek_hole_data
2020-09-29 8:46 ` Jan Kara
@ 2020-09-29 12:42 ` Matthew Wilcox
2020-09-29 13:39 ` Matthew Wilcox
0 siblings, 1 reply; 38+ messages in thread
From: Matthew Wilcox @ 2020-09-29 12:42 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Yang Shi, Dave Chinner, linux-kernel
On Tue, Sep 29, 2020 at 10:46:53AM +0200, Jan Kara wrote:
> On Mon 14-09-20 14:00:34, Matthew Wilcox (Oracle) wrote:
> > Rewrite shmem_seek_hole_data() and move it to filemap.c.
> >
> > + rcu_read_lock();
> > + while ((page = xas_find_get_entry(&xas, max, XA_PRESENT))) {
> > + loff_t pos = xas.xa_index * PAGE_SIZE;
>
> OK, but for ordinary filesystems this could be problematic because of
> exceptional entries?
For ordinary filesystems, I have this queued up on top:
http://git.infradead.org/users/willy/pagecache.git/commitdiff/02c740b215bab901f95a560759b3bd906648da08
which handles exceptional entries. It treats shadow/swap/DAX entries
the same -- there's definitely data there, it's just not in a struct
page right now.
> Also for shmem you've dropped the PageUptodate check which I'm not sure is
> safe?
That was unintentional. I did run xfstests against this patch (just did
it again ... it passes), so I suspect it doesn't create a !Uptodate page.
I'll see if I can enhance the existing xfstests to catch this case.
The patch I link to above also doesn't handle !Uptodate pages on shmem
filesystems the same way that the current code does. So ... on top of
this patch, I propose doing this:
@@ -2416,6 +2416,14 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_ite
r *iter)
}
EXPORT_SYMBOL(generic_file_read_iter);
+static inline loff_t page_seek_hole_data(struct page *page,
+ loff_t start, loff_t end, bool seek_data)
+{
+ if (xa_is_value(page) || PageUptodate(page))
+ return seek_data ? start : end;
+ return seek_data ? end : start;
+}
+
static inline
unsigned int seek_page_size(struct xa_state *xas, struct page *page)
{
@@ -2463,10 +2471,10 @@ loff_t mapping_seek_hole_data(struct address_space *mapping, loff_t start,
start = pos;
}
- if (seek_data)
+ pos += seek_page_size(&xas, page);
+ start = page_seek_hole_data(page, start, pos, seek_data);
+ if (start < pos)
goto unlock;
-
- start = pos + seek_page_size(&xas, page);
}
rcu_read_unlock();
... and then rebasing the other patch on top of this works out nicely.
Here's the result:
http://git.infradead.org/users/willy/pagecache.git/commitdiff/9eb3f496b7cdcdcae83026e861e148f46921c367
http://git.infradead.org/users/willy/pagecache.git/commitdiff/7d93274088f0872d849a906d783dc260bee106b9
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: [PATCH v2 04/12] mm/filemap: Add mapping_seek_hole_data
2020-09-29 12:42 ` Matthew Wilcox
@ 2020-09-29 13:39 ` Matthew Wilcox
0 siblings, 0 replies; 38+ messages in thread
From: Matthew Wilcox @ 2020-09-29 13:39 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Yang Shi, Dave Chinner, linux-kernel
On Tue, Sep 29, 2020 at 01:42:51PM +0100, Matthew Wilcox wrote:
> On Tue, Sep 29, 2020 at 10:46:53AM +0200, Jan Kara wrote:
> > Also for shmem you've dropped the PageUptodate check which I'm not sure is
> > safe?
>
> That was unintentional. I did run xfstests against this patch (just did
> it again ... it passes), so I suspect it doesn't create a !Uptodate page.
> I'll see if I can enhance the existing xfstests to catch this case.
Ah. Diff'ing the output between before and after ...
-Test skipped as fs doesn't support unwritten extents.
+07.01 SEEK_HOLE expected 0 or 45056, got 0. succ
+07.02 SEEK_HOLE expected 1 or 45056, got 1. succ
+07.03 SEEK_DATA expected 40960 or 40960, got 40960. succ
+07.04 SEEK_DATA expected 40960 or 40960, got 40960. succ
so, er, the tests didn't report that I'd broken it because it was just an
automatically skipped test. Not sure what to do about that; obviously
we should skip tests that aren't applicable, but it'd be nice to see a
warning that tmpfs used to support this and now doesn't.
^ permalink raw reply [flat|nested] 38+ messages in thread
* [PATCH v2 05/12] mm: Add and use find_lock_entries
2020-09-14 13:00 [PATCH v2 00/12] Overhaul multi-page lookups for THP Matthew Wilcox (Oracle)
` (3 preceding siblings ...)
2020-09-14 13:00 ` [PATCH v2 04/12] mm/filemap: Add mapping_seek_hole_data Matthew Wilcox (Oracle)
@ 2020-09-14 13:00 ` Matthew Wilcox (Oracle)
2020-09-29 8:58 ` Jan Kara
2020-09-14 13:00 ` [PATCH v2 06/12] mm: Add an 'end' parameter to find_get_entries Matthew Wilcox (Oracle)
` (8 subsequent siblings)
13 siblings, 1 reply; 38+ messages in thread
From: Matthew Wilcox (Oracle) @ 2020-09-14 13:00 UTC (permalink / raw)
To: linux-mm
Cc: Matthew Wilcox (Oracle),
Andrew Morton, Hugh Dickins, William Kucharski, Johannes Weiner,
Jan Kara, Yang Shi, Dave Chinner, linux-kernel
We have three functions (shmem_undo_range(), truncate_inode_pages_range()
and invalidate_mapping_pages()) which want exactly this function, so
add it to filemap.c.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
mm/filemap.c | 57 ++++++++++++++++++++++++++++++++
mm/internal.h | 2 ++
mm/shmem.c | 22 +++----------
mm/truncate.c | 90 +++++++--------------------------------------------
4 files changed, 75 insertions(+), 96 deletions(-)
diff --git a/mm/filemap.c b/mm/filemap.c
index 5662f932b85b..9f7f6b46aee4 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1854,6 +1854,63 @@ unsigned find_get_entries(struct address_space *mapping,
return ret;
}
+/**
+ * find_lock_entries - Find a batch of pagecache entries.
+ * @mapping: The address_space to search.
+ * @start: The starting page cache index.
+ * @end: The final page index (inclusive).
+ * @pvec: Where the resulting entries are placed.
+ * @indices: The cache indices of the entries in @pvec.
+ *
+ * find_lock_entries() will return a batch of entries from @mapping.
+ * Swap, shadow and DAX entries are included. Pages are returned
+ * locked and with an incremented refcount. Pages which are locked by
+ * somebody else or under writeback are skipped. Only the head page of
+ * a THP is returned. Pages which are partially outside the range are
+ * not returned.
+ *
+ * The entries have ascending indexes. The indices may not be consecutive
+ * due to not-present entries, THP pages, pages which could not be locked
+ * or pages under writeback.
+ *
+ * Return: The number of entries which were found.
+ */
+unsigned find_lock_entries(struct address_space *mapping, pgoff_t start,
+ pgoff_t end, struct pagevec *pvec, pgoff_t *indices)
+{
+ XA_STATE(xas, &mapping->i_pages, start);
+ struct page *page;
+
+ rcu_read_lock();
+ while ((page = xas_find_get_entry(&xas, end, XA_PRESENT))) {
+ if (!xa_is_value(page)) {
+ if (page->index < start)
+ goto put;
+ VM_BUG_ON_PAGE(page->index != xas.xa_index, page);
+ if (page->index + thp_nr_pages(page) - 1 > end)
+ goto put;
+ if (!trylock_page(page))
+ goto put;
+ if (page->mapping != mapping || PageWriteback(page))
+ goto unlock;
+ }
+ indices[pvec->nr] = xas.xa_index;
+ if (!pagevec_add(pvec, page))
+ break;
+ goto next;
+unlock:
+ unlock_page(page);
+put:
+ put_page(page);
+next:
+ if (!xa_is_value(page) && PageTransHuge(page))
+ xas_set(&xas, page->index + thp_nr_pages(page));
+ }
+ rcu_read_unlock();
+
+ return pagevec_count(pvec);
+}
+
/**
* find_get_pages_range - gang pagecache lookup
* @mapping: The address_space to search
diff --git a/mm/internal.h b/mm/internal.h
index 6345b08ce86c..1c5f4425ed74 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -62,6 +62,8 @@ static inline void force_page_cache_readahead(struct address_space *mapping,
struct page *find_get_entry(struct address_space *mapping, pgoff_t index);
struct page *find_lock_entry(struct address_space *mapping, pgoff_t index);
+unsigned find_lock_entries(struct address_space *mapping, pgoff_t start,
+ pgoff_t end, struct pagevec *pvec, pgoff_t *indices);
/**
* page_evictable - test whether a page is evictable
diff --git a/mm/shmem.c b/mm/shmem.c
index b65263d9bb67..a73ce8ce28e3 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -905,12 +905,8 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
pagevec_init(&pvec);
index = start;
- while (index < end) {
- pvec.nr = find_get_entries(mapping, index,
- min(end - index, (pgoff_t)PAGEVEC_SIZE),
- pvec.pages, indices);
- if (!pvec.nr)
- break;
+ while (index < end && find_lock_entries(mapping, index, end - 1,
+ &pvec, indices)) {
for (i = 0; i < pagevec_count(&pvec); i++) {
struct page *page = pvec.pages[i];
@@ -925,18 +921,10 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
index, page);
continue;
}
+ index += thp_nr_pages(page) - 1;
- VM_BUG_ON_PAGE(page_to_pgoff(page) != index, page);
-
- if (!trylock_page(page))
- continue;
-
- if ((!unfalloc || !PageUptodate(page)) &&
- page_mapping(page) == mapping) {
- VM_BUG_ON_PAGE(PageWriteback(page), page);
- if (shmem_punch_compound(page, start, end))
- truncate_inode_page(mapping, page);
- }
+ if (!unfalloc || !PageUptodate(page))
+ truncate_inode_page(mapping, page);
unlock_page(page);
}
pagevec_remove_exceptionals(&pvec);
diff --git a/mm/truncate.c b/mm/truncate.c
index 1cc93b57fb41..5dbe0c77b5ac 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -326,51 +326,19 @@ void truncate_inode_pages_range(struct address_space *mapping,
pagevec_init(&pvec);
index = start;
- while (index < end && pagevec_lookup_entries(&pvec, mapping, index,
- min(end - index, (pgoff_t)PAGEVEC_SIZE),
- indices)) {
- /*
- * Pagevec array has exceptional entries and we may also fail
- * to lock some pages. So we store pages that can be deleted
- * in a new pagevec.
- */
- struct pagevec locked_pvec;
-
- pagevec_init(&locked_pvec);
- for (i = 0; i < pagevec_count(&pvec); i++) {
- struct page *page = pvec.pages[i];
-
- /* We rely upon deletion not changing page->index */
- index = indices[i];
- if (index >= end)
- break;
-
- if (xa_is_value(page))
- continue;
-
- if (!trylock_page(page))
- continue;
- WARN_ON(page_to_index(page) != index);
- if (PageWriteback(page)) {
- unlock_page(page);
- continue;
- }
- if (page->mapping != mapping) {
- unlock_page(page);
- continue;
- }
- pagevec_add(&locked_pvec, page);
- }
- for (i = 0; i < pagevec_count(&locked_pvec); i++)
- truncate_cleanup_page(mapping, locked_pvec.pages[i]);
- delete_from_page_cache_batch(mapping, &locked_pvec);
- for (i = 0; i < pagevec_count(&locked_pvec); i++)
- unlock_page(locked_pvec.pages[i]);
+ while (index < end && find_lock_entries(mapping, index, end - 1,
+ &pvec, indices)) {
+ index = indices[pagevec_count(&pvec) - 1] + 1;
truncate_exceptional_pvec_entries(mapping, &pvec, indices, end);
+ for (i = 0; i < pagevec_count(&pvec); i++)
+ truncate_cleanup_page(mapping, pvec.pages[i]);
+ delete_from_page_cache_batch(mapping, &pvec);
+ for (i = 0; i < pagevec_count(&pvec); i++)
+ unlock_page(pvec.pages[i]);
pagevec_release(&pvec);
cond_resched();
- index++;
}
+
if (partial_start) {
struct page *page = find_lock_page(mapping, start - 1);
if (page) {
@@ -554,9 +522,7 @@ unsigned long invalidate_mapping_pages(struct address_space *mapping,
int i;
pagevec_init(&pvec);
- while (index <= end && pagevec_lookup_entries(&pvec, mapping, index,
- min(end - index, (pgoff_t)PAGEVEC_SIZE - 1) + 1,
- indices)) {
+ while (find_lock_entries(mapping, index, end, &pvec, indices)) {
for (i = 0; i < pagevec_count(&pvec); i++) {
struct page *page = pvec.pages[i];
@@ -570,39 +536,7 @@ unsigned long invalidate_mapping_pages(struct address_space *mapping,
page);
continue;
}
-
- if (!trylock_page(page))
- continue;
-
- WARN_ON(page_to_index(page) != index);
-
- /* Middle of THP: skip */
- if (PageTransTail(page)) {
- unlock_page(page);
- continue;
- } else if (PageTransHuge(page)) {
- index += HPAGE_PMD_NR - 1;
- i += HPAGE_PMD_NR - 1;
- /*
- * 'end' is in the middle of THP. Don't
- * invalidate the page as the part outside of
- * 'end' could be still useful.
- */
- if (index > end) {
- unlock_page(page);
- continue;
- }
-
- /* Take a pin outside pagevec */
- get_page(page);
-
- /*
- * Drop extra pins before trying to invalidate
- * the huge page.
- */
- pagevec_remove_exceptionals(&pvec);
- pagevec_release(&pvec);
- }
+ index += thp_nr_pages(page) - 1;
ret = invalidate_inode_page(page);
unlock_page(page);
@@ -612,8 +546,6 @@ unsigned long invalidate_mapping_pages(struct address_space *mapping,
*/
if (!ret)
deactivate_file_page(page);
- if (PageTransHuge(page))
- put_page(page);
count += ret;
}
pagevec_remove_exceptionals(&pvec);
--
2.28.0
^ permalink raw reply related [flat|nested] 38+ messages in thread* Re: [PATCH v2 05/12] mm: Add and use find_lock_entries
2020-09-14 13:00 ` [PATCH v2 05/12] mm: Add and use find_lock_entries Matthew Wilcox (Oracle)
@ 2020-09-29 8:58 ` Jan Kara
2020-09-29 12:48 ` Matthew Wilcox
0 siblings, 1 reply; 38+ messages in thread
From: Jan Kara @ 2020-09-29 8:58 UTC (permalink / raw)
To: Matthew Wilcox (Oracle)
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Jan Kara, Yang Shi, Dave Chinner, linux-kernel
On Mon 14-09-20 14:00:35, Matthew Wilcox (Oracle) wrote:
> We have three functions (shmem_undo_range(), truncate_inode_pages_range()
> and invalidate_mapping_pages()) which want exactly this function, so
> add it to filemap.c.
>
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
> diff --git a/mm/shmem.c b/mm/shmem.c
...
> index b65263d9bb67..a73ce8ce28e3 100644
> --- a/mm/shmem.c
> +++ b/mm/shmem.c
> @@ -905,12 +905,8 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
>
> pagevec_init(&pvec);
> index = start;
> - while (index < end) {
> - pvec.nr = find_get_entries(mapping, index,
> - min(end - index, (pgoff_t)PAGEVEC_SIZE),
> - pvec.pages, indices);
> - if (!pvec.nr)
> - break;
> + while (index < end && find_lock_entries(mapping, index, end - 1,
> + &pvec, indices)) {
> for (i = 0; i < pagevec_count(&pvec); i++) {
> struct page *page = pvec.pages[i];
>
> @@ -925,18 +921,10 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
> index, page);
> continue;
> }
> + index += thp_nr_pages(page) - 1;
>
> - VM_BUG_ON_PAGE(page_to_pgoff(page) != index, page);
> -
> - if (!trylock_page(page))
> - continue;
> -
> - if ((!unfalloc || !PageUptodate(page)) &&
> - page_mapping(page) == mapping) {
> - VM_BUG_ON_PAGE(PageWriteback(page), page);
> - if (shmem_punch_compound(page, start, end))
> - truncate_inode_page(mapping, page);
> - }
> + if (!unfalloc || !PageUptodate(page))
> + truncate_inode_page(mapping, page);
Is dropping shmem_punch_compound() really safe? AFAICS it can also call
split_huge_page() which will try to split THP to be able to truncate it.
That being said there's another loop in shmem_undo_range() which will try
again so what you did might make a difference with performance but not much
else. But still it would be good to at least comment about this in the
changelog...
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: [PATCH v2 05/12] mm: Add and use find_lock_entries
2020-09-29 8:58 ` Jan Kara
@ 2020-09-29 12:48 ` Matthew Wilcox
2020-09-30 10:40 ` Jan Kara
0 siblings, 1 reply; 38+ messages in thread
From: Matthew Wilcox @ 2020-09-29 12:48 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Yang Shi, Dave Chinner, linux-kernel
On Tue, Sep 29, 2020 at 10:58:55AM +0200, Jan Kara wrote:
> On Mon 14-09-20 14:00:35, Matthew Wilcox (Oracle) wrote:
> > We have three functions (shmem_undo_range(), truncate_inode_pages_range()
> > and invalidate_mapping_pages()) which want exactly this function, so
> > add it to filemap.c.
> >
> > Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
> > diff --git a/mm/shmem.c b/mm/shmem.c
> ...
> > index b65263d9bb67..a73ce8ce28e3 100644
> > --- a/mm/shmem.c
> > +++ b/mm/shmem.c
> > @@ -905,12 +905,8 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
> >
> > pagevec_init(&pvec);
> > index = start;
> > - while (index < end) {
> > - pvec.nr = find_get_entries(mapping, index,
> > - min(end - index, (pgoff_t)PAGEVEC_SIZE),
> > - pvec.pages, indices);
> > - if (!pvec.nr)
> > - break;
> > + while (index < end && find_lock_entries(mapping, index, end - 1,
> > + &pvec, indices)) {
> > for (i = 0; i < pagevec_count(&pvec); i++) {
> > struct page *page = pvec.pages[i];
> >
> > @@ -925,18 +921,10 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
> > index, page);
> > continue;
> > }
> > + index += thp_nr_pages(page) - 1;
> >
> > - VM_BUG_ON_PAGE(page_to_pgoff(page) != index, page);
> > -
> > - if (!trylock_page(page))
> > - continue;
> > -
> > - if ((!unfalloc || !PageUptodate(page)) &&
> > - page_mapping(page) == mapping) {
> > - VM_BUG_ON_PAGE(PageWriteback(page), page);
> > - if (shmem_punch_compound(page, start, end))
> > - truncate_inode_page(mapping, page);
> > - }
> > + if (!unfalloc || !PageUptodate(page))
> > + truncate_inode_page(mapping, page);
>
> Is dropping shmem_punch_compound() really safe? AFAICS it can also call
> split_huge_page() which will try to split THP to be able to truncate it.
> That being said there's another loop in shmem_undo_range() which will try
> again so what you did might make a difference with performance but not much
> else. But still it would be good to at least comment about this in the
> changelog...
OK, I need to provide better argumentation in the changelog.
shmem_punch_compound() handles partial THPs. By the end of this series,
we handle the partial pages in the next part of the function ... the
part where we're handling partial PAGE_SIZE pages. At this point in
the series, it's safe to remove the shmem_punch_compound() call because
the new find_lock_entries() loop will only return THPs that lie entirely
within the range.
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: [PATCH v2 05/12] mm: Add and use find_lock_entries
2020-09-29 12:48 ` Matthew Wilcox
@ 2020-09-30 10:40 ` Jan Kara
0 siblings, 0 replies; 38+ messages in thread
From: Jan Kara @ 2020-09-30 10:40 UTC (permalink / raw)
To: Matthew Wilcox
Cc: Jan Kara, linux-mm, Andrew Morton, Hugh Dickins,
William Kucharski, Johannes Weiner, Yang Shi, Dave Chinner,
linux-kernel
On Tue 29-09-20 13:48:06, Matthew Wilcox wrote:
> On Tue, Sep 29, 2020 at 10:58:55AM +0200, Jan Kara wrote:
> > On Mon 14-09-20 14:00:35, Matthew Wilcox (Oracle) wrote:
> > > We have three functions (shmem_undo_range(), truncate_inode_pages_range()
> > > and invalidate_mapping_pages()) which want exactly this function, so
> > > add it to filemap.c.
> > >
> > > Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
> > > diff --git a/mm/shmem.c b/mm/shmem.c
> > ...
> > > index b65263d9bb67..a73ce8ce28e3 100644
> > > --- a/mm/shmem.c
> > > +++ b/mm/shmem.c
> > > @@ -905,12 +905,8 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
> > >
> > > pagevec_init(&pvec);
> > > index = start;
> > > - while (index < end) {
> > > - pvec.nr = find_get_entries(mapping, index,
> > > - min(end - index, (pgoff_t)PAGEVEC_SIZE),
> > > - pvec.pages, indices);
> > > - if (!pvec.nr)
> > > - break;
> > > + while (index < end && find_lock_entries(mapping, index, end - 1,
> > > + &pvec, indices)) {
> > > for (i = 0; i < pagevec_count(&pvec); i++) {
> > > struct page *page = pvec.pages[i];
> > >
> > > @@ -925,18 +921,10 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
> > > index, page);
> > > continue;
> > > }
> > > + index += thp_nr_pages(page) - 1;
> > >
> > > - VM_BUG_ON_PAGE(page_to_pgoff(page) != index, page);
> > > -
> > > - if (!trylock_page(page))
> > > - continue;
> > > -
> > > - if ((!unfalloc || !PageUptodate(page)) &&
> > > - page_mapping(page) == mapping) {
> > > - VM_BUG_ON_PAGE(PageWriteback(page), page);
> > > - if (shmem_punch_compound(page, start, end))
> > > - truncate_inode_page(mapping, page);
> > > - }
> > > + if (!unfalloc || !PageUptodate(page))
> > > + truncate_inode_page(mapping, page);
> >
> > Is dropping shmem_punch_compound() really safe? AFAICS it can also call
> > split_huge_page() which will try to split THP to be able to truncate it.
> > That being said there's another loop in shmem_undo_range() which will try
> > again so what you did might make a difference with performance but not much
> > else. But still it would be good to at least comment about this in the
> > changelog...
>
> OK, I need to provide better argumentation in the changelog.
>
> shmem_punch_compound() handles partial THPs. By the end of this series,
> we handle the partial pages in the next part of the function ... the
> part where we're handling partial PAGE_SIZE pages. At this point in
> the series, it's safe to remove the shmem_punch_compound() call because
> the new find_lock_entries() loop will only return THPs that lie entirely
> within the range.
Yes, plus transitioning the first loop in shmem_undo_range() to
find_lock_entries() which skips partial THPs is safe at this point in the
series because the second loop in find_lock_entries() still uses
find_get_entries() and shmem_punch_compound() and so properly treats
partial THPs.
Anyway, I'm now convinced the patch is fine so feel free to add:
Reviewed-by: Jan Kara <jack@suse.cz>
after expanding the changelog.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 38+ messages in thread
* [PATCH v2 06/12] mm: Add an 'end' parameter to find_get_entries
2020-09-14 13:00 [PATCH v2 00/12] Overhaul multi-page lookups for THP Matthew Wilcox (Oracle)
` (4 preceding siblings ...)
2020-09-14 13:00 ` [PATCH v2 05/12] mm: Add and use find_lock_entries Matthew Wilcox (Oracle)
@ 2020-09-14 13:00 ` Matthew Wilcox (Oracle)
2020-09-14 13:00 ` [PATCH v2 07/12] mm: Add an 'end' parameter to pagevec_lookup_entries Matthew Wilcox (Oracle)
` (7 subsequent siblings)
13 siblings, 0 replies; 38+ messages in thread
From: Matthew Wilcox (Oracle) @ 2020-09-14 13:00 UTC (permalink / raw)
To: linux-mm
Cc: Matthew Wilcox (Oracle),
Andrew Morton, Hugh Dickins, William Kucharski, Johannes Weiner,
Jan Kara, Yang Shi, Dave Chinner, linux-kernel
This simplifies the callers and leads to a more efficient implementation
since the XArray has this functionality already.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: William Kucharski <william.kucharski@oracle.com>
---
include/linux/pagemap.h | 4 ++--
mm/filemap.c | 9 +++++----
mm/shmem.c | 10 ++--------
mm/swap.c | 2 +-
4 files changed, 10 insertions(+), 15 deletions(-)
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 869dc371b800..d440c6750757 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -414,8 +414,8 @@ static inline struct page *find_subpage(struct page *head, pgoff_t index)
}
unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
- unsigned int nr_entries, struct page **entries,
- pgoff_t *indices);
+ pgoff_t end, unsigned int nr_entries, struct page **entries,
+ pgoff_t *indices);
unsigned find_get_pages_range(struct address_space *mapping, pgoff_t *start,
pgoff_t end, unsigned int nr_pages,
struct page **pages);
diff --git a/mm/filemap.c b/mm/filemap.c
index 9f7f6b46aee4..6dc0a9b8c0fa 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1799,6 +1799,7 @@ static inline struct page *xas_find_get_entry(struct xa_state *xas,
* find_get_entries - gang pagecache lookup
* @mapping: The address_space to search
* @start: The starting page cache index
+ * @end: The final page index (inclusive).
* @nr_entries: The maximum number of entries
* @entries: Where the resulting entries are placed
* @indices: The cache indices corresponding to the entries in @entries
@@ -1822,9 +1823,9 @@ static inline struct page *xas_find_get_entry(struct xa_state *xas,
*
* Return: the number of pages and shadow entries which were found.
*/
-unsigned find_get_entries(struct address_space *mapping,
- pgoff_t start, unsigned int nr_entries,
- struct page **entries, pgoff_t *indices)
+unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
+ pgoff_t end, unsigned int nr_entries, struct page **entries,
+ pgoff_t *indices)
{
XA_STATE(xas, &mapping->i_pages, start);
struct page *page;
@@ -1834,7 +1835,7 @@ unsigned find_get_entries(struct address_space *mapping,
return 0;
rcu_read_lock();
- while ((page = xas_find_get_entry(&xas, ULONG_MAX, XA_PRESENT))) {
+ while ((page = xas_find_get_entry(&xas, end, XA_PRESENT))) {
/*
* Terminate early on finding a THP, to allow the caller to
* handle it all at once; but continue if this is hugetlbfs.
diff --git a/mm/shmem.c b/mm/shmem.c
index a73ce8ce28e3..404e45c285ca 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -911,8 +911,6 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
struct page *page = pvec.pages[i];
index = indices[i];
- if (index >= end)
- break;
if (xa_is_value(page)) {
if (unfalloc)
@@ -965,9 +963,8 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
while (index < end) {
cond_resched();
- pvec.nr = find_get_entries(mapping, index,
- min(end - index, (pgoff_t)PAGEVEC_SIZE),
- pvec.pages, indices);
+ pvec.nr = find_get_entries(mapping, index, end - 1,
+ PAGEVEC_SIZE, pvec.pages, indices);
if (!pvec.nr) {
/* If all gone or hole-punch or unfalloc, we're done */
if (index == start || end != -1)
@@ -980,9 +977,6 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
struct page *page = pvec.pages[i];
index = indices[i];
- if (index >= end)
- break;
-
if (xa_is_value(page)) {
if (unfalloc)
continue;
diff --git a/mm/swap.c b/mm/swap.c
index d16d65d9b4e0..fcf6ccb94b09 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -1060,7 +1060,7 @@ unsigned pagevec_lookup_entries(struct pagevec *pvec,
pgoff_t start, unsigned nr_entries,
pgoff_t *indices)
{
- pvec->nr = find_get_entries(mapping, start, nr_entries,
+ pvec->nr = find_get_entries(mapping, start, ULONG_MAX, nr_entries,
pvec->pages, indices);
return pagevec_count(pvec);
}
--
2.28.0
^ permalink raw reply related [flat|nested] 38+ messages in thread* [PATCH v2 07/12] mm: Add an 'end' parameter to pagevec_lookup_entries
2020-09-14 13:00 [PATCH v2 00/12] Overhaul multi-page lookups for THP Matthew Wilcox (Oracle)
` (5 preceding siblings ...)
2020-09-14 13:00 ` [PATCH v2 06/12] mm: Add an 'end' parameter to find_get_entries Matthew Wilcox (Oracle)
@ 2020-09-14 13:00 ` Matthew Wilcox (Oracle)
2020-09-29 9:02 ` Jan Kara
2020-09-14 13:00 ` [PATCH v2 08/12] mm: Remove nr_entries parameter from pagevec_lookup_entries Matthew Wilcox (Oracle)
` (6 subsequent siblings)
13 siblings, 1 reply; 38+ messages in thread
From: Matthew Wilcox (Oracle) @ 2020-09-14 13:00 UTC (permalink / raw)
To: linux-mm
Cc: Matthew Wilcox (Oracle),
Andrew Morton, Hugh Dickins, William Kucharski, Johannes Weiner,
Jan Kara, Yang Shi, Dave Chinner, linux-kernel
Simplifies the callers and uses the existing functionality
in find_get_entries(). We can also drop the final argument of
truncate_exceptional_pvec_entries() and simplify the logic in that
function.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
include/linux/pagevec.h | 5 ++---
mm/swap.c | 8 ++++----
mm/truncate.c | 41 ++++++++++-------------------------------
3 files changed, 16 insertions(+), 38 deletions(-)
diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h
index 081d934eda64..4b245592262c 100644
--- a/include/linux/pagevec.h
+++ b/include/linux/pagevec.h
@@ -26,9 +26,8 @@ struct pagevec {
void __pagevec_release(struct pagevec *pvec);
void __pagevec_lru_add(struct pagevec *pvec);
unsigned pagevec_lookup_entries(struct pagevec *pvec,
- struct address_space *mapping,
- pgoff_t start, unsigned nr_entries,
- pgoff_t *indices);
+ struct address_space *mapping, pgoff_t start, pgoff_t end,
+ unsigned nr_entries, pgoff_t *indices);
void pagevec_remove_exceptionals(struct pagevec *pvec);
unsigned pagevec_lookup_range(struct pagevec *pvec,
struct address_space *mapping,
diff --git a/mm/swap.c b/mm/swap.c
index fcf6ccb94b09..b6e56a84b466 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -1036,6 +1036,7 @@ void __pagevec_lru_add(struct pagevec *pvec)
* @pvec: Where the resulting entries are placed
* @mapping: The address_space to search
* @start: The starting entry index
+ * @end: The highest index to return (inclusive).
* @nr_entries: The maximum number of pages
* @indices: The cache indices corresponding to the entries in @pvec
*
@@ -1056,11 +1057,10 @@ void __pagevec_lru_add(struct pagevec *pvec)
* found.
*/
unsigned pagevec_lookup_entries(struct pagevec *pvec,
- struct address_space *mapping,
- pgoff_t start, unsigned nr_entries,
- pgoff_t *indices)
+ struct address_space *mapping, pgoff_t start, pgoff_t end,
+ unsigned nr_entries, pgoff_t *indices)
{
- pvec->nr = find_get_entries(mapping, start, ULONG_MAX, nr_entries,
+ pvec->nr = find_get_entries(mapping, start, end, nr_entries,
pvec->pages, indices);
return pagevec_count(pvec);
}
diff --git a/mm/truncate.c b/mm/truncate.c
index 5dbe0c77b5ac..69ea72e7fc1c 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -57,11 +57,10 @@ static void clear_shadow_entry(struct address_space *mapping, pgoff_t index,
* exceptional entries similar to what pagevec_remove_exceptionals does.
*/
static void truncate_exceptional_pvec_entries(struct address_space *mapping,
- struct pagevec *pvec, pgoff_t *indices,
- pgoff_t end)
+ struct pagevec *pvec, pgoff_t *indices)
{
int i, j;
- bool dax, lock;
+ bool dax;
/* Handled by shmem itself */
if (shmem_mapping(mapping))
@@ -75,8 +74,7 @@ static void truncate_exceptional_pvec_entries(struct address_space *mapping,
return;
dax = dax_mapping(mapping);
- lock = !dax && indices[j] < end;
- if (lock)
+ if (!dax)
xa_lock_irq(&mapping->i_pages);
for (i = j; i < pagevec_count(pvec); i++) {
@@ -88,9 +86,6 @@ static void truncate_exceptional_pvec_entries(struct address_space *mapping,
continue;
}
- if (index >= end)
- continue;
-
if (unlikely(dax)) {
dax_delete_mapping_entry(mapping, index);
continue;
@@ -99,7 +94,7 @@ static void truncate_exceptional_pvec_entries(struct address_space *mapping,
__clear_shadow_entry(mapping, index, page);
}
- if (lock)
+ if (!dax)
xa_unlock_irq(&mapping->i_pages);
pvec->nr = j;
}
@@ -329,7 +324,7 @@ void truncate_inode_pages_range(struct address_space *mapping,
while (index < end && find_lock_entries(mapping, index, end - 1,
&pvec, indices)) {
index = indices[pagevec_count(&pvec) - 1] + 1;
- truncate_exceptional_pvec_entries(mapping, &pvec, indices, end);
+ truncate_exceptional_pvec_entries(mapping, &pvec, indices);
for (i = 0; i < pagevec_count(&pvec); i++)
truncate_cleanup_page(mapping, pvec.pages[i]);
delete_from_page_cache_batch(mapping, &pvec);
@@ -381,8 +376,8 @@ void truncate_inode_pages_range(struct address_space *mapping,
index = start;
for ( ; ; ) {
cond_resched();
- if (!pagevec_lookup_entries(&pvec, mapping, index,
- min(end - index, (pgoff_t)PAGEVEC_SIZE), indices)) {
+ if (!pagevec_lookup_entries(&pvec, mapping, index, end - 1,
+ PAGEVEC_SIZE, indices)) {
/* If all gone from start onwards, we're done */
if (index == start)
break;
@@ -390,23 +385,12 @@ void truncate_inode_pages_range(struct address_space *mapping,
index = start;
continue;
}
- if (index == start && indices[0] >= end) {
- /* All gone out of hole to be punched, we're done */
- pagevec_remove_exceptionals(&pvec);
- pagevec_release(&pvec);
- break;
- }
for (i = 0; i < pagevec_count(&pvec); i++) {
struct page *page = pvec.pages[i];
/* We rely upon deletion not changing page->index */
index = indices[i];
- if (index >= end) {
- /* Restart punch to make sure all gone */
- index = start - 1;
- break;
- }
if (xa_is_value(page))
continue;
@@ -417,7 +401,7 @@ void truncate_inode_pages_range(struct address_space *mapping,
truncate_inode_page(mapping, page);
unlock_page(page);
}
- truncate_exceptional_pvec_entries(mapping, &pvec, indices, end);
+ truncate_exceptional_pvec_entries(mapping, &pvec, indices);
pagevec_release(&pvec);
index++;
}
@@ -528,8 +512,6 @@ unsigned long invalidate_mapping_pages(struct address_space *mapping,
/* We rely upon deletion not changing page->index */
index = indices[i];
- if (index > end)
- break;
if (xa_is_value(page)) {
invalidate_exceptional_entry(mapping, index,
@@ -629,16 +611,13 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
pagevec_init(&pvec);
index = start;
- while (index <= end && pagevec_lookup_entries(&pvec, mapping, index,
- min(end - index, (pgoff_t)PAGEVEC_SIZE - 1) + 1,
- indices)) {
+ while (pagevec_lookup_entries(&pvec, mapping, index, end,
+ PAGEVEC_SIZE, indices)) {
for (i = 0; i < pagevec_count(&pvec); i++) {
struct page *page = pvec.pages[i];
/* We rely upon deletion not changing page->index */
index = indices[i];
- if (index > end)
- break;
if (xa_is_value(page)) {
if (!invalidate_exceptional_entry2(mapping,
--
2.28.0
^ permalink raw reply related [flat|nested] 38+ messages in thread* Re: [PATCH v2 07/12] mm: Add an 'end' parameter to pagevec_lookup_entries
2020-09-14 13:00 ` [PATCH v2 07/12] mm: Add an 'end' parameter to pagevec_lookup_entries Matthew Wilcox (Oracle)
@ 2020-09-29 9:02 ` Jan Kara
0 siblings, 0 replies; 38+ messages in thread
From: Jan Kara @ 2020-09-29 9:02 UTC (permalink / raw)
To: Matthew Wilcox (Oracle)
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Jan Kara, Yang Shi, Dave Chinner, linux-kernel
On Mon 14-09-20 14:00:37, Matthew Wilcox (Oracle) wrote:
> Simplifies the callers and uses the existing functionality
> in find_get_entries(). We can also drop the final argument of
> truncate_exceptional_pvec_entries() and simplify the logic in that
> function.
>
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Looks good to me. You can add:
Reviewed-by: Jan Kara <jack@suse.cz>
Honza
> ---
> include/linux/pagevec.h | 5 ++---
> mm/swap.c | 8 ++++----
> mm/truncate.c | 41 ++++++++++-------------------------------
> 3 files changed, 16 insertions(+), 38 deletions(-)
>
> diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h
> index 081d934eda64..4b245592262c 100644
> --- a/include/linux/pagevec.h
> +++ b/include/linux/pagevec.h
> @@ -26,9 +26,8 @@ struct pagevec {
> void __pagevec_release(struct pagevec *pvec);
> void __pagevec_lru_add(struct pagevec *pvec);
> unsigned pagevec_lookup_entries(struct pagevec *pvec,
> - struct address_space *mapping,
> - pgoff_t start, unsigned nr_entries,
> - pgoff_t *indices);
> + struct address_space *mapping, pgoff_t start, pgoff_t end,
> + unsigned nr_entries, pgoff_t *indices);
> void pagevec_remove_exceptionals(struct pagevec *pvec);
> unsigned pagevec_lookup_range(struct pagevec *pvec,
> struct address_space *mapping,
> diff --git a/mm/swap.c b/mm/swap.c
> index fcf6ccb94b09..b6e56a84b466 100644
> --- a/mm/swap.c
> +++ b/mm/swap.c
> @@ -1036,6 +1036,7 @@ void __pagevec_lru_add(struct pagevec *pvec)
> * @pvec: Where the resulting entries are placed
> * @mapping: The address_space to search
> * @start: The starting entry index
> + * @end: The highest index to return (inclusive).
> * @nr_entries: The maximum number of pages
> * @indices: The cache indices corresponding to the entries in @pvec
> *
> @@ -1056,11 +1057,10 @@ void __pagevec_lru_add(struct pagevec *pvec)
> * found.
> */
> unsigned pagevec_lookup_entries(struct pagevec *pvec,
> - struct address_space *mapping,
> - pgoff_t start, unsigned nr_entries,
> - pgoff_t *indices)
> + struct address_space *mapping, pgoff_t start, pgoff_t end,
> + unsigned nr_entries, pgoff_t *indices)
> {
> - pvec->nr = find_get_entries(mapping, start, ULONG_MAX, nr_entries,
> + pvec->nr = find_get_entries(mapping, start, end, nr_entries,
> pvec->pages, indices);
> return pagevec_count(pvec);
> }
> diff --git a/mm/truncate.c b/mm/truncate.c
> index 5dbe0c77b5ac..69ea72e7fc1c 100644
> --- a/mm/truncate.c
> +++ b/mm/truncate.c
> @@ -57,11 +57,10 @@ static void clear_shadow_entry(struct address_space *mapping, pgoff_t index,
> * exceptional entries similar to what pagevec_remove_exceptionals does.
> */
> static void truncate_exceptional_pvec_entries(struct address_space *mapping,
> - struct pagevec *pvec, pgoff_t *indices,
> - pgoff_t end)
> + struct pagevec *pvec, pgoff_t *indices)
> {
> int i, j;
> - bool dax, lock;
> + bool dax;
>
> /* Handled by shmem itself */
> if (shmem_mapping(mapping))
> @@ -75,8 +74,7 @@ static void truncate_exceptional_pvec_entries(struct address_space *mapping,
> return;
>
> dax = dax_mapping(mapping);
> - lock = !dax && indices[j] < end;
> - if (lock)
> + if (!dax)
> xa_lock_irq(&mapping->i_pages);
>
> for (i = j; i < pagevec_count(pvec); i++) {
> @@ -88,9 +86,6 @@ static void truncate_exceptional_pvec_entries(struct address_space *mapping,
> continue;
> }
>
> - if (index >= end)
> - continue;
> -
> if (unlikely(dax)) {
> dax_delete_mapping_entry(mapping, index);
> continue;
> @@ -99,7 +94,7 @@ static void truncate_exceptional_pvec_entries(struct address_space *mapping,
> __clear_shadow_entry(mapping, index, page);
> }
>
> - if (lock)
> + if (!dax)
> xa_unlock_irq(&mapping->i_pages);
> pvec->nr = j;
> }
> @@ -329,7 +324,7 @@ void truncate_inode_pages_range(struct address_space *mapping,
> while (index < end && find_lock_entries(mapping, index, end - 1,
> &pvec, indices)) {
> index = indices[pagevec_count(&pvec) - 1] + 1;
> - truncate_exceptional_pvec_entries(mapping, &pvec, indices, end);
> + truncate_exceptional_pvec_entries(mapping, &pvec, indices);
> for (i = 0; i < pagevec_count(&pvec); i++)
> truncate_cleanup_page(mapping, pvec.pages[i]);
> delete_from_page_cache_batch(mapping, &pvec);
> @@ -381,8 +376,8 @@ void truncate_inode_pages_range(struct address_space *mapping,
> index = start;
> for ( ; ; ) {
> cond_resched();
> - if (!pagevec_lookup_entries(&pvec, mapping, index,
> - min(end - index, (pgoff_t)PAGEVEC_SIZE), indices)) {
> + if (!pagevec_lookup_entries(&pvec, mapping, index, end - 1,
> + PAGEVEC_SIZE, indices)) {
> /* If all gone from start onwards, we're done */
> if (index == start)
> break;
> @@ -390,23 +385,12 @@ void truncate_inode_pages_range(struct address_space *mapping,
> index = start;
> continue;
> }
> - if (index == start && indices[0] >= end) {
> - /* All gone out of hole to be punched, we're done */
> - pagevec_remove_exceptionals(&pvec);
> - pagevec_release(&pvec);
> - break;
> - }
>
> for (i = 0; i < pagevec_count(&pvec); i++) {
> struct page *page = pvec.pages[i];
>
> /* We rely upon deletion not changing page->index */
> index = indices[i];
> - if (index >= end) {
> - /* Restart punch to make sure all gone */
> - index = start - 1;
> - break;
> - }
>
> if (xa_is_value(page))
> continue;
> @@ -417,7 +401,7 @@ void truncate_inode_pages_range(struct address_space *mapping,
> truncate_inode_page(mapping, page);
> unlock_page(page);
> }
> - truncate_exceptional_pvec_entries(mapping, &pvec, indices, end);
> + truncate_exceptional_pvec_entries(mapping, &pvec, indices);
> pagevec_release(&pvec);
> index++;
> }
> @@ -528,8 +512,6 @@ unsigned long invalidate_mapping_pages(struct address_space *mapping,
>
> /* We rely upon deletion not changing page->index */
> index = indices[i];
> - if (index > end)
> - break;
>
> if (xa_is_value(page)) {
> invalidate_exceptional_entry(mapping, index,
> @@ -629,16 +611,13 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
>
> pagevec_init(&pvec);
> index = start;
> - while (index <= end && pagevec_lookup_entries(&pvec, mapping, index,
> - min(end - index, (pgoff_t)PAGEVEC_SIZE - 1) + 1,
> - indices)) {
> + while (pagevec_lookup_entries(&pvec, mapping, index, end,
> + PAGEVEC_SIZE, indices)) {
> for (i = 0; i < pagevec_count(&pvec); i++) {
> struct page *page = pvec.pages[i];
>
> /* We rely upon deletion not changing page->index */
> index = indices[i];
> - if (index > end)
> - break;
>
> if (xa_is_value(page)) {
> if (!invalidate_exceptional_entry2(mapping,
> --
> 2.28.0
>
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 38+ messages in thread
* [PATCH v2 08/12] mm: Remove nr_entries parameter from pagevec_lookup_entries
2020-09-14 13:00 [PATCH v2 00/12] Overhaul multi-page lookups for THP Matthew Wilcox (Oracle)
` (6 preceding siblings ...)
2020-09-14 13:00 ` [PATCH v2 07/12] mm: Add an 'end' parameter to pagevec_lookup_entries Matthew Wilcox (Oracle)
@ 2020-09-14 13:00 ` Matthew Wilcox (Oracle)
2020-09-29 9:03 ` Jan Kara
2020-09-14 13:00 ` [PATCH v2 09/12] mm: Pass pvec directly to find_get_entries Matthew Wilcox (Oracle)
` (5 subsequent siblings)
13 siblings, 1 reply; 38+ messages in thread
From: Matthew Wilcox (Oracle) @ 2020-09-14 13:00 UTC (permalink / raw)
To: linux-mm
Cc: Matthew Wilcox (Oracle),
Andrew Morton, Hugh Dickins, William Kucharski, Johannes Weiner,
Jan Kara, Yang Shi, Dave Chinner, linux-kernel
All callers want to fetch the full size of the pvec.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
include/linux/pagevec.h | 2 +-
mm/swap.c | 4 ++--
mm/truncate.c | 5 ++---
3 files changed, 5 insertions(+), 6 deletions(-)
diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h
index 4b245592262c..ce77724a2ab7 100644
--- a/include/linux/pagevec.h
+++ b/include/linux/pagevec.h
@@ -27,7 +27,7 @@ void __pagevec_release(struct pagevec *pvec);
void __pagevec_lru_add(struct pagevec *pvec);
unsigned pagevec_lookup_entries(struct pagevec *pvec,
struct address_space *mapping, pgoff_t start, pgoff_t end,
- unsigned nr_entries, pgoff_t *indices);
+ pgoff_t *indices);
void pagevec_remove_exceptionals(struct pagevec *pvec);
unsigned pagevec_lookup_range(struct pagevec *pvec,
struct address_space *mapping,
diff --git a/mm/swap.c b/mm/swap.c
index b6e56a84b466..d4e3ba4c967c 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -1058,9 +1058,9 @@ void __pagevec_lru_add(struct pagevec *pvec)
*/
unsigned pagevec_lookup_entries(struct pagevec *pvec,
struct address_space *mapping, pgoff_t start, pgoff_t end,
- unsigned nr_entries, pgoff_t *indices)
+ pgoff_t *indices)
{
- pvec->nr = find_get_entries(mapping, start, end, nr_entries,
+ pvec->nr = find_get_entries(mapping, start, end, PAGEVEC_SIZE,
pvec->pages, indices);
return pagevec_count(pvec);
}
diff --git a/mm/truncate.c b/mm/truncate.c
index 69ea72e7fc1c..bbd0a02eaed7 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -377,7 +377,7 @@ void truncate_inode_pages_range(struct address_space *mapping,
for ( ; ; ) {
cond_resched();
if (!pagevec_lookup_entries(&pvec, mapping, index, end - 1,
- PAGEVEC_SIZE, indices)) {
+ indices)) {
/* If all gone from start onwards, we're done */
if (index == start)
break;
@@ -611,8 +611,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
pagevec_init(&pvec);
index = start;
- while (pagevec_lookup_entries(&pvec, mapping, index, end,
- PAGEVEC_SIZE, indices)) {
+ while (pagevec_lookup_entries(&pvec, mapping, index, end, indices)) {
for (i = 0; i < pagevec_count(&pvec); i++) {
struct page *page = pvec.pages[i];
--
2.28.0
^ permalink raw reply related [flat|nested] 38+ messages in thread* Re: [PATCH v2 08/12] mm: Remove nr_entries parameter from pagevec_lookup_entries
2020-09-14 13:00 ` [PATCH v2 08/12] mm: Remove nr_entries parameter from pagevec_lookup_entries Matthew Wilcox (Oracle)
@ 2020-09-29 9:03 ` Jan Kara
0 siblings, 0 replies; 38+ messages in thread
From: Jan Kara @ 2020-09-29 9:03 UTC (permalink / raw)
To: Matthew Wilcox (Oracle)
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Jan Kara, Yang Shi, Dave Chinner, linux-kernel
On Mon 14-09-20 14:00:38, Matthew Wilcox (Oracle) wrote:
> All callers want to fetch the full size of the pvec.
>
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Looks good. You can add:
Reviewed-by: Jan Kara <jack@suse.cz>
Honza
> ---
> include/linux/pagevec.h | 2 +-
> mm/swap.c | 4 ++--
> mm/truncate.c | 5 ++---
> 3 files changed, 5 insertions(+), 6 deletions(-)
>
> diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h
> index 4b245592262c..ce77724a2ab7 100644
> --- a/include/linux/pagevec.h
> +++ b/include/linux/pagevec.h
> @@ -27,7 +27,7 @@ void __pagevec_release(struct pagevec *pvec);
> void __pagevec_lru_add(struct pagevec *pvec);
> unsigned pagevec_lookup_entries(struct pagevec *pvec,
> struct address_space *mapping, pgoff_t start, pgoff_t end,
> - unsigned nr_entries, pgoff_t *indices);
> + pgoff_t *indices);
> void pagevec_remove_exceptionals(struct pagevec *pvec);
> unsigned pagevec_lookup_range(struct pagevec *pvec,
> struct address_space *mapping,
> diff --git a/mm/swap.c b/mm/swap.c
> index b6e56a84b466..d4e3ba4c967c 100644
> --- a/mm/swap.c
> +++ b/mm/swap.c
> @@ -1058,9 +1058,9 @@ void __pagevec_lru_add(struct pagevec *pvec)
> */
> unsigned pagevec_lookup_entries(struct pagevec *pvec,
> struct address_space *mapping, pgoff_t start, pgoff_t end,
> - unsigned nr_entries, pgoff_t *indices)
> + pgoff_t *indices)
> {
> - pvec->nr = find_get_entries(mapping, start, end, nr_entries,
> + pvec->nr = find_get_entries(mapping, start, end, PAGEVEC_SIZE,
> pvec->pages, indices);
> return pagevec_count(pvec);
> }
> diff --git a/mm/truncate.c b/mm/truncate.c
> index 69ea72e7fc1c..bbd0a02eaed7 100644
> --- a/mm/truncate.c
> +++ b/mm/truncate.c
> @@ -377,7 +377,7 @@ void truncate_inode_pages_range(struct address_space *mapping,
> for ( ; ; ) {
> cond_resched();
> if (!pagevec_lookup_entries(&pvec, mapping, index, end - 1,
> - PAGEVEC_SIZE, indices)) {
> + indices)) {
> /* If all gone from start onwards, we're done */
> if (index == start)
> break;
> @@ -611,8 +611,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
>
> pagevec_init(&pvec);
> index = start;
> - while (pagevec_lookup_entries(&pvec, mapping, index, end,
> - PAGEVEC_SIZE, indices)) {
> + while (pagevec_lookup_entries(&pvec, mapping, index, end, indices)) {
> for (i = 0; i < pagevec_count(&pvec); i++) {
> struct page *page = pvec.pages[i];
>
> --
> 2.28.0
>
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 38+ messages in thread
* [PATCH v2 09/12] mm: Pass pvec directly to find_get_entries
2020-09-14 13:00 [PATCH v2 00/12] Overhaul multi-page lookups for THP Matthew Wilcox (Oracle)
` (7 preceding siblings ...)
2020-09-14 13:00 ` [PATCH v2 08/12] mm: Remove nr_entries parameter from pagevec_lookup_entries Matthew Wilcox (Oracle)
@ 2020-09-14 13:00 ` Matthew Wilcox (Oracle)
2020-09-29 9:07 ` Jan Kara
2020-09-14 13:00 ` [PATCH v2 10/12] mm: Remove pagevec_lookup_entries Matthew Wilcox (Oracle)
` (4 subsequent siblings)
13 siblings, 1 reply; 38+ messages in thread
From: Matthew Wilcox (Oracle) @ 2020-09-14 13:00 UTC (permalink / raw)
To: linux-mm
Cc: Matthew Wilcox (Oracle),
Andrew Morton, Hugh Dickins, William Kucharski, Johannes Weiner,
Jan Kara, Yang Shi, Dave Chinner, linux-kernel
All callers of find_get_entries() use a pvec, so pass it directly
instead of manipulating it in the caller.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
include/linux/pagemap.h | 3 +--
mm/filemap.c | 21 +++++++++------------
mm/shmem.c | 5 ++---
mm/swap.c | 4 +---
4 files changed, 13 insertions(+), 20 deletions(-)
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index d440c6750757..788c280737e0 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -414,8 +414,7 @@ static inline struct page *find_subpage(struct page *head, pgoff_t index)
}
unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
- pgoff_t end, unsigned int nr_entries, struct page **entries,
- pgoff_t *indices);
+ pgoff_t end, struct pagevec *pvec, pgoff_t *indices);
unsigned find_get_pages_range(struct address_space *mapping, pgoff_t *start,
pgoff_t end, unsigned int nr_pages,
struct page **pages);
diff --git a/mm/filemap.c b/mm/filemap.c
index 6dc0a9b8c0fa..a4ad294a34dc 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1800,14 +1800,12 @@ static inline struct page *xas_find_get_entry(struct xa_state *xas,
* @mapping: The address_space to search
* @start: The starting page cache index
* @end: The final page index (inclusive).
- * @nr_entries: The maximum number of entries
- * @entries: Where the resulting entries are placed
+ * @pvec: Where the resulting entries are placed.
* @indices: The cache indices corresponding to the entries in @entries
*
- * find_get_entries() will search for and return a group of up to
- * @nr_entries entries in the mapping. The entries are placed at
- * @entries. find_get_entries() takes a reference against any actual
- * pages it returns.
+ * find_get_entries() will search for and return a batch of entries in
+ * the mapping. The entries are placed in @pvec. find_get_entries()
+ * takes a reference on any actual pages it returns.
*
* The search returns a group of mapping-contiguous page cache entries
* with ascending indexes. There may be holes in the indices due to
@@ -1824,15 +1822,12 @@ static inline struct page *xas_find_get_entry(struct xa_state *xas,
* Return: the number of pages and shadow entries which were found.
*/
unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
- pgoff_t end, unsigned int nr_entries, struct page **entries,
- pgoff_t *indices)
+ pgoff_t end, struct pagevec *pvec, pgoff_t *indices)
{
XA_STATE(xas, &mapping->i_pages, start);
struct page *page;
unsigned int ret = 0;
-
- if (!nr_entries)
- return 0;
+ unsigned nr_entries = PAGEVEC_SIZE;
rcu_read_lock();
while ((page = xas_find_get_entry(&xas, end, XA_PRESENT))) {
@@ -1847,11 +1842,13 @@ unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
}
indices[ret] = xas.xa_index;
- entries[ret] = page;
+ pvec->pages[ret] = page;
if (++ret == nr_entries)
break;
}
rcu_read_unlock();
+
+ pvec->nr = ret;
return ret;
}
diff --git a/mm/shmem.c b/mm/shmem.c
index 404e45c285ca..b2b94025841a 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -963,9 +963,8 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
while (index < end) {
cond_resched();
- pvec.nr = find_get_entries(mapping, index, end - 1,
- PAGEVEC_SIZE, pvec.pages, indices);
- if (!pvec.nr) {
+ if (!find_get_entries(mapping, index, end - 1, &pvec,
+ indices)) {
/* If all gone or hole-punch or unfalloc, we're done */
if (index == start || end != -1)
break;
diff --git a/mm/swap.c b/mm/swap.c
index d4e3ba4c967c..40b23300d353 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -1060,9 +1060,7 @@ unsigned pagevec_lookup_entries(struct pagevec *pvec,
struct address_space *mapping, pgoff_t start, pgoff_t end,
pgoff_t *indices)
{
- pvec->nr = find_get_entries(mapping, start, end, PAGEVEC_SIZE,
- pvec->pages, indices);
- return pagevec_count(pvec);
+ return find_get_entries(mapping, start, end, pvec, indices);
}
/**
--
2.28.0
^ permalink raw reply related [flat|nested] 38+ messages in thread* Re: [PATCH v2 09/12] mm: Pass pvec directly to find_get_entries
2020-09-14 13:00 ` [PATCH v2 09/12] mm: Pass pvec directly to find_get_entries Matthew Wilcox (Oracle)
@ 2020-09-29 9:07 ` Jan Kara
0 siblings, 0 replies; 38+ messages in thread
From: Jan Kara @ 2020-09-29 9:07 UTC (permalink / raw)
To: Matthew Wilcox (Oracle)
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Jan Kara, Yang Shi, Dave Chinner, linux-kernel
On Mon 14-09-20 14:00:39, Matthew Wilcox (Oracle) wrote:
> All callers of find_get_entries() use a pvec, so pass it directly
> instead of manipulating it in the caller.
>
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Looks good. You can add:
Reviewed-by: Jan Kara <jack@suse.cz>
Honza
> ---
> include/linux/pagemap.h | 3 +--
> mm/filemap.c | 21 +++++++++------------
> mm/shmem.c | 5 ++---
> mm/swap.c | 4 +---
> 4 files changed, 13 insertions(+), 20 deletions(-)
>
> diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
> index d440c6750757..788c280737e0 100644
> --- a/include/linux/pagemap.h
> +++ b/include/linux/pagemap.h
> @@ -414,8 +414,7 @@ static inline struct page *find_subpage(struct page *head, pgoff_t index)
> }
>
> unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
> - pgoff_t end, unsigned int nr_entries, struct page **entries,
> - pgoff_t *indices);
> + pgoff_t end, struct pagevec *pvec, pgoff_t *indices);
> unsigned find_get_pages_range(struct address_space *mapping, pgoff_t *start,
> pgoff_t end, unsigned int nr_pages,
> struct page **pages);
> diff --git a/mm/filemap.c b/mm/filemap.c
> index 6dc0a9b8c0fa..a4ad294a34dc 100644
> --- a/mm/filemap.c
> +++ b/mm/filemap.c
> @@ -1800,14 +1800,12 @@ static inline struct page *xas_find_get_entry(struct xa_state *xas,
> * @mapping: The address_space to search
> * @start: The starting page cache index
> * @end: The final page index (inclusive).
> - * @nr_entries: The maximum number of entries
> - * @entries: Where the resulting entries are placed
> + * @pvec: Where the resulting entries are placed.
> * @indices: The cache indices corresponding to the entries in @entries
> *
> - * find_get_entries() will search for and return a group of up to
> - * @nr_entries entries in the mapping. The entries are placed at
> - * @entries. find_get_entries() takes a reference against any actual
> - * pages it returns.
> + * find_get_entries() will search for and return a batch of entries in
> + * the mapping. The entries are placed in @pvec. find_get_entries()
> + * takes a reference on any actual pages it returns.
> *
> * The search returns a group of mapping-contiguous page cache entries
> * with ascending indexes. There may be holes in the indices due to
> @@ -1824,15 +1822,12 @@ static inline struct page *xas_find_get_entry(struct xa_state *xas,
> * Return: the number of pages and shadow entries which were found.
> */
> unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
> - pgoff_t end, unsigned int nr_entries, struct page **entries,
> - pgoff_t *indices)
> + pgoff_t end, struct pagevec *pvec, pgoff_t *indices)
> {
> XA_STATE(xas, &mapping->i_pages, start);
> struct page *page;
> unsigned int ret = 0;
> -
> - if (!nr_entries)
> - return 0;
> + unsigned nr_entries = PAGEVEC_SIZE;
>
> rcu_read_lock();
> while ((page = xas_find_get_entry(&xas, end, XA_PRESENT))) {
> @@ -1847,11 +1842,13 @@ unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
> }
>
> indices[ret] = xas.xa_index;
> - entries[ret] = page;
> + pvec->pages[ret] = page;
> if (++ret == nr_entries)
> break;
> }
> rcu_read_unlock();
> +
> + pvec->nr = ret;
> return ret;
> }
>
> diff --git a/mm/shmem.c b/mm/shmem.c
> index 404e45c285ca..b2b94025841a 100644
> --- a/mm/shmem.c
> +++ b/mm/shmem.c
> @@ -963,9 +963,8 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
> while (index < end) {
> cond_resched();
>
> - pvec.nr = find_get_entries(mapping, index, end - 1,
> - PAGEVEC_SIZE, pvec.pages, indices);
> - if (!pvec.nr) {
> + if (!find_get_entries(mapping, index, end - 1, &pvec,
> + indices)) {
> /* If all gone or hole-punch or unfalloc, we're done */
> if (index == start || end != -1)
> break;
> diff --git a/mm/swap.c b/mm/swap.c
> index d4e3ba4c967c..40b23300d353 100644
> --- a/mm/swap.c
> +++ b/mm/swap.c
> @@ -1060,9 +1060,7 @@ unsigned pagevec_lookup_entries(struct pagevec *pvec,
> struct address_space *mapping, pgoff_t start, pgoff_t end,
> pgoff_t *indices)
> {
> - pvec->nr = find_get_entries(mapping, start, end, PAGEVEC_SIZE,
> - pvec->pages, indices);
> - return pagevec_count(pvec);
> + return find_get_entries(mapping, start, end, pvec, indices);
> }
>
> /**
> --
> 2.28.0
>
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 38+ messages in thread
* [PATCH v2 10/12] mm: Remove pagevec_lookup_entries
2020-09-14 13:00 [PATCH v2 00/12] Overhaul multi-page lookups for THP Matthew Wilcox (Oracle)
` (8 preceding siblings ...)
2020-09-14 13:00 ` [PATCH v2 09/12] mm: Pass pvec directly to find_get_entries Matthew Wilcox (Oracle)
@ 2020-09-14 13:00 ` Matthew Wilcox (Oracle)
2020-09-29 9:08 ` Jan Kara
2020-09-14 13:00 ` [PATCH v2 11/12] mm/truncate,shmem: Handle truncates that split THPs Matthew Wilcox (Oracle)
` (3 subsequent siblings)
13 siblings, 1 reply; 38+ messages in thread
From: Matthew Wilcox (Oracle) @ 2020-09-14 13:00 UTC (permalink / raw)
To: linux-mm
Cc: Matthew Wilcox (Oracle),
Andrew Morton, Hugh Dickins, William Kucharski, Johannes Weiner,
Jan Kara, Yang Shi, Dave Chinner, linux-kernel
pagevec_lookup_entries() is now just a wrapper around find_get_entries()
so remove it and convert all its callers.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
include/linux/pagevec.h | 3 ---
mm/swap.c | 36 ++----------------------------------
mm/truncate.c | 4 ++--
3 files changed, 4 insertions(+), 39 deletions(-)
diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h
index ce77724a2ab7..a45bea4b4d08 100644
--- a/include/linux/pagevec.h
+++ b/include/linux/pagevec.h
@@ -25,9 +25,6 @@ struct pagevec {
void __pagevec_release(struct pagevec *pvec);
void __pagevec_lru_add(struct pagevec *pvec);
-unsigned pagevec_lookup_entries(struct pagevec *pvec,
- struct address_space *mapping, pgoff_t start, pgoff_t end,
- pgoff_t *indices);
void pagevec_remove_exceptionals(struct pagevec *pvec);
unsigned pagevec_lookup_range(struct pagevec *pvec,
struct address_space *mapping,
diff --git a/mm/swap.c b/mm/swap.c
index 40b23300d353..cc0d648f79d4 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -1031,44 +1031,12 @@ void __pagevec_lru_add(struct pagevec *pvec)
pagevec_lru_move_fn(pvec, __pagevec_lru_add_fn, NULL);
}
-/**
- * pagevec_lookup_entries - gang pagecache lookup
- * @pvec: Where the resulting entries are placed
- * @mapping: The address_space to search
- * @start: The starting entry index
- * @end: The highest index to return (inclusive).
- * @nr_entries: The maximum number of pages
- * @indices: The cache indices corresponding to the entries in @pvec
- *
- * pagevec_lookup_entries() will search for and return a group of up
- * to @nr_pages pages and shadow entries in the mapping. All
- * entries are placed in @pvec. pagevec_lookup_entries() takes a
- * reference against actual pages in @pvec.
- *
- * The search returns a group of mapping-contiguous entries with
- * ascending indexes. There may be holes in the indices due to
- * not-present entries.
- *
- * Only one subpage of a Transparent Huge Page is returned in one call:
- * allowing truncate_inode_pages_range() to evict the whole THP without
- * cycling through a pagevec of extra references.
- *
- * pagevec_lookup_entries() returns the number of entries which were
- * found.
- */
-unsigned pagevec_lookup_entries(struct pagevec *pvec,
- struct address_space *mapping, pgoff_t start, pgoff_t end,
- pgoff_t *indices)
-{
- return find_get_entries(mapping, start, end, pvec, indices);
-}
-
/**
* pagevec_remove_exceptionals - pagevec exceptionals pruning
* @pvec: The pagevec to prune
*
- * pagevec_lookup_entries() fills both pages and exceptional radix
- * tree entries into the pagevec. This function prunes all
+ * find_get_entries() fills both pages and XArray value entries (aka
+ * exceptional entries) into the pagevec. This function prunes all
* exceptionals from @pvec without leaving holes, so that it can be
* passed on to page-only pagevec operations.
*/
diff --git a/mm/truncate.c b/mm/truncate.c
index bbd0a02eaed7..d62aeffbffcc 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -376,7 +376,7 @@ void truncate_inode_pages_range(struct address_space *mapping,
index = start;
for ( ; ; ) {
cond_resched();
- if (!pagevec_lookup_entries(&pvec, mapping, index, end - 1,
+ if (!find_get_entries(mapping, index, end - 1, &pvec,
indices)) {
/* If all gone from start onwards, we're done */
if (index == start)
@@ -611,7 +611,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
pagevec_init(&pvec);
index = start;
- while (pagevec_lookup_entries(&pvec, mapping, index, end, indices)) {
+ while (find_get_entries(mapping, index, end, &pvec, indices)) {
for (i = 0; i < pagevec_count(&pvec); i++) {
struct page *page = pvec.pages[i];
--
2.28.0
^ permalink raw reply related [flat|nested] 38+ messages in thread* Re: [PATCH v2 10/12] mm: Remove pagevec_lookup_entries
2020-09-14 13:00 ` [PATCH v2 10/12] mm: Remove pagevec_lookup_entries Matthew Wilcox (Oracle)
@ 2020-09-29 9:08 ` Jan Kara
0 siblings, 0 replies; 38+ messages in thread
From: Jan Kara @ 2020-09-29 9:08 UTC (permalink / raw)
To: Matthew Wilcox (Oracle)
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Jan Kara, Yang Shi, Dave Chinner, linux-kernel
On Mon 14-09-20 14:00:40, Matthew Wilcox (Oracle) wrote:
> pagevec_lookup_entries() is now just a wrapper around find_get_entries()
> so remove it and convert all its callers.
>
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Looks good. You can add:
Reviewed-by: Jan Kara <jack@suse.cz>
Honza
> ---
> include/linux/pagevec.h | 3 ---
> mm/swap.c | 36 ++----------------------------------
> mm/truncate.c | 4 ++--
> 3 files changed, 4 insertions(+), 39 deletions(-)
>
> diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h
> index ce77724a2ab7..a45bea4b4d08 100644
> --- a/include/linux/pagevec.h
> +++ b/include/linux/pagevec.h
> @@ -25,9 +25,6 @@ struct pagevec {
>
> void __pagevec_release(struct pagevec *pvec);
> void __pagevec_lru_add(struct pagevec *pvec);
> -unsigned pagevec_lookup_entries(struct pagevec *pvec,
> - struct address_space *mapping, pgoff_t start, pgoff_t end,
> - pgoff_t *indices);
> void pagevec_remove_exceptionals(struct pagevec *pvec);
> unsigned pagevec_lookup_range(struct pagevec *pvec,
> struct address_space *mapping,
> diff --git a/mm/swap.c b/mm/swap.c
> index 40b23300d353..cc0d648f79d4 100644
> --- a/mm/swap.c
> +++ b/mm/swap.c
> @@ -1031,44 +1031,12 @@ void __pagevec_lru_add(struct pagevec *pvec)
> pagevec_lru_move_fn(pvec, __pagevec_lru_add_fn, NULL);
> }
>
> -/**
> - * pagevec_lookup_entries - gang pagecache lookup
> - * @pvec: Where the resulting entries are placed
> - * @mapping: The address_space to search
> - * @start: The starting entry index
> - * @end: The highest index to return (inclusive).
> - * @nr_entries: The maximum number of pages
> - * @indices: The cache indices corresponding to the entries in @pvec
> - *
> - * pagevec_lookup_entries() will search for and return a group of up
> - * to @nr_pages pages and shadow entries in the mapping. All
> - * entries are placed in @pvec. pagevec_lookup_entries() takes a
> - * reference against actual pages in @pvec.
> - *
> - * The search returns a group of mapping-contiguous entries with
> - * ascending indexes. There may be holes in the indices due to
> - * not-present entries.
> - *
> - * Only one subpage of a Transparent Huge Page is returned in one call:
> - * allowing truncate_inode_pages_range() to evict the whole THP without
> - * cycling through a pagevec of extra references.
> - *
> - * pagevec_lookup_entries() returns the number of entries which were
> - * found.
> - */
> -unsigned pagevec_lookup_entries(struct pagevec *pvec,
> - struct address_space *mapping, pgoff_t start, pgoff_t end,
> - pgoff_t *indices)
> -{
> - return find_get_entries(mapping, start, end, pvec, indices);
> -}
> -
> /**
> * pagevec_remove_exceptionals - pagevec exceptionals pruning
> * @pvec: The pagevec to prune
> *
> - * pagevec_lookup_entries() fills both pages and exceptional radix
> - * tree entries into the pagevec. This function prunes all
> + * find_get_entries() fills both pages and XArray value entries (aka
> + * exceptional entries) into the pagevec. This function prunes all
> * exceptionals from @pvec without leaving holes, so that it can be
> * passed on to page-only pagevec operations.
> */
> diff --git a/mm/truncate.c b/mm/truncate.c
> index bbd0a02eaed7..d62aeffbffcc 100644
> --- a/mm/truncate.c
> +++ b/mm/truncate.c
> @@ -376,7 +376,7 @@ void truncate_inode_pages_range(struct address_space *mapping,
> index = start;
> for ( ; ; ) {
> cond_resched();
> - if (!pagevec_lookup_entries(&pvec, mapping, index, end - 1,
> + if (!find_get_entries(mapping, index, end - 1, &pvec,
> indices)) {
> /* If all gone from start onwards, we're done */
> if (index == start)
> @@ -611,7 +611,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping,
>
> pagevec_init(&pvec);
> index = start;
> - while (pagevec_lookup_entries(&pvec, mapping, index, end, indices)) {
> + while (find_get_entries(mapping, index, end, &pvec, indices)) {
> for (i = 0; i < pagevec_count(&pvec); i++) {
> struct page *page = pvec.pages[i];
>
> --
> 2.28.0
>
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 38+ messages in thread
* [PATCH v2 11/12] mm/truncate,shmem: Handle truncates that split THPs
2020-09-14 13:00 [PATCH v2 00/12] Overhaul multi-page lookups for THP Matthew Wilcox (Oracle)
` (9 preceding siblings ...)
2020-09-14 13:00 ` [PATCH v2 10/12] mm: Remove pagevec_lookup_entries Matthew Wilcox (Oracle)
@ 2020-09-14 13:00 ` Matthew Wilcox (Oracle)
2020-09-30 11:59 ` Jan Kara
2020-09-14 13:00 ` [PATCH v2 12/12] mm/filemap: Return only head pages from find_get_entries Matthew Wilcox (Oracle)
` (2 subsequent siblings)
13 siblings, 1 reply; 38+ messages in thread
From: Matthew Wilcox (Oracle) @ 2020-09-14 13:00 UTC (permalink / raw)
To: linux-mm
Cc: Matthew Wilcox (Oracle),
Andrew Morton, Hugh Dickins, William Kucharski, Johannes Weiner,
Jan Kara, Yang Shi, Dave Chinner, linux-kernel
Handle THP splitting in the parts of the truncation functions which
already handle partial pages. Factor all that code out into a new
function called truncate_inode_partial_page().
We lose the easy 'bail out' path if a truncate or hole punch is entirely
within a single page. We can add some more complex logic to restore
the optimisation if it proves to be worthwhile.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
mm/internal.h | 1 +
mm/shmem.c | 98 +++++++++++++++--------------------------
mm/truncate.c | 119 +++++++++++++++++++++++++++++++-------------------
3 files changed, 110 insertions(+), 108 deletions(-)
diff --git a/mm/internal.h b/mm/internal.h
index 1c5f4425ed74..d17d9060902f 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -620,4 +620,5 @@ struct migration_target_control {
gfp_t gfp_mask;
};
+bool truncate_inode_partial_page(struct page *page, loff_t start, loff_t end);
#endif /* __MM_INTERNAL_H */
diff --git a/mm/shmem.c b/mm/shmem.c
index b2b94025841a..23bb79ec070f 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -855,32 +855,6 @@ void shmem_unlock_mapping(struct address_space *mapping)
}
}
-/*
- * Check whether a hole-punch or truncation needs to split a huge page,
- * returning true if no split was required, or the split has been successful.
- *
- * Eviction (or truncation to 0 size) should never need to split a huge page;
- * but in rare cases might do so, if shmem_undo_range() failed to trylock on
- * head, and then succeeded to trylock on tail.
- *
- * A split can only succeed when there are no additional references on the
- * huge page: so the split below relies upon find_get_entries() having stopped
- * when it found a subpage of the huge page, without getting further references.
- */
-static bool shmem_punch_compound(struct page *page, pgoff_t start, pgoff_t end)
-{
- if (!PageTransCompound(page))
- return true;
-
- /* Just proceed to delete a huge page wholly within the range punched */
- if (PageHead(page) &&
- page->index >= start && page->index + HPAGE_PMD_NR <= end)
- return true;
-
- /* Try to split huge page, so we can truly punch the hole or truncate */
- return split_huge_page(page) >= 0;
-}
-
/*
* Remove range of pages and swap entries from page cache, and free them.
* If !unfalloc, truncate or punch hole; if unfalloc, undo failed fallocate.
@@ -892,10 +866,9 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
struct shmem_inode_info *info = SHMEM_I(inode);
pgoff_t start = (lstart + PAGE_SIZE - 1) >> PAGE_SHIFT;
pgoff_t end = (lend + 1) >> PAGE_SHIFT;
- unsigned int partial_start = lstart & (PAGE_SIZE - 1);
- unsigned int partial_end = (lend + 1) & (PAGE_SIZE - 1);
struct pagevec pvec;
pgoff_t indices[PAGEVEC_SIZE];
+ struct page *page;
long nr_swaps_freed = 0;
pgoff_t index;
int i;
@@ -908,7 +881,7 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
while (index < end && find_lock_entries(mapping, index, end - 1,
&pvec, indices)) {
for (i = 0; i < pagevec_count(&pvec); i++) {
- struct page *page = pvec.pages[i];
+ page = pvec.pages[i];
index = indices[i];
@@ -931,33 +904,39 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
index++;
}
- if (partial_start) {
- struct page *page = NULL;
- shmem_getpage(inode, start - 1, &page, SGP_READ);
- if (page) {
- unsigned int top = PAGE_SIZE;
- if (start > end) {
- top = partial_end;
- partial_end = 0;
- }
- zero_user_segment(page, partial_start, top);
- set_page_dirty(page);
- unlock_page(page);
- put_page(page);
+ index = -1;
+ if (end != -1 && ((lend + 1) % PAGE_SIZE))
+ index = lend >> PAGE_SHIFT;
+ page = NULL;
+ shmem_getpage(inode, lstart >> PAGE_SHIFT, &page, SGP_READ);
+ if (page) {
+ bool same_page;
+
+ page = thp_head(page);
+ same_page = lend + 1 < page_offset(page) + thp_size(page);
+ if (same_page)
+ index = -1;
+ set_page_dirty(page);
+ if (!truncate_inode_partial_page(page, lstart, lend)) {
+ start = page->index + thp_nr_pages(page);
+ if (same_page)
+ end = page->index;
}
+ unlock_page(page);
+ put_page(page);
+ page = NULL;
}
- if (partial_end) {
- struct page *page = NULL;
+
+ if (index != -1)
shmem_getpage(inode, end, &page, SGP_READ);
- if (page) {
- zero_user_segment(page, 0, partial_end);
- set_page_dirty(page);
- unlock_page(page);
- put_page(page);
- }
+ if (page) {
+ page = thp_head(page);
+ set_page_dirty(page);
+ if (!truncate_inode_partial_page(page, lstart, lend))
+ end = page->index;
+ unlock_page(page);
+ put_page(page);
}
- if (start >= end)
- return;
index = start;
while (index < end) {
@@ -973,7 +952,7 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
continue;
}
for (i = 0; i < pagevec_count(&pvec); i++) {
- struct page *page = pvec.pages[i];
+ page = pvec.pages[i];
index = indices[i];
if (xa_is_value(page)) {
@@ -998,18 +977,9 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
break;
}
VM_BUG_ON_PAGE(PageWriteback(page), page);
- if (shmem_punch_compound(page, start, end))
- truncate_inode_page(mapping, page);
- else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) {
- /* Wipe the page and don't get stuck */
- clear_highpage(page);
- flush_dcache_page(page);
- set_page_dirty(page);
- if (index <
- round_up(start, HPAGE_PMD_NR))
- start = index + 1;
- }
+ truncate_inode_page(mapping, page);
}
+ index = page->index + thp_nr_pages(page) - 1;
unlock_page(page);
}
pagevec_remove_exceptionals(&pvec);
diff --git a/mm/truncate.c b/mm/truncate.c
index d62aeffbffcc..06ed2f93069d 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -224,6 +224,53 @@ int truncate_inode_page(struct address_space *mapping, struct page *page)
return 0;
}
+/*
+ * Handle partial (transparent) pages. The page may be entirely within the
+ * range if a split has raced with us. If not, we zero the part of the
+ * page that's within the (start, end] range, and then split the page if
+ * it's a THP. split_page_range() will discard pages which now lie beyond
+ * i_size, and we rely on the caller to discard pages which lie within a
+ * newly created hole.
+ *
+ * Returns false if THP splitting failed so the caller can can avoid
+ * discarding the entire page which is stubbornly unsplit.
+ */
+bool truncate_inode_partial_page(struct page *page, loff_t start, loff_t end)
+{
+ loff_t pos = page_offset(page);
+ unsigned int offset, length;
+
+ if (pos < start)
+ offset = start - pos;
+ else
+ offset = 0;
+ length = thp_size(page);
+ if (pos + length <= (u64)end)
+ length = length - offset;
+ else
+ length = end + 1 - pos - offset;
+
+ wait_on_page_writeback(page);
+ if (length == thp_size(page)) {
+ truncate_inode_page(page->mapping, page);
+ return true;
+ }
+
+ /*
+ * We may be zeroing pages we're about to discard, but it avoids
+ * doing a complex calculation here, and then doing the zeroing
+ * anyway if the page split fails.
+ */
+ zero_user(page, offset, length);
+
+ cleancache_invalidate_page(page->mapping, page);
+ if (page_has_private(page))
+ do_invalidatepage(page, offset, length);
+ if (!PageTransHuge(page))
+ return true;
+ return split_huge_page(page) == 0;
+}
+
/*
* Used to get rid of pages on hardware memory corruption.
*/
@@ -288,20 +335,15 @@ void truncate_inode_pages_range(struct address_space *mapping,
{
pgoff_t start; /* inclusive */
pgoff_t end; /* exclusive */
- unsigned int partial_start; /* inclusive */
- unsigned int partial_end; /* exclusive */
struct pagevec pvec;
pgoff_t indices[PAGEVEC_SIZE];
pgoff_t index;
int i;
+ struct page * page;
if (mapping->nrpages == 0 && mapping->nrexceptional == 0)
goto out;
- /* Offsets within partial pages */
- partial_start = lstart & (PAGE_SIZE - 1);
- partial_end = (lend + 1) & (PAGE_SIZE - 1);
-
/*
* 'start' and 'end' always covers the range of pages to be fully
* truncated. Partial pages are covered with 'partial_start' at the
@@ -334,48 +376,37 @@ void truncate_inode_pages_range(struct address_space *mapping,
cond_resched();
}
- if (partial_start) {
- struct page *page = find_lock_page(mapping, start - 1);
- if (page) {
- unsigned int top = PAGE_SIZE;
- if (start > end) {
- /* Truncation within a single page */
- top = partial_end;
- partial_end = 0;
- }
- wait_on_page_writeback(page);
- zero_user_segment(page, partial_start, top);
- cleancache_invalidate_page(mapping, page);
- if (page_has_private(page))
- do_invalidatepage(page, partial_start,
- top - partial_start);
- unlock_page(page);
- put_page(page);
+ index = -1;
+ if (end != -1 && ((lend + 1) % PAGE_SIZE))
+ index = lend >> PAGE_SHIFT;
+ page = find_lock_head(mapping, lstart >> PAGE_SHIFT);
+ if (page) {
+ bool same_page = lend + 1 < page_offset(page) + thp_size(page);
+ if (same_page)
+ index = -1;
+ if (!truncate_inode_partial_page(page, lstart, lend)) {
+ start = page->index + thp_nr_pages(page);
+ if (same_page)
+ end = page->index;
}
+ unlock_page(page);
+ put_page(page);
+ page = NULL;
}
- if (partial_end) {
- struct page *page = find_lock_page(mapping, end);
- if (page) {
- wait_on_page_writeback(page);
- zero_user_segment(page, 0, partial_end);
- cleancache_invalidate_page(mapping, page);
- if (page_has_private(page))
- do_invalidatepage(page, 0,
- partial_end);
- unlock_page(page);
- put_page(page);
- }
+
+ if (index != -1)
+ page = find_lock_head(mapping, index);
+ if (page) {
+ if (!truncate_inode_partial_page(page, lstart, lend))
+ end = page->index;
+ unlock_page(page);
+ put_page(page);
}
- /*
- * If the truncation happened within a single page no pages
- * will be released, just zeroed, so we can bail out now.
- */
- if (start >= end)
- goto out;
index = start;
- for ( ; ; ) {
+ while (index < end) {
cond_resched();
+
if (!find_get_entries(mapping, index, end - 1, &pvec,
indices)) {
/* If all gone from start onwards, we're done */
@@ -387,7 +418,7 @@ void truncate_inode_pages_range(struct address_space *mapping,
}
for (i = 0; i < pagevec_count(&pvec); i++) {
- struct page *page = pvec.pages[i];
+ page = pvec.pages[i];
/* We rely upon deletion not changing page->index */
index = indices[i];
@@ -396,7 +427,7 @@ void truncate_inode_pages_range(struct address_space *mapping,
continue;
lock_page(page);
- WARN_ON(page_to_index(page) != index);
+ index = page->index + thp_nr_pages(page) - 1;
wait_on_page_writeback(page);
truncate_inode_page(mapping, page);
unlock_page(page);
--
2.28.0
^ permalink raw reply related [flat|nested] 38+ messages in thread* Re: [PATCH v2 11/12] mm/truncate,shmem: Handle truncates that split THPs
2020-09-14 13:00 ` [PATCH v2 11/12] mm/truncate,shmem: Handle truncates that split THPs Matthew Wilcox (Oracle)
@ 2020-09-30 11:59 ` Jan Kara
2020-09-30 14:51 ` Matthew Wilcox
0 siblings, 1 reply; 38+ messages in thread
From: Jan Kara @ 2020-09-30 11:59 UTC (permalink / raw)
To: Matthew Wilcox (Oracle)
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Jan Kara, Yang Shi, Dave Chinner, linux-kernel
On Mon 14-09-20 14:00:41, Matthew Wilcox (Oracle) wrote:
> Handle THP splitting in the parts of the truncation functions which
> already handle partial pages. Factor all that code out into a new
> function called truncate_inode_partial_page().
>
> We lose the easy 'bail out' path if a truncate or hole punch is entirely
> within a single page. We can add some more complex logic to restore
> the optimisation if it proves to be worthwhile.
>
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Overall this looks OK. Some smaller suggestions below...
> @@ -931,33 +904,39 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
> index++;
> }
>
> - if (partial_start) {
> - struct page *page = NULL;
> - shmem_getpage(inode, start - 1, &page, SGP_READ);
> - if (page) {
> - unsigned int top = PAGE_SIZE;
> - if (start > end) {
> - top = partial_end;
> - partial_end = 0;
> - }
> - zero_user_segment(page, partial_start, top);
> - set_page_dirty(page);
> - unlock_page(page);
> - put_page(page);
> + index = -1;
> + if (end != -1 && ((lend + 1) % PAGE_SIZE))
^^
Hum, is this guaranteed to compile properly on 32-bit archs without
optimization? It would be 64-bit division... Maybe we don't care, it just
caught my eye...
BTW you could just drop end != -1 part because end == -1 iff lend == -1 so
(lend + 1) % PAGE_SIZE is stronger.
> + index = lend >> PAGE_SHIFT;
> + page = NULL;
> + shmem_getpage(inode, lstart >> PAGE_SHIFT, &page, SGP_READ);
> + if (page) {
> + bool same_page;
> +
> + page = thp_head(page);
> + same_page = lend + 1 < page_offset(page) + thp_size(page);
^^^^^^^^ Just lend here?
> + if (same_page)
> + index = -1;
> + set_page_dirty(page);
> + if (!truncate_inode_partial_page(page, lstart, lend)) {
> + start = page->index + thp_nr_pages(page);
> + if (same_page)
> + end = page->index;
> }
> + unlock_page(page);
> + put_page(page);
> + page = NULL;
> }
> - if (partial_end) {
> - struct page *page = NULL;
> +
> + if (index != -1)
> shmem_getpage(inode, end, &page, SGP_READ);
> - if (page) {
> - zero_user_segment(page, 0, partial_end);
> - set_page_dirty(page);
> - unlock_page(page);
> - put_page(page);
> - }
> + if (page) {
> + page = thp_head(page);
> + set_page_dirty(page);
> + if (!truncate_inode_partial_page(page, lstart, lend))
> + end = page->index;
> + unlock_page(page);
> + put_page(page);
> }
> - if (start >= end)
> - return;
You use 'index' effectively as bool in all of the above (only ever check
index != -1). And effectively you only use it to communicate whether tail
partial page got already handled or not. Maybe there's some less cryptic
way to achieve that? Even separate bool just for that would be probably
better that this.
> index = start;
> while (index < end) {
> diff --git a/mm/truncate.c b/mm/truncate.c
> index d62aeffbffcc..06ed2f93069d 100644
> --- a/mm/truncate.c
> +++ b/mm/truncate.c
> @@ -224,6 +224,53 @@ int truncate_inode_page(struct address_space *mapping, struct page *page)
> return 0;
> }
>
> +/*
> + * Handle partial (transparent) pages. The page may be entirely within the
> + * range if a split has raced with us. If not, we zero the part of the
> + * page that's within the (start, end] range, and then split the page if
^ '[' here - start is inclusive as well...
> + * it's a THP. split_page_range() will discard pages which now lie beyond
> + * i_size, and we rely on the caller to discard pages which lie within a
> + * newly created hole.
> + *
> + * Returns false if THP splitting failed so the caller can can avoid
^^^^^^^ just 'can'
> + * discarding the entire page which is stubbornly unsplit.
> + */
> +bool truncate_inode_partial_page(struct page *page, loff_t start, loff_t end)
> +{
> + loff_t pos = page_offset(page);
> + unsigned int offset, length;
> +
> + if (pos < start)
> + offset = start - pos;
> + else
> + offset = 0;
> + length = thp_size(page);
> + if (pos + length <= (u64)end)
> + length = length - offset;
> + else
> + length = end + 1 - pos - offset;
> +
> + wait_on_page_writeback(page);
> + if (length == thp_size(page)) {
> + truncate_inode_page(page->mapping, page);
> + return true;
> + }
> +
> + /*
> + * We may be zeroing pages we're about to discard, but it avoids
> + * doing a complex calculation here, and then doing the zeroing
> + * anyway if the page split fails.
> + */
> + zero_user(page, offset, length);
> +
> + cleancache_invalidate_page(page->mapping, page);
> + if (page_has_private(page))
> + do_invalidatepage(page, offset, length);
> + if (!PageTransHuge(page))
> + return true;
> + return split_huge_page(page) == 0;
> +}
> +
> /*
> * Used to get rid of pages on hardware memory corruption.
> */
> @@ -288,20 +335,15 @@ void truncate_inode_pages_range(struct address_space *mapping,
> {
> pgoff_t start; /* inclusive */
> pgoff_t end; /* exclusive */
> - unsigned int partial_start; /* inclusive */
> - unsigned int partial_end; /* exclusive */
> struct pagevec pvec;
> pgoff_t indices[PAGEVEC_SIZE];
> pgoff_t index;
> int i;
> + struct page * page;
>
> if (mapping->nrpages == 0 && mapping->nrexceptional == 0)
> goto out;
>
> - /* Offsets within partial pages */
> - partial_start = lstart & (PAGE_SIZE - 1);
> - partial_end = (lend + 1) & (PAGE_SIZE - 1);
> -
> /*
> * 'start' and 'end' always covers the range of pages to be fully
> * truncated. Partial pages are covered with 'partial_start' at the
> @@ -334,48 +376,37 @@ void truncate_inode_pages_range(struct address_space *mapping,
> cond_resched();
> }
>
> - if (partial_start) {
> - struct page *page = find_lock_page(mapping, start - 1);
> - if (page) {
> - unsigned int top = PAGE_SIZE;
> - if (start > end) {
> - /* Truncation within a single page */
> - top = partial_end;
> - partial_end = 0;
> - }
> - wait_on_page_writeback(page);
> - zero_user_segment(page, partial_start, top);
> - cleancache_invalidate_page(mapping, page);
> - if (page_has_private(page))
> - do_invalidatepage(page, partial_start,
> - top - partial_start);
> - unlock_page(page);
> - put_page(page);
> + index = -1;
> + if (end != -1 && ((lend + 1) % PAGE_SIZE))
^^^^^^^^^ Again this is unnecessary...
> + index = lend >> PAGE_SHIFT;
> + page = find_lock_head(mapping, lstart >> PAGE_SHIFT);
> + if (page) {
> + bool same_page = lend + 1 < page_offset(page) + thp_size(page);
^^^^^^^ Again why +1 here?
> + if (same_page)
> + index = -1;
> + if (!truncate_inode_partial_page(page, lstart, lend)) {
> + start = page->index + thp_nr_pages(page);
> + if (same_page)
> + end = page->index;
> }
> + unlock_page(page);
> + put_page(page);
> + page = NULL;
> }
> - if (partial_end) {
> - struct page *page = find_lock_page(mapping, end);
> - if (page) {
> - wait_on_page_writeback(page);
> - zero_user_segment(page, 0, partial_end);
> - cleancache_invalidate_page(mapping, page);
> - if (page_has_private(page))
> - do_invalidatepage(page, 0,
> - partial_end);
> - unlock_page(page);
> - put_page(page);
> - }
> +
> + if (index != -1)
> + page = find_lock_head(mapping, index);
Similarly to shmem the use of index is a bit confusing here but it at least
gets used in this case so OK. But I'd still find something like:
if (!tail_page_already_truncated)
page = find_lock_head(mapping, lend >> PAGE_SHIFT);
easier to grasp.
> + if (page) {
> + if (!truncate_inode_partial_page(page, lstart, lend))
> + end = page->index;
> + unlock_page(page);
> + put_page(page);
> }
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: [PATCH v2 11/12] mm/truncate,shmem: Handle truncates that split THPs
2020-09-30 11:59 ` Jan Kara
@ 2020-09-30 14:51 ` Matthew Wilcox
0 siblings, 0 replies; 38+ messages in thread
From: Matthew Wilcox @ 2020-09-30 14:51 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Yang Shi, Dave Chinner, linux-kernel
On Wed, Sep 30, 2020 at 01:59:19PM +0200, Jan Kara wrote:
> > @@ -931,33 +904,39 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend,
> > index++;
> > }
> >
> > - if (partial_start) {
> > - struct page *page = NULL;
> > - shmem_getpage(inode, start - 1, &page, SGP_READ);
> > - if (page) {
> > - unsigned int top = PAGE_SIZE;
> > - if (start > end) {
> > - top = partial_end;
> > - partial_end = 0;
> > - }
> > - zero_user_segment(page, partial_start, top);
> > - set_page_dirty(page);
> > - unlock_page(page);
> > - put_page(page);
> > + index = -1;
> > + if (end != -1 && ((lend + 1) % PAGE_SIZE))
> ^^
> Hum, is this guaranteed to compile properly on 32-bit archs without
> optimization? It would be 64-bit division... Maybe we don't care, it just
> caught my eye...
Looks like GCC properly reduces it:
00000000 <f>:
int f(long long x)
{
if ((x + 1) % 4096)
0: 8b 44 24 04 mov 0x4(%esp),%eax
4: 83 c0 01 add $0x1,%eax
7: 25 ff 0f 00 00 and $0xfff,%eax
c: 0f 95 c0 setne %al
f: 0f b6 c0 movzbl %al,%eax
return 1;
return 0;
}
12: c3 ret
> BTW you could just drop end != -1 part because end == -1 iff lend == -1 so
> (lend + 1) % PAGE_SIZE is stronger.
Yes. I think that actually makes it easier to read.
> > + index = lend >> PAGE_SHIFT;
> > + page = NULL;
> > + shmem_getpage(inode, lstart >> PAGE_SHIFT, &page, SGP_READ);
> > + if (page) {
> > + bool same_page;
> > +
> > + page = thp_head(page);
> > + same_page = lend + 1 < page_offset(page) + thp_size(page);
> ^^^^^^^^ Just lend here?
Oops. Yes. If lstart is 4096 and lend is 8191, this is definitely same_page.
> > - if (partial_end) {
> > - struct page *page = NULL;
> > +
> > + if (index != -1)
> > shmem_getpage(inode, end, &page, SGP_READ);
> > - if (page) {
> > - zero_user_segment(page, 0, partial_end);
> > - set_page_dirty(page);
> > - unlock_page(page);
> > - put_page(page);
> > - }
> > + if (page) {
> > + page = thp_head(page);
> > + set_page_dirty(page);
> > + if (!truncate_inode_partial_page(page, lstart, lend))
> > + end = page->index;
> > + unlock_page(page);
> > + put_page(page);
> > }
> > - if (start >= end)
> > - return;
>
> You use 'index' effectively as bool in all of the above (only ever check
> index != -1). And effectively you only use it to communicate whether tail
> partial page got already handled or not. Maybe there's some less cryptic
> way to achieve that? Even separate bool just for that would be probably
> better that this.
As you note below, it makes more sense in truncate.c and for my own
sanity I was trying to keep the two functions as similar as possible.
I'm not sure why I looked up the page with 'end' instead of 'index'.
This didn't end up being as big a simplification as I thought it was going
to be. I think I'll reintroduce partial_end as a bool, like you suggest.
> > index = start;
> > while (index < end) {
>
> > diff --git a/mm/truncate.c b/mm/truncate.c
> > index d62aeffbffcc..06ed2f93069d 100644
> > --- a/mm/truncate.c
> > +++ b/mm/truncate.c
> > @@ -224,6 +224,53 @@ int truncate_inode_page(struct address_space *mapping, struct page *page)
> > return 0;
> > }
> >
> > +/*
> > + * Handle partial (transparent) pages. The page may be entirely within the
> > + * range if a split has raced with us. If not, we zero the part of the
> > + * page that's within the (start, end] range, and then split the page if
> ^ '[' here - start is inclusive as well...
Good point.
> > + * it's a THP. split_page_range() will discard pages which now lie beyond
> > + * i_size, and we rely on the caller to discard pages which lie within a
> > + * newly created hole.
> > + *
> > + * Returns false if THP splitting failed so the caller can can avoid
> ^^^^^^^ just 'can'
You're going to put Randy out of a job ;-)
^ permalink raw reply [flat|nested] 38+ messages in thread
* [PATCH v2 12/12] mm/filemap: Return only head pages from find_get_entries
2020-09-14 13:00 [PATCH v2 00/12] Overhaul multi-page lookups for THP Matthew Wilcox (Oracle)
` (10 preceding siblings ...)
2020-09-14 13:00 ` [PATCH v2 11/12] mm/truncate,shmem: Handle truncates that split THPs Matthew Wilcox (Oracle)
@ 2020-09-14 13:00 ` Matthew Wilcox (Oracle)
2020-09-30 12:15 ` Jan Kara
2020-09-28 20:13 ` [PATCH v2 00/12] Overhaul multi-page lookups for THP Matthew Wilcox
2020-09-29 8:50 ` William Kucharski
13 siblings, 1 reply; 38+ messages in thread
From: Matthew Wilcox (Oracle) @ 2020-09-14 13:00 UTC (permalink / raw)
To: linux-mm
Cc: Matthew Wilcox (Oracle),
Andrew Morton, Hugh Dickins, William Kucharski, Johannes Weiner,
Jan Kara, Yang Shi, Dave Chinner, linux-kernel
All callers now expect head (and base) pages, and can handle multiple
head pages in a single batch, so make find_get_entries() behave that way.
Also take the opportunity to make it use the pagevec infrastructure
instead of open-coding how pvecs behave. This has the side-effect of
being able to append to a pagevec with existing contents, although we
don't make use of that functionality anywhere yet.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
include/linux/pagemap.h | 2 --
mm/filemap.c | 36 ++++++++----------------------------
mm/internal.h | 2 ++
3 files changed, 10 insertions(+), 30 deletions(-)
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 788c280737e0..50d2c39b47ab 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -413,8 +413,6 @@ static inline struct page *find_subpage(struct page *head, pgoff_t index)
return head + (index & (thp_nr_pages(head) - 1));
}
-unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
- pgoff_t end, struct pagevec *pvec, pgoff_t *indices);
unsigned find_get_pages_range(struct address_space *mapping, pgoff_t *start,
pgoff_t end, unsigned int nr_pages,
struct page **pages);
diff --git a/mm/filemap.c b/mm/filemap.c
index a4ad294a34dc..dbe3f7050897 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1807,49 +1807,29 @@ static inline struct page *xas_find_get_entry(struct xa_state *xas,
* the mapping. The entries are placed in @pvec. find_get_entries()
* takes a reference on any actual pages it returns.
*
- * The search returns a group of mapping-contiguous page cache entries
- * with ascending indexes. There may be holes in the indices due to
- * not-present pages.
+ * The entries have ascending indexes. The indices may not be consecutive
+ * due to not-present entries or THPs.
*
* Any shadow entries of evicted pages, or swap entries from
* shmem/tmpfs, are included in the returned array.
*
- * If it finds a Transparent Huge Page, head or tail, find_get_entries()
- * stops at that page: the caller is likely to have a better way to handle
- * the compound page as a whole, and then skip its extent, than repeatedly
- * calling find_get_entries() to return all its tails.
- *
- * Return: the number of pages and shadow entries which were found.
+ * Return: The number of entries which were found.
*/
unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
pgoff_t end, struct pagevec *pvec, pgoff_t *indices)
{
XA_STATE(xas, &mapping->i_pages, start);
struct page *page;
- unsigned int ret = 0;
- unsigned nr_entries = PAGEVEC_SIZE;
rcu_read_lock();
while ((page = xas_find_get_entry(&xas, end, XA_PRESENT))) {
- /*
- * Terminate early on finding a THP, to allow the caller to
- * handle it all at once; but continue if this is hugetlbfs.
- */
- if (!xa_is_value(page) && PageTransHuge(page) &&
- !PageHuge(page)) {
- page = find_subpage(page, xas.xa_index);
- nr_entries = ret + 1;
- }
-
- indices[ret] = xas.xa_index;
- pvec->pages[ret] = page;
- if (++ret == nr_entries)
+ indices[pvec->nr] = xas.xa_index;
+ if (!pagevec_add(pvec, page))
break;
}
rcu_read_unlock();
- pvec->nr = ret;
- return ret;
+ return pagevec_count(pvec);
}
/**
@@ -1868,8 +1848,8 @@ unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
* not returned.
*
* The entries have ascending indexes. The indices may not be consecutive
- * due to not-present entries, THP pages, pages which could not be locked
- * or pages under writeback.
+ * due to not-present entries, THPs, pages which could not be locked or
+ * pages under writeback.
*
* Return: The number of entries which were found.
*/
diff --git a/mm/internal.h b/mm/internal.h
index d17d9060902f..55da9860a54b 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -62,6 +62,8 @@ static inline void force_page_cache_readahead(struct address_space *mapping,
struct page *find_get_entry(struct address_space *mapping, pgoff_t index);
struct page *find_lock_entry(struct address_space *mapping, pgoff_t index);
+unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
+ pgoff_t end, struct pagevec *pvec, pgoff_t *indices);
unsigned find_lock_entries(struct address_space *mapping, pgoff_t start,
pgoff_t end, struct pagevec *pvec, pgoff_t *indices);
--
2.28.0
^ permalink raw reply related [flat|nested] 38+ messages in thread* Re: [PATCH v2 12/12] mm/filemap: Return only head pages from find_get_entries
2020-09-14 13:00 ` [PATCH v2 12/12] mm/filemap: Return only head pages from find_get_entries Matthew Wilcox (Oracle)
@ 2020-09-30 12:15 ` Jan Kara
2020-09-30 12:36 ` Matthew Wilcox
0 siblings, 1 reply; 38+ messages in thread
From: Jan Kara @ 2020-09-30 12:15 UTC (permalink / raw)
To: Matthew Wilcox (Oracle)
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Jan Kara, Yang Shi, Dave Chinner, linux-kernel
On Mon 14-09-20 14:00:42, Matthew Wilcox (Oracle) wrote:
> All callers now expect head (and base) pages, and can handle multiple
> head pages in a single batch, so make find_get_entries() behave that way.
> Also take the opportunity to make it use the pagevec infrastructure
> instead of open-coding how pvecs behave. This has the side-effect of
> being able to append to a pagevec with existing contents, although we
> don't make use of that functionality anywhere yet.
>
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Looks good to me. You can add:
Reviewed-by: Jan Kara <jack@suse.cz>
I'm just curious: What has happened to pagevec_lookup_entries() call in
invalidate_inode_pages2_range()? Your series appears to be based on a tree
where the call already does not exist...
Honza
> ---
> include/linux/pagemap.h | 2 --
> mm/filemap.c | 36 ++++++++----------------------------
> mm/internal.h | 2 ++
> 3 files changed, 10 insertions(+), 30 deletions(-)
>
> diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
> index 788c280737e0..50d2c39b47ab 100644
> --- a/include/linux/pagemap.h
> +++ b/include/linux/pagemap.h
> @@ -413,8 +413,6 @@ static inline struct page *find_subpage(struct page *head, pgoff_t index)
> return head + (index & (thp_nr_pages(head) - 1));
> }
>
> -unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
> - pgoff_t end, struct pagevec *pvec, pgoff_t *indices);
> unsigned find_get_pages_range(struct address_space *mapping, pgoff_t *start,
> pgoff_t end, unsigned int nr_pages,
> struct page **pages);
> diff --git a/mm/filemap.c b/mm/filemap.c
> index a4ad294a34dc..dbe3f7050897 100644
> --- a/mm/filemap.c
> +++ b/mm/filemap.c
> @@ -1807,49 +1807,29 @@ static inline struct page *xas_find_get_entry(struct xa_state *xas,
> * the mapping. The entries are placed in @pvec. find_get_entries()
> * takes a reference on any actual pages it returns.
> *
> - * The search returns a group of mapping-contiguous page cache entries
> - * with ascending indexes. There may be holes in the indices due to
> - * not-present pages.
> + * The entries have ascending indexes. The indices may not be consecutive
> + * due to not-present entries or THPs.
> *
> * Any shadow entries of evicted pages, or swap entries from
> * shmem/tmpfs, are included in the returned array.
> *
> - * If it finds a Transparent Huge Page, head or tail, find_get_entries()
> - * stops at that page: the caller is likely to have a better way to handle
> - * the compound page as a whole, and then skip its extent, than repeatedly
> - * calling find_get_entries() to return all its tails.
> - *
> - * Return: the number of pages and shadow entries which were found.
> + * Return: The number of entries which were found.
> */
> unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
> pgoff_t end, struct pagevec *pvec, pgoff_t *indices)
> {
> XA_STATE(xas, &mapping->i_pages, start);
> struct page *page;
> - unsigned int ret = 0;
> - unsigned nr_entries = PAGEVEC_SIZE;
>
> rcu_read_lock();
> while ((page = xas_find_get_entry(&xas, end, XA_PRESENT))) {
> - /*
> - * Terminate early on finding a THP, to allow the caller to
> - * handle it all at once; but continue if this is hugetlbfs.
> - */
> - if (!xa_is_value(page) && PageTransHuge(page) &&
> - !PageHuge(page)) {
> - page = find_subpage(page, xas.xa_index);
> - nr_entries = ret + 1;
> - }
> -
> - indices[ret] = xas.xa_index;
> - pvec->pages[ret] = page;
> - if (++ret == nr_entries)
> + indices[pvec->nr] = xas.xa_index;
> + if (!pagevec_add(pvec, page))
> break;
> }
> rcu_read_unlock();
>
> - pvec->nr = ret;
> - return ret;
> + return pagevec_count(pvec);
> }
>
> /**
> @@ -1868,8 +1848,8 @@ unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
> * not returned.
> *
> * The entries have ascending indexes. The indices may not be consecutive
> - * due to not-present entries, THP pages, pages which could not be locked
> - * or pages under writeback.
> + * due to not-present entries, THPs, pages which could not be locked or
> + * pages under writeback.
> *
> * Return: The number of entries which were found.
> */
> diff --git a/mm/internal.h b/mm/internal.h
> index d17d9060902f..55da9860a54b 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -62,6 +62,8 @@ static inline void force_page_cache_readahead(struct address_space *mapping,
>
> struct page *find_get_entry(struct address_space *mapping, pgoff_t index);
> struct page *find_lock_entry(struct address_space *mapping, pgoff_t index);
> +unsigned find_get_entries(struct address_space *mapping, pgoff_t start,
> + pgoff_t end, struct pagevec *pvec, pgoff_t *indices);
> unsigned find_lock_entries(struct address_space *mapping, pgoff_t start,
> pgoff_t end, struct pagevec *pvec, pgoff_t *indices);
>
> --
> 2.28.0
>
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: [PATCH v2 12/12] mm/filemap: Return only head pages from find_get_entries
2020-09-30 12:15 ` Jan Kara
@ 2020-09-30 12:36 ` Matthew Wilcox
2020-09-30 17:08 ` Jan Kara
0 siblings, 1 reply; 38+ messages in thread
From: Matthew Wilcox @ 2020-09-30 12:36 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Yang Shi, Dave Chinner, linux-kernel
On Wed, Sep 30, 2020 at 02:15:12PM +0200, Jan Kara wrote:
> On Mon 14-09-20 14:00:42, Matthew Wilcox (Oracle) wrote:
> > All callers now expect head (and base) pages, and can handle multiple
> > head pages in a single batch, so make find_get_entries() behave that way.
> > Also take the opportunity to make it use the pagevec infrastructure
> > instead of open-coding how pvecs behave. This has the side-effect of
> > being able to append to a pagevec with existing contents, although we
> > don't make use of that functionality anywhere yet.
> >
> > Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
>
> Looks good to me. You can add:
>
> Reviewed-by: Jan Kara <jack@suse.cz>
>
> I'm just curious: What has happened to pagevec_lookup_entries() call in
> invalidate_inode_pages2_range()? Your series appears to be based on a tree
> where the call already does not exist...
That went away in patch 10 of this series.
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 12/12] mm/filemap: Return only head pages from find_get_entries
2020-09-30 12:36 ` Matthew Wilcox
@ 2020-09-30 17:08 ` Jan Kara
2020-09-30 17:23 ` Matthew Wilcox
0 siblings, 1 reply; 38+ messages in thread
From: Jan Kara @ 2020-09-30 17:08 UTC (permalink / raw)
To: Matthew Wilcox
Cc: Jan Kara, linux-mm, Andrew Morton, Hugh Dickins,
William Kucharski, Johannes Weiner, Yang Shi, Dave Chinner,
linux-kernel
On Wed 30-09-20 13:36:37, Matthew Wilcox wrote:
> On Wed, Sep 30, 2020 at 02:15:12PM +0200, Jan Kara wrote:
> > On Mon 14-09-20 14:00:42, Matthew Wilcox (Oracle) wrote:
> > > All callers now expect head (and base) pages, and can handle multiple
> > > head pages in a single batch, so make find_get_entries() behave that way.
> > > Also take the opportunity to make it use the pagevec infrastructure
> > > instead of open-coding how pvecs behave. This has the side-effect of
> > > being able to append to a pagevec with existing contents, although we
> > > don't make use of that functionality anywhere yet.
> > >
> > > Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
> >
> > Looks good to me. You can add:
> >
> > Reviewed-by: Jan Kara <jack@suse.cz>
> >
> > I'm just curious: What has happened to pagevec_lookup_entries() call in
> > invalidate_inode_pages2_range()? Your series appears to be based on a tree
> > where the call already does not exist...
>
> That went away in patch 10 of this series.
Ah, I see. Thanks. Then I'm somewhat wondering is really
invalidate_inode_pages2_range() safe for THP head pages? At least the:
unmap_mapping_pages(mapping, index, 1, false);
doesn't look adequate for THP head pages... do_launder_page() is also
doubtful but probably currently OK because THPs cannot be dirty at this
moment. But how about THPs that are partialy inside start-end range? So far
the function didn't care because it was operating on page basis so it
didn't care but now it is probably relevant... At least it would warrant a
comment in some changelog if you are convinced everything is safe.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 12/12] mm/filemap: Return only head pages from find_get_entries
2020-09-30 17:08 ` Jan Kara
@ 2020-09-30 17:23 ` Matthew Wilcox
2020-10-01 7:17 ` Jan Kara
0 siblings, 1 reply; 38+ messages in thread
From: Matthew Wilcox @ 2020-09-30 17:23 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Yang Shi, Dave Chinner, linux-kernel
On Wed, Sep 30, 2020 at 07:08:07PM +0200, Jan Kara wrote:
> On Wed 30-09-20 13:36:37, Matthew Wilcox wrote:
> > On Wed, Sep 30, 2020 at 02:15:12PM +0200, Jan Kara wrote:
> > > On Mon 14-09-20 14:00:42, Matthew Wilcox (Oracle) wrote:
> > > > All callers now expect head (and base) pages, and can handle multiple
> > > > head pages in a single batch, so make find_get_entries() behave that way.
> > > > Also take the opportunity to make it use the pagevec infrastructure
> > > > instead of open-coding how pvecs behave. This has the side-effect of
> > > > being able to append to a pagevec with existing contents, although we
> > > > don't make use of that functionality anywhere yet.
> > > >
> > > > Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
> > >
> > > Looks good to me. You can add:
> > >
> > > Reviewed-by: Jan Kara <jack@suse.cz>
> > >
> > > I'm just curious: What has happened to pagevec_lookup_entries() call in
> > > invalidate_inode_pages2_range()? Your series appears to be based on a tree
> > > where the call already does not exist...
> >
> > That went away in patch 10 of this series.
>
> Ah, I see. Thanks. Then I'm somewhat wondering is really
> invalidate_inode_pages2_range() safe for THP head pages? At least the:
>
> unmap_mapping_pages(mapping, index, 1, false);
>
> doesn't look adequate for THP head pages... do_launder_page() is also
> doubtful but probably currently OK because THPs cannot be dirty at this
> moment. But how about THPs that are partialy inside start-end range? So far
> the function didn't care because it was operating on page basis so it
> didn't care but now it is probably relevant... At least it would warrant a
> comment in some changelog if you are convinced everything is safe.
You're right, it's inadequate. It's safe to apply this series to the
mainline as-is because the only filesystem which creates THP today
is tmpfs and it won't call invalidate_inode_pages2_range() (afaics).
I have a followup patch which isn't part of this series which fixes it:
http://git.infradead.org/users/willy/pagecache.git/commitdiff/364283163847d1c106463223b858308c730592a1
There are other similar fixes which are also needed before we get to
allowing THPs to be added to the page cache for normal filesystems,
but I think I'm out of time for this merge window. My plan is to submit
everything that's in this tree for the next merge window; the page cache
pieces through Andrew and the filesystem/iomap/XFS pieces through Darrick.
With good luck, all of this will be in place for 5.11. I have about 50
patches in flight for 5.10 and it'll be about another 50 for 5.11.
Reviewer bandwidth is probably the biggest issue right now, and I
really appreciate your thoughtful comments.
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 12/12] mm/filemap: Return only head pages from find_get_entries
2020-09-30 17:23 ` Matthew Wilcox
@ 2020-10-01 7:17 ` Jan Kara
2020-10-25 23:19 ` Matthew Wilcox
0 siblings, 1 reply; 38+ messages in thread
From: Jan Kara @ 2020-10-01 7:17 UTC (permalink / raw)
To: Matthew Wilcox
Cc: Jan Kara, linux-mm, Andrew Morton, Hugh Dickins,
William Kucharski, Johannes Weiner, Yang Shi, Dave Chinner,
linux-kernel
On Wed 30-09-20 18:23:21, Matthew Wilcox wrote:
> On Wed, Sep 30, 2020 at 07:08:07PM +0200, Jan Kara wrote:
> > On Wed 30-09-20 13:36:37, Matthew Wilcox wrote:
> > > On Wed, Sep 30, 2020 at 02:15:12PM +0200, Jan Kara wrote:
> > > > On Mon 14-09-20 14:00:42, Matthew Wilcox (Oracle) wrote:
> > > > > All callers now expect head (and base) pages, and can handle multiple
> > > > > head pages in a single batch, so make find_get_entries() behave that way.
> > > > > Also take the opportunity to make it use the pagevec infrastructure
> > > > > instead of open-coding how pvecs behave. This has the side-effect of
> > > > > being able to append to a pagevec with existing contents, although we
> > > > > don't make use of that functionality anywhere yet.
> > > > >
> > > > > Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
> > > >
> > > > Looks good to me. You can add:
> > > >
> > > > Reviewed-by: Jan Kara <jack@suse.cz>
> > > >
> > > > I'm just curious: What has happened to pagevec_lookup_entries() call in
> > > > invalidate_inode_pages2_range()? Your series appears to be based on a tree
> > > > where the call already does not exist...
> > >
> > > That went away in patch 10 of this series.
> >
> > Ah, I see. Thanks. Then I'm somewhat wondering is really
> > invalidate_inode_pages2_range() safe for THP head pages? At least the:
> >
> > unmap_mapping_pages(mapping, index, 1, false);
> >
> > doesn't look adequate for THP head pages... do_launder_page() is also
> > doubtful but probably currently OK because THPs cannot be dirty at this
> > moment. But how about THPs that are partialy inside start-end range? So far
> > the function didn't care because it was operating on page basis so it
> > didn't care but now it is probably relevant... At least it would warrant a
> > comment in some changelog if you are convinced everything is safe.
>
> You're right, it's inadequate. It's safe to apply this series to the
> mainline as-is because the only filesystem which creates THP today
> is tmpfs and it won't call invalidate_inode_pages2_range() (afaics).
Yeah, correct.
> I have a followup patch which isn't part of this series which fixes it:
>
> http://git.infradead.org/users/willy/pagecache.git/commitdiff/364283163847d1c106463223b858308c730592a1
Yeah, that looks good. How about partial THPs? The way you've implemented
it we will now possibly evict more than strictly required. But OTOH
evicting exactly may require THP split which is a bit unfortunate. But
probably still a better option because otherwise we could have pages being
repeatedly brought in and out of cache just because e.g. workload mixes
direct and buffered IO and is not aligned to THP boundary (and although I
find loads mixing buffered and direct IO to the same file badly designed,
I know for a fact that they do exist and if the file ranges are not
overlapping, it is not that insane design).
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 12/12] mm/filemap: Return only head pages from find_get_entries
2020-10-01 7:17 ` Jan Kara
@ 2020-10-25 23:19 ` Matthew Wilcox
2020-10-26 9:11 ` Jan Kara
0 siblings, 1 reply; 38+ messages in thread
From: Matthew Wilcox @ 2020-10-25 23:19 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, Andrew Morton, Hugh Dickins, William Kucharski,
Johannes Weiner, Yang Shi, Dave Chinner, linux-kernel
On Thu, Oct 01, 2020 at 09:17:28AM +0200, Jan Kara wrote:
> > I have a followup patch which isn't part of this series which fixes it:
> >
> > http://git.infradead.org/users/willy/pagecache.git/commitdiff/364283163847d1c106463223b858308c730592a1
>
> Yeah, that looks good. How about partial THPs? The way you've implemented
> it we will now possibly evict more than strictly required. But OTOH
> evicting exactly may require THP split which is a bit unfortunate. But
> probably still a better option because otherwise we could have pages being
> repeatedly brought in and out of cache just because e.g. workload mixes
> direct and buffered IO and is not aligned to THP boundary (and although I
> find loads mixing buffered and direct IO to the same file badly designed,
> I know for a fact that they do exist and if the file ranges are not
> overlapping, it is not that insane design).
Sorry, forgot to reply to this.
In this patchset, THPs are created by readahead. We always start
out by allocating order-0 pages and only ramp up after hitting a page
marked as PageReadahead. So it's not like tmpfs where we'll try to jump
straight to order-9 pages and have to worry about the behaviour you're
describing above. That means in this kind of scenario, we might have,
eg, an order-6 page in the cache, remove the whole thing, then bring
back in some order-0 pages. If we hit on those, we'll bring in some
order-2 pages. We won't bring in order-6 pages again until we've hit
in the readahead window twice more.
I think the ramp-up is probably too aggressive, but it's fun for testing.
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 12/12] mm/filemap: Return only head pages from find_get_entries
2020-10-25 23:19 ` Matthew Wilcox
@ 2020-10-26 9:11 ` Jan Kara
0 siblings, 0 replies; 38+ messages in thread
From: Jan Kara @ 2020-10-26 9:11 UTC (permalink / raw)
To: Matthew Wilcox
Cc: Jan Kara, linux-mm, Andrew Morton, Hugh Dickins,
William Kucharski, Johannes Weiner, Yang Shi, Dave Chinner,
linux-kernel
On Sun 25-10-20 23:19:34, Matthew Wilcox wrote:
> On Thu, Oct 01, 2020 at 09:17:28AM +0200, Jan Kara wrote:
> > > I have a followup patch which isn't part of this series which fixes it:
> > >
> > > http://git.infradead.org/users/willy/pagecache.git/commitdiff/364283163847d1c106463223b858308c730592a1
> >
> > Yeah, that looks good. How about partial THPs? The way you've implemented
> > it we will now possibly evict more than strictly required. But OTOH
> > evicting exactly may require THP split which is a bit unfortunate. But
> > probably still a better option because otherwise we could have pages being
> > repeatedly brought in and out of cache just because e.g. workload mixes
> > direct and buffered IO and is not aligned to THP boundary (and although I
> > find loads mixing buffered and direct IO to the same file badly designed,
> > I know for a fact that they do exist and if the file ranges are not
> > overlapping, it is not that insane design).
>
> Sorry, forgot to reply to this.
>
> In this patchset, THPs are created by readahead. We always start
> out by allocating order-0 pages and only ramp up after hitting a page
> marked as PageReadahead. So it's not like tmpfs where we'll try to jump
> straight to order-9 pages and have to worry about the behaviour you're
> describing above. That means in this kind of scenario, we might have,
> eg, an order-6 page in the cache, remove the whole thing, then bring
> back in some order-0 pages. If we hit on those, we'll bring in some
> order-2 pages. We won't bring in order-6 pages again until we've hit
> in the readahead window twice more.
>
> I think the ramp-up is probably too aggressive, but it's fun for testing.
OK, sounds good then. Thanks for explanation.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 00/12] Overhaul multi-page lookups for THP
2020-09-14 13:00 [PATCH v2 00/12] Overhaul multi-page lookups for THP Matthew Wilcox (Oracle)
` (11 preceding siblings ...)
2020-09-14 13:00 ` [PATCH v2 12/12] mm/filemap: Return only head pages from find_get_entries Matthew Wilcox (Oracle)
@ 2020-09-28 20:13 ` Matthew Wilcox
2020-09-29 8:50 ` William Kucharski
13 siblings, 0 replies; 38+ messages in thread
From: Matthew Wilcox @ 2020-09-28 20:13 UTC (permalink / raw)
To: linux-mm
Cc: Andrew Morton, Hugh Dickins, William Kucharski, Johannes Weiner,
Jan Kara, Yang Shi, Dave Chinner, linux-kernel
ping
On Mon, Sep 14, 2020 at 02:00:30PM +0100, Matthew Wilcox (Oracle) wrote:
> The critical patch to review here is patch 11, "Handle truncates that
> split THPs". This code is shared with shmem, and while xfstests passes
> (both with the tmpfs filesystem and with THPs enabled for XFS), it is
> terribly subtle.
>
> I posted a similar patch series a few weeks ago [1], but this goes a few
> steps further than that one did. In addition to the unification of
> find_get_entries() and pagevec_lookup_entries(), this patch series
> includes:
>
> - Only return the head pages from tagged lookups
> - Factor a lot of common code out of the various batch lookup routines
> - Add mapping_seek_hole_data()
> - Only return head pages from find_get_entries
>
> I also have a patch to iomap to use mapping_seek_hole_data(), but I'm
> not including that as part of this batch of patches -- I'll send it
> through the iomap tree once mapping_seek_hole_data() lands upstream.
>
> [1] https://lore.kernel.org/linux-mm/20200819184850.24779-1-willy@infradead.org/
>
> Matthew Wilcox (Oracle) (12):
> mm: Make pagecache tagged lookups return only head pages
> mm/shmem: Use pagevec_lookup in shmem_unlock_mapping
> mm/filemap: Add helper for finding pages
> mm/filemap: Add mapping_seek_hole_data
> mm: Add and use find_lock_entries
> mm: Add an 'end' parameter to find_get_entries
> mm: Add an 'end' parameter to pagevec_lookup_entries
> mm: Remove nr_entries parameter from pagevec_lookup_entries
> mm: Pass pvec directly to find_get_entries
> mm: Remove pagevec_lookup_entries
> mm/truncate,shmem: Handle truncates that split THPs
> mm/filemap: Return only head pages from find_get_entries
>
> include/linux/pagemap.h | 5 +-
> include/linux/pagevec.h | 4 -
> mm/filemap.c | 267 +++++++++++++++++++++++++++-------------
> mm/internal.h | 5 +
> mm/shmem.c | 214 +++++++-------------------------
> mm/swap.c | 38 +-----
> mm/truncate.c | 249 ++++++++++++++-----------------------
> 7 files changed, 329 insertions(+), 453 deletions(-)
>
> --
> 2.28.0
>
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: [PATCH v2 00/12] Overhaul multi-page lookups for THP
2020-09-14 13:00 [PATCH v2 00/12] Overhaul multi-page lookups for THP Matthew Wilcox (Oracle)
` (12 preceding siblings ...)
2020-09-28 20:13 ` [PATCH v2 00/12] Overhaul multi-page lookups for THP Matthew Wilcox
@ 2020-09-29 8:50 ` William Kucharski
13 siblings, 0 replies; 38+ messages in thread
From: William Kucharski @ 2020-09-29 8:50 UTC (permalink / raw)
To: Matthew Wilcox (Oracle)
Cc: linux-mm, Andrew Morton, Hugh Dickins, Johannes Weiner, Jan Kara,
Yang Shi, Dave Chinner, linux-kernel
Looks good to me; I really like the addition of the "end" parameter to
find_get_entries() and the conversion of pagevec_lookup_entries().\x03\x03
For the series:
Reviewed-by: William Kucharski <william.kucharski@oracle.com>
> On Sep 14, 2020, at 7:00 AM, Matthew Wilcox (Oracle) <willy@infradead.org> wrote:
>
> The critical patch to review here is patch 11, "Handle truncates that
> split THPs". This code is shared with shmem, and while xfstests passes
> (both with the tmpfs filesystem and with THPs enabled for XFS), it is
> terribly subtle.
>
> I posted a similar patch series a few weeks ago [1], but this goes a few
> steps further than that one did. In addition to the unification of
> find_get_entries() and pagevec_lookup_entries(), this patch series
> includes:
>
> - Only return the head pages from tagged lookups
> - Factor a lot of common code out of the various batch lookup routines
> - Add mapping_seek_hole_data()
> - Only return head pages from find_get_entries
>
> I also have a patch to iomap to use mapping_seek_hole_data(), but I'm
> not including that as part of this batch of patches -- I'll send it
> through the iomap tree once mapping_seek_hole_data() lands upstream.
>
> [1] https://lore.kernel.org/linux-mm/20200819184850.24779-1-willy@infradead.org/
>
> Matthew Wilcox (Oracle) (12):
> mm: Make pagecache tagged lookups return only head pages
> mm/shmem: Use pagevec_lookup in shmem_unlock_mapping
> mm/filemap: Add helper for finding pages
> mm/filemap: Add mapping_seek_hole_data
> mm: Add and use find_lock_entries
> mm: Add an 'end' parameter to find_get_entries
> mm: Add an 'end' parameter to pagevec_lookup_entries
> mm: Remove nr_entries parameter from pagevec_lookup_entries
> mm: Pass pvec directly to find_get_entries
> mm: Remove pagevec_lookup_entries
> mm/truncate,shmem: Handle truncates that split THPs
> mm/filemap: Return only head pages from find_get_entries
>
> include/linux/pagemap.h | 5 +-
> include/linux/pagevec.h | 4 -
> mm/filemap.c | 267 +++++++++++++++++++++++++++-------------
> mm/internal.h | 5 +
> mm/shmem.c | 214 +++++++-------------------------
> mm/swap.c | 38 +-----
> mm/truncate.c | 249 ++++++++++++++-----------------------
> 7 files changed, 329 insertions(+), 453 deletions(-)
>
> --
> 2.28.0
>
^ permalink raw reply [flat|nested] 38+ messages in thread