From: Michal Hocko <mhocko@kernel.org>
To: Minchan Kim <minchan@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>,
LKML <linux-kernel@vger.kernel.org>,
linux-mm <linux-mm@kvack.org>,
Johannes Weiner <hannes@cmpxchg.org>,
Tim Murray <timmurray@google.com>,
Joel Fernandes <joel@joelfernandes.org>,
Suren Baghdasaryan <surenb@google.com>,
Daniel Colascione <dancol@google.com>,
Shakeel Butt <shakeelb@google.com>,
Sonny Rao <sonnyrao@google.com>,
Brian Geffon <bgeffon@google.com>,
linux-api@vger.kernel.org
Subject: Re: [RFC 3/7] mm: introduce MADV_COLD
Date: Mon, 20 May 2019 10:27:03 +0200 [thread overview]
Message-ID: <20190520082703.GX6836@dhcp22.suse.cz> (raw)
In-Reply-To: <20190520035254.57579-4-minchan@kernel.org>
[Cc linux-api]
On Mon 20-05-19 12:52:50, Minchan Kim wrote:
> When a process expects no accesses to a certain memory range
> for a long time, it could hint kernel that the pages can be
> reclaimed instantly but data should be preserved for future use.
> This could reduce workingset eviction so it ends up increasing
> performance.
>
> This patch introduces the new MADV_COLD hint to madvise(2)
> syscall. MADV_COLD can be used by a process to mark a memory range
> as not expected to be used for a long time. The hint can help
> kernel in deciding which pages to evict proactively.
As mentioned in other email this looks like a non-destructive
MADV_DONTNEED alternative.
> Internally, it works via reclaiming memory in process context
> the syscall is called. If the page is dirty but backing storage
> is not synchronous device, the written page will be rotate back
> into LRU's tail once the write is done so they will reclaim easily
> when memory pressure happens. If backing storage is
> synchrnous device(e.g., zram), hte page will be reclaimed instantly.
Why do we special case async backing storage? Please always try to
explain _why_ the decision is made.
I haven't checked the implementation yet so I cannot comment on that.
> Signed-off-by: Minchan Kim <minchan@kernel.org>
> ---
> include/linux/swap.h | 1 +
> include/uapi/asm-generic/mman-common.h | 1 +
> mm/madvise.c | 123 +++++++++++++++++++++++++
> mm/vmscan.c | 74 +++++++++++++++
> 4 files changed, 199 insertions(+)
>
> diff --git a/include/linux/swap.h b/include/linux/swap.h
> index 64795abea003..7f32a948fc6a 100644
> --- a/include/linux/swap.h
> +++ b/include/linux/swap.h
> @@ -365,6 +365,7 @@ extern int vm_swappiness;
> extern int remove_mapping(struct address_space *mapping, struct page *page);
> extern unsigned long vm_total_pages;
>
> +extern unsigned long reclaim_pages(struct list_head *page_list);
> #ifdef CONFIG_NUMA
> extern int node_reclaim_mode;
> extern int sysctl_min_unmapped_ratio;
> diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h
> index f7a4a5d4b642..b9b51eeb8e1a 100644
> --- a/include/uapi/asm-generic/mman-common.h
> +++ b/include/uapi/asm-generic/mman-common.h
> @@ -43,6 +43,7 @@
> #define MADV_WILLNEED 3 /* will need these pages */
> #define MADV_DONTNEED 4 /* don't need these pages */
> #define MADV_COOL 5 /* deactivatie these pages */
> +#define MADV_COLD 6 /* reclaim these pages */
>
> /* common parameters: try to keep these consistent across architectures */
> #define MADV_FREE 8 /* free pages only if memory pressure */
> diff --git a/mm/madvise.c b/mm/madvise.c
> index c05817fb570d..9a6698b56845 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -42,6 +42,7 @@ static int madvise_need_mmap_write(int behavior)
> case MADV_WILLNEED:
> case MADV_DONTNEED:
> case MADV_COOL:
> + case MADV_COLD:
> case MADV_FREE:
> return 0;
> default:
> @@ -416,6 +417,125 @@ static long madvise_cool(struct vm_area_struct *vma,
> return 0;
> }
>
> +static int madvise_cold_pte_range(pmd_t *pmd, unsigned long addr,
> + unsigned long end, struct mm_walk *walk)
> +{
> + pte_t *orig_pte, *pte, ptent;
> + spinlock_t *ptl;
> + LIST_HEAD(page_list);
> + struct page *page;
> + int isolated = 0;
> + struct vm_area_struct *vma = walk->vma;
> + unsigned long next;
> +
> + next = pmd_addr_end(addr, end);
> + if (pmd_trans_huge(*pmd)) {
> + spinlock_t *ptl;
> +
> + ptl = pmd_trans_huge_lock(pmd, vma);
> + if (!ptl)
> + return 0;
> +
> + if (is_huge_zero_pmd(*pmd))
> + goto huge_unlock;
> +
> + page = pmd_page(*pmd);
> + if (page_mapcount(page) > 1)
> + goto huge_unlock;
> +
> + if (next - addr != HPAGE_PMD_SIZE) {
> + int err;
> +
> + get_page(page);
> + spin_unlock(ptl);
> + lock_page(page);
> + err = split_huge_page(page);
> + unlock_page(page);
> + put_page(page);
> + if (!err)
> + goto regular_page;
> + return 0;
> + }
> +
> + if (isolate_lru_page(page))
> + goto huge_unlock;
> +
> + list_add(&page->lru, &page_list);
> +huge_unlock:
> + spin_unlock(ptl);
> + reclaim_pages(&page_list);
> + return 0;
> + }
> +
> + if (pmd_trans_unstable(pmd))
> + return 0;
> +regular_page:
> + orig_pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
> + for (pte = orig_pte; addr < end; pte++, addr += PAGE_SIZE) {
> + ptent = *pte;
> + if (!pte_present(ptent))
> + continue;
> +
> + page = vm_normal_page(vma, addr, ptent);
> + if (!page)
> + continue;
> +
> + if (page_mapcount(page) > 1)
> + continue;
> +
> + if (isolate_lru_page(page))
> + continue;
> +
> + isolated++;
> + list_add(&page->lru, &page_list);
> + if (isolated >= SWAP_CLUSTER_MAX) {
> + pte_unmap_unlock(orig_pte, ptl);
> + reclaim_pages(&page_list);
> + isolated = 0;
> + pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
> + orig_pte = pte;
> + }
> + }
> +
> + pte_unmap_unlock(orig_pte, ptl);
> + reclaim_pages(&page_list);
> + cond_resched();
> +
> + return 0;
> +}
> +
> +static void madvise_cold_page_range(struct mmu_gather *tlb,
> + struct vm_area_struct *vma,
> + unsigned long addr, unsigned long end)
> +{
> + struct mm_walk warm_walk = {
> + .pmd_entry = madvise_cold_pte_range,
> + .mm = vma->vm_mm,
> + };
> +
> + tlb_start_vma(tlb, vma);
> + walk_page_range(addr, end, &warm_walk);
> + tlb_end_vma(tlb, vma);
> +}
> +
> +
> +static long madvise_cold(struct vm_area_struct *vma,
> + unsigned long start_addr, unsigned long end_addr)
> +{
> + struct mm_struct *mm = vma->vm_mm;
> + struct mmu_gather tlb;
> +
> + if (vma->vm_flags & (VM_LOCKED|VM_HUGETLB|VM_PFNMAP))
> + return -EINVAL;
> +
> + lru_add_drain();
> + tlb_gather_mmu(&tlb, mm, start_addr, end_addr);
> + madvise_cold_page_range(&tlb, vma, start_addr, end_addr);
> + tlb_finish_mmu(&tlb, start_addr, end_addr);
> +
> + return 0;
> +}
> +
> static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,
> unsigned long end, struct mm_walk *walk)
>
> @@ -806,6 +926,8 @@ madvise_vma(struct vm_area_struct *vma, struct vm_area_struct **prev,
> return madvise_willneed(vma, prev, start, end);
> case MADV_COOL:
> return madvise_cool(vma, start, end);
> + case MADV_COLD:
> + return madvise_cold(vma, start, end);
> case MADV_FREE:
> case MADV_DONTNEED:
> return madvise_dontneed_free(vma, prev, start, end, behavior);
> @@ -828,6 +950,7 @@ madvise_behavior_valid(int behavior)
> case MADV_DONTNEED:
> case MADV_FREE:
> case MADV_COOL:
> + case MADV_COLD:
> #ifdef CONFIG_KSM
> case MADV_MERGEABLE:
> case MADV_UNMERGEABLE:
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index a28e5d17b495..1701b31f70a8 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -2096,6 +2096,80 @@ static void shrink_active_list(unsigned long nr_to_scan,
> nr_deactivate, nr_rotated, sc->priority, file);
> }
>
> +unsigned long reclaim_pages(struct list_head *page_list)
> +{
> + int nid = -1;
> + unsigned long nr_isolated[2] = {0, };
> + unsigned long nr_reclaimed = 0;
> + LIST_HEAD(node_page_list);
> + struct reclaim_stat dummy_stat;
> + struct scan_control sc = {
> + .gfp_mask = GFP_KERNEL,
> + .priority = DEF_PRIORITY,
> + .may_writepage = 1,
> + .may_unmap = 1,
> + .may_swap = 1,
> + };
> +
> + while (!list_empty(page_list)) {
> + struct page *page;
> +
> + page = lru_to_page(page_list);
> + list_del(&page->lru);
> +
> + if (nid == -1) {
> + nid = page_to_nid(page);
> + INIT_LIST_HEAD(&node_page_list);
> + nr_isolated[0] = nr_isolated[1] = 0;
> + }
> +
> + if (nid == page_to_nid(page)) {
> + list_add(&page->lru, &node_page_list);
> + nr_isolated[!!page_is_file_cache(page)] +=
> + hpage_nr_pages(page);
> + continue;
> + }
> +
> + nid = page_to_nid(page);
> +
> + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_ANON,
> + nr_isolated[0]);
> + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_FILE,
> + nr_isolated[1]);
> + nr_reclaimed += shrink_page_list(&node_page_list,
> + NODE_DATA(nid), &sc, TTU_IGNORE_ACCESS,
> + &dummy_stat, true);
> + while (!list_empty(&node_page_list)) {
> + struct page *page = lru_to_page(page_list);
> +
> + list_del(&page->lru);
> + putback_lru_page(page);
> + }
> + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_ANON,
> + -nr_isolated[0]);
> + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_FILE,
> + -nr_isolated[1]);
> + nr_isolated[0] = nr_isolated[1] = 0;
> + INIT_LIST_HEAD(&node_page_list);
> + }
> +
> + if (!list_empty(&node_page_list)) {
> + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_ANON,
> + nr_isolated[0]);
> + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_FILE,
> + nr_isolated[1]);
> + nr_reclaimed += shrink_page_list(&node_page_list,
> + NODE_DATA(nid), &sc, TTU_IGNORE_ACCESS,
> + &dummy_stat, true);
> + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_ANON,
> + -nr_isolated[0]);
> + mod_node_page_state(NODE_DATA(nid), NR_ISOLATED_FILE,
> + -nr_isolated[1]);
> + }
> +
> + return nr_reclaimed;
> +}
> +
> /*
> * The inactive anon list should be small enough that the VM never has
> * to do too much work.
> --
> 2.21.0.1020.gf2820cf01a-goog
>
--
Michal Hocko
SUSE Labs
next prev parent reply other threads:[~2019-05-20 8:27 UTC|newest]
Thread overview: 138+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-20 3:52 [RFC 0/7] introduce memory hinting API for external process Minchan Kim
2019-05-20 3:52 ` [RFC 1/7] mm: introduce MADV_COOL Minchan Kim
2019-05-20 8:16 ` Michal Hocko
2019-05-20 8:19 ` Michal Hocko
2019-05-20 15:08 ` Suren Baghdasaryan
2019-05-20 22:55 ` Minchan Kim
2019-05-20 22:54 ` Minchan Kim
2019-05-21 6:04 ` Michal Hocko
2019-05-21 9:11 ` Minchan Kim
2019-05-21 10:05 ` Michal Hocko
2019-05-28 8:53 ` Hillf Danton
2019-05-28 10:58 ` Minchan Kim
2019-05-20 3:52 ` [RFC 2/7] mm: change PAGEREF_RECLAIM_CLEAN with PAGE_REFRECLAIM Minchan Kim
2019-05-20 16:50 ` Johannes Weiner
2019-05-20 22:57 ` Minchan Kim
2019-05-20 3:52 ` [RFC 3/7] mm: introduce MADV_COLD Minchan Kim
2019-05-20 8:27 ` Michal Hocko [this message]
2019-05-20 23:00 ` Minchan Kim
2019-05-21 6:08 ` Michal Hocko
2019-05-21 9:13 ` Minchan Kim
2019-05-28 14:54 ` Hillf Danton
2019-05-30 0:45 ` Minchan Kim
2019-05-20 3:52 ` [RFC 4/7] mm: factor out madvise's core functionality Minchan Kim
2019-05-20 14:26 ` Oleksandr Natalenko
2019-05-21 1:26 ` Minchan Kim
2019-05-21 6:36 ` Oleksandr Natalenko
2019-05-21 6:50 ` Michal Hocko
2019-05-21 7:06 ` Oleksandr Natalenko
2019-05-21 10:52 ` Minchan Kim
2019-05-21 11:00 ` Michal Hocko
2019-05-21 11:24 ` Minchan Kim
2019-05-21 11:32 ` Michal Hocko
2019-05-21 10:49 ` Minchan Kim
2019-05-21 10:55 ` Michal Hocko
2019-05-20 3:52 ` [RFC 5/7] mm: introduce external memory hinting API Minchan Kim
2019-05-20 9:18 ` Michal Hocko
2019-05-21 2:41 ` Minchan Kim
2019-05-21 6:17 ` Michal Hocko
2019-05-21 10:32 ` Minchan Kim
2019-05-21 9:01 ` Christian Brauner
2019-05-21 11:35 ` Minchan Kim
2019-05-21 11:51 ` Christian Brauner
2019-05-21 15:31 ` Oleg Nesterov
2019-05-27 7:43 ` Minchan Kim
2019-05-27 15:12 ` Oleg Nesterov
2019-05-27 23:33 ` Minchan Kim
2019-05-28 7:23 ` Michal Hocko
2019-05-29 3:41 ` Hillf Danton
2019-05-30 0:38 ` Minchan Kim
2019-05-20 3:52 ` [RFC 6/7] mm: extend process_madvise syscall to support vector arrary Minchan Kim
2019-05-20 9:22 ` Michal Hocko
2019-05-21 2:48 ` Minchan Kim
2019-05-21 6:24 ` Michal Hocko
2019-05-21 10:26 ` Minchan Kim
2019-05-21 10:37 ` Michal Hocko
2019-05-27 7:49 ` Minchan Kim
2019-05-29 10:08 ` Daniel Colascione
2019-05-29 10:33 ` Michal Hocko
2019-05-30 2:17 ` Minchan Kim
2019-05-30 6:57 ` Michal Hocko
2019-05-30 8:02 ` Minchan Kim
2019-05-30 16:19 ` Daniel Colascione
2019-05-30 18:47 ` Michal Hocko
2019-05-29 4:14 ` Hillf Danton
2019-05-30 0:35 ` Minchan Kim
2019-05-20 3:52 ` [RFC 7/7] mm: madvise support MADV_ANONYMOUS_FILTER and MADV_FILE_FILTER Minchan Kim
2019-05-20 9:28 ` Michal Hocko
2019-05-21 2:55 ` Minchan Kim
2019-05-21 6:26 ` Michal Hocko
2019-05-27 7:58 ` Minchan Kim
2019-05-27 12:44 ` Michal Hocko
2019-05-28 3:26 ` Minchan Kim
2019-05-28 6:29 ` Michal Hocko
2019-05-28 8:13 ` Minchan Kim
2019-05-28 8:31 ` Daniel Colascione
2019-05-28 8:49 ` Minchan Kim
2019-05-28 9:08 ` Michal Hocko
2019-05-28 9:39 ` Daniel Colascione
2019-05-28 10:33 ` Michal Hocko
2019-05-28 11:21 ` Daniel Colascione
2019-05-28 11:49 ` Michal Hocko
2019-05-28 12:11 ` Daniel Colascione
2019-05-28 12:32 ` Michal Hocko
2019-05-28 10:32 ` Minchan Kim
2019-05-28 10:41 ` Michal Hocko
2019-05-28 11:12 ` Minchan Kim
2019-05-28 11:28 ` Michal Hocko
2019-05-28 11:42 ` Daniel Colascione
2019-05-28 11:56 ` Michal Hocko
2019-05-28 12:18 ` Daniel Colascione
2019-05-28 12:38 ` Michal Hocko
2019-05-28 12:10 ` Minchan Kim
2019-05-28 11:44 ` Minchan Kim
2019-05-28 11:51 ` Daniel Colascione
2019-05-28 12:06 ` Michal Hocko
2019-05-28 12:22 ` Minchan Kim
2019-05-28 11:28 ` Daniel Colascione
2019-05-21 15:33 ` Johannes Weiner
2019-05-22 1:50 ` Minchan Kim
2019-05-29 4:36 ` Hillf Danton
2019-05-30 1:00 ` Minchan Kim
2019-05-20 6:37 ` [RFC 0/7] introduce memory hinting API for external process Anshuman Khandual
2019-05-20 16:59 ` Tim Murray
2019-05-21 2:55 ` Anshuman Khandual
2019-05-21 5:14 ` Minchan Kim
2019-05-21 10:34 ` Michal Hocko
2019-05-28 10:50 ` Anshuman Khandual
2019-05-21 12:56 ` Shakeel Butt
2019-05-22 4:15 ` Brian Geffon
2019-05-22 4:23 ` Brian Geffon
2019-05-20 9:28 ` Michal Hocko
2019-05-20 14:42 ` Oleksandr Natalenko
2019-05-21 2:56 ` Minchan Kim
2019-05-20 16:46 ` Johannes Weiner
2019-05-21 4:39 ` Minchan Kim
2019-05-21 6:32 ` Michal Hocko
2019-05-21 1:44 ` Matthew Wilcox
2019-05-21 5:01 ` Minchan Kim
2019-05-21 6:34 ` Michal Hocko
2019-05-21 8:42 ` Christian Brauner
2019-05-21 11:05 ` Minchan Kim
2019-05-21 11:30 ` Christian Brauner
2019-05-21 11:39 ` Christian Brauner
2019-05-22 5:11 ` Daniel Colascione
2019-05-22 8:22 ` Christian Brauner
2019-05-22 13:16 ` Daniel Colascione
2019-05-22 14:52 ` Christian Brauner
2019-05-22 15:17 ` Daniel Colascione
2019-05-22 15:48 ` Christian Brauner
2019-05-22 15:57 ` Daniel Colascione
2019-05-22 16:01 ` Christian Brauner
2019-05-22 16:01 ` Daniel Colascione
2019-05-23 13:07 ` Minchan Kim
2019-05-27 8:06 ` Minchan Kim
2019-05-21 11:41 ` Minchan Kim
2019-05-21 12:04 ` Christian Brauner
2019-05-21 12:15 ` Oleksandr Natalenko
2019-05-21 12:53 ` Shakeel Butt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190520082703.GX6836@dhcp22.suse.cz \
--to=mhocko@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=bgeffon@google.com \
--cc=dancol@google.com \
--cc=hannes@cmpxchg.org \
--cc=joel@joelfernandes.org \
--cc=linux-api@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=minchan@kernel.org \
--cc=shakeelb@google.com \
--cc=sonnyrao@google.com \
--cc=surenb@google.com \
--cc=timmurray@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).