Re: + mm-reclaim-madv_free-pages.patch added to -mm tree

Re: + mm-reclaim-madv_free-pages.patch added to -mm tree

[Minchan Kim]

Hi,

On Tue, Feb 28, 2017 at 04:32:38PM -0800, akpm@linux-foundation.org wrote:
> 
> The patch titled
>      Subject: mm: reclaim MADV_FREE pages
> has been added to the -mm tree.  Its filename is
>      mm-reclaim-madv_free-pages.patch
> 
> This patch should soon appear at
>     http://ozlabs.org/~akpm/mmots/broken-out/mm-reclaim-madv_free-pages.patch
> and later at
>     http://ozlabs.org/~akpm/mmotm/broken-out/mm-reclaim-madv_free-pages.patch
> 
> Before you just go and hit "reply", please:
>    a) Consider who else should be cc'ed
>    b) Prefer to cc a suitable mailing list as well
>    c) Ideally: find the original patch on the mailing list and do a
>       reply-to-all to that, adding suitable additional cc's
> 
> *** Remember to use Documentation/SubmitChecklist when testing your code ***
> 
> The -mm tree is included into linux-next and is updated
> there every 3-4 working days
> 
> ------------------------------------------------------
> From: Shaohua Li <shli@fb.com>
> Subject: mm: reclaim MADV_FREE pages
> 
> When memory pressure is high, we free MADV_FREE pages.  If the pages are
> not dirty in pte, the pages could be freed immediately.  Otherwise we
> can't reclaim them.  We put the pages back to anonumous LRU list (by
> setting SwapBacked flag) and the pages will be reclaimed in normal swapout
> way.
> 
> We use normal page reclaim policy.  Since MADV_FREE pages are put into
> inactive file list, such pages and inactive file pages are reclaimed
> according to their age.  This is expected, because we don't want to
> reclaim too many MADV_FREE pages before used once pages.
> 
> Based on Minchan's original patch
> 
> Link: http://lkml.kernel.org/r/14b8eb1d3f6bf6cc492833f183ac8c304e560484.1487965799.git.shli@fb.com
> Signed-off-by: Shaohua Li <shli@fb.com>
> Acked-by: Minchan Kim <minchan@kernel.org>
> Acked-by: Michal Hocko <mhocko@suse.com>
> Acked-by: Johannes Weiner <hannes@cmpxchg.org>
> Acked-by: Hillf Danton <hillf.zj@alibaba-inc.com>
> Cc: Hugh Dickins <hughd@google.com>
> Cc: Rik van Riel <riel@redhat.com>
> Cc: Mel Gorman <mgorman@techsingularity.net>
> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
> ---

< snip >

> @@ -1419,11 +1413,21 @@ static int try_to_unmap_one(struct page
>  			VM_BUG_ON_PAGE(!PageSwapCache(page) && PageSwapBacked(page),
>  				page);
>  
> -			if (!PageDirty(page)) {
> +			/*
> +			 * swapin page could be clean, it has data stored in
> +			 * swap. We can't silently discard it without setting
> +			 * swap entry in the page table.
> +			 */
> +			if (!PageDirty(page) && !PageSwapCache(page)) {
>  				/* It's a freeable page by MADV_FREE */
>  				dec_mm_counter(mm, MM_ANONPAGES);
> -				rp->lazyfreed++;
>  				goto discard;
> +			} else if (!PageSwapBacked(page)) {
> +				/* dirty MADV_FREE page */
> +				set_pte_at(mm, address, pvmw.pte, pteval);
> +				ret = SWAP_DIRTY;
> +				page_vma_mapped_walk_done(&pvmw);
> +				break;
>  			}

There is no point to make this logic complicated with clean swapin-page.

Andrew,
Could you fold below patch into the mm-reclaim-madv_free-pages.patch
if others are not against?

Thanks.

>From 0c28f6560fbc4e65da4f4a8cc4664ab9f7b11cf3 Mon Sep 17 00:00:00 2001
From: Minchan Kim <minchan@kernel.org>
Date: Fri, 3 Mar 2017 11:42:52 +0900
Subject: [PATCH] mm: clean up lazyfree page handling

We can make it simple to understand without need to be aware of
clean-swapin page.
This patch just clean up lazyfree page handling in try_to_unmap_one.

Signed-off-by: Minchan Kim <minchan@kernel.org>
---
 mm/rmap.c | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/mm/rmap.c b/mm/rmap.c
index bb45712..f7eab40 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1413,17 +1413,17 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
 			VM_BUG_ON_PAGE(!PageSwapCache(page) && PageSwapBacked(page),
 				page);
 
-			/*
-			 * swapin page could be clean, it has data stored in
-			 * swap. We can't silently discard it without setting
-			 * swap entry in the page table.
-			 */
-			if (!PageDirty(page) && !PageSwapCache(page)) {
-				/* It's a freeable page by MADV_FREE */
-				dec_mm_counter(mm, MM_ANONPAGES);
-				goto discard;
-			} else if (!PageSwapBacked(page)) {
-				/* dirty MADV_FREE page */
+			/* MADV_FREE page check */
+			if (!PageSwapBacked(page)) {
+				if (!PageDirty(page)) {
+					dec_mm_counter(mm, MM_ANONPAGES);
+					goto discard;
+				}
+
+				/*
+				 * If the page was redirtied, it cannot be
+				 * discarded. Remap the page to page table.
+				 */
 				set_pte_at(mm, address, pvmw.pte, pteval);
 				ret = SWAP_DIRTY;
 				page_vma_mapped_walk_done(&pvmw);
-- 
2.7.4

Re: + mm-reclaim-madv_free-pages.patch added to -mm tree

[Johannes Weiner]

On Fri, Mar 03, 2017 at 11:52:37AM +0900, Minchan Kim wrote:
> On Tue, Feb 28, 2017 at 04:32:38PM -0800, akpm@linux-foundation.org wrote:
> > 
> > The patch titled
> >      Subject: mm: reclaim MADV_FREE pages
> > has been added to the -mm tree.  Its filename is
> >      mm-reclaim-madv_free-pages.patch
> > 
> > This patch should soon appear at
> >     http://ozlabs.org/~akpm/mmots/broken-out/mm-reclaim-madv_free-pages.patch
> > and later at
> >     http://ozlabs.org/~akpm/mmotm/broken-out/mm-reclaim-madv_free-pages.patch
> > 
> > Before you just go and hit "reply", please:
> >    a) Consider who else should be cc'ed
> >    b) Prefer to cc a suitable mailing list as well
> >    c) Ideally: find the original patch on the mailing list and do a
> >       reply-to-all to that, adding suitable additional cc's
> > 
> > *** Remember to use Documentation/SubmitChecklist when testing your code ***
> > 
> > The -mm tree is included into linux-next and is updated
> > there every 3-4 working days
> > 
> > ------------------------------------------------------
> > From: Shaohua Li <shli@fb.com>
> > Subject: mm: reclaim MADV_FREE pages
> > 
> > When memory pressure is high, we free MADV_FREE pages.  If the pages are
> > not dirty in pte, the pages could be freed immediately.  Otherwise we
> > can't reclaim them.  We put the pages back to anonumous LRU list (by
> > setting SwapBacked flag) and the pages will be reclaimed in normal swapout
> > way.
> > 
> > We use normal page reclaim policy.  Since MADV_FREE pages are put into
> > inactive file list, such pages and inactive file pages are reclaimed
> > according to their age.  This is expected, because we don't want to
> > reclaim too many MADV_FREE pages before used once pages.
> > 
> > Based on Minchan's original patch
> > 
> > Link: http://lkml.kernel.org/r/14b8eb1d3f6bf6cc492833f183ac8c304e560484.1487965799.git.shli@fb.com
> > Signed-off-by: Shaohua Li <shli@fb.com>
> > Acked-by: Minchan Kim <minchan@kernel.org>
> > Acked-by: Michal Hocko <mhocko@suse.com>
> > Acked-by: Johannes Weiner <hannes@cmpxchg.org>
> > Acked-by: Hillf Danton <hillf.zj@alibaba-inc.com>
> > Cc: Hugh Dickins <hughd@google.com>
> > Cc: Rik van Riel <riel@redhat.com>
> > Cc: Mel Gorman <mgorman@techsingularity.net>
> > Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
> > ---
> 
> < snip >
> 
> > @@ -1419,11 +1413,21 @@ static int try_to_unmap_one(struct page
> >  			VM_BUG_ON_PAGE(!PageSwapCache(page) && PageSwapBacked(page),
> >  				page);
> >  
> > -			if (!PageDirty(page)) {
> > +			/*
> > +			 * swapin page could be clean, it has data stored in
> > +			 * swap. We can't silently discard it without setting
> > +			 * swap entry in the page table.
> > +			 */
> > +			if (!PageDirty(page) && !PageSwapCache(page)) {
> >  				/* It's a freeable page by MADV_FREE */
> >  				dec_mm_counter(mm, MM_ANONPAGES);
> > -				rp->lazyfreed++;
> >  				goto discard;
> > +			} else if (!PageSwapBacked(page)) {
> > +				/* dirty MADV_FREE page */
> > +				set_pte_at(mm, address, pvmw.pte, pteval);
> > +				ret = SWAP_DIRTY;
> > +				page_vma_mapped_walk_done(&pvmw);
> > +				break;
> >  			}
> 
> There is no point to make this logic complicated with clean swapin-page.
> 
> Andrew,
> Could you fold below patch into the mm-reclaim-madv_free-pages.patch
> if others are not against?
> 
> Thanks.
> 
> From 0c28f6560fbc4e65da4f4a8cc4664ab9f7b11cf3 Mon Sep 17 00:00:00 2001
> From: Minchan Kim <minchan@kernel.org>
> Date: Fri, 3 Mar 2017 11:42:52 +0900
> Subject: [PATCH] mm: clean up lazyfree page handling
> 
> We can make it simple to understand without need to be aware of
> clean-swapin page.
> This patch just clean up lazyfree page handling in try_to_unmap_one.
> 
> Signed-off-by: Minchan Kim <minchan@kernel.org>

Agreed, this is a litle easier to follow.

Acked-by: Johannes Weiner <hannes@cmpxchg.org>

> ---
>  mm/rmap.c | 22 +++++++++++-----------
>  1 file changed, 11 insertions(+), 11 deletions(-)
> 
> diff --git a/mm/rmap.c b/mm/rmap.c
> index bb45712..f7eab40 100644
> --- a/mm/rmap.c
> +++ b/mm/rmap.c
> @@ -1413,17 +1413,17 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
>  			VM_BUG_ON_PAGE(!PageSwapCache(page) && PageSwapBacked(page),
>  				page);

Since you're removing the PageSwapCache() check and we're now assuming
that !swapbacked is not in the swapcache, can you modify this to check
PageSwapBacked(page) != PageSwapCache(page)?

Better yet, change it into a warning and SWAP_FAIL.

Re: + mm-reclaim-madv_free-pages.patch added to -mm tree

[Minchan Kim]

On Fri, Mar 03, 2017 at 10:18:51AM -0500, Johannes Weiner wrote:
> On Fri, Mar 03, 2017 at 11:52:37AM +0900, Minchan Kim wrote:
> > On Tue, Feb 28, 2017 at 04:32:38PM -0800, akpm@linux-foundation.org wrote:
> > > 
> > > The patch titled
> > >      Subject: mm: reclaim MADV_FREE pages
> > > has been added to the -mm tree.  Its filename is
> > >      mm-reclaim-madv_free-pages.patch
> > > 
> > > This patch should soon appear at
> > >     http://ozlabs.org/~akpm/mmots/broken-out/mm-reclaim-madv_free-pages.patch
> > > and later at
> > >     http://ozlabs.org/~akpm/mmotm/broken-out/mm-reclaim-madv_free-pages.patch
> > > 
> > > Before you just go and hit "reply", please:
> > >    a) Consider who else should be cc'ed
> > >    b) Prefer to cc a suitable mailing list as well
> > >    c) Ideally: find the original patch on the mailing list and do a
> > >       reply-to-all to that, adding suitable additional cc's
> > > 
> > > *** Remember to use Documentation/SubmitChecklist when testing your code ***
> > > 
> > > The -mm tree is included into linux-next and is updated
> > > there every 3-4 working days
> > > 
> > > ------------------------------------------------------
> > > From: Shaohua Li <shli@fb.com>
> > > Subject: mm: reclaim MADV_FREE pages
> > > 
> > > When memory pressure is high, we free MADV_FREE pages.  If the pages are
> > > not dirty in pte, the pages could be freed immediately.  Otherwise we
> > > can't reclaim them.  We put the pages back to anonumous LRU list (by
> > > setting SwapBacked flag) and the pages will be reclaimed in normal swapout
> > > way.
> > > 
> > > We use normal page reclaim policy.  Since MADV_FREE pages are put into
> > > inactive file list, such pages and inactive file pages are reclaimed
> > > according to their age.  This is expected, because we don't want to
> > > reclaim too many MADV_FREE pages before used once pages.
> > > 
> > > Based on Minchan's original patch
> > > 
> > > Link: http://lkml.kernel.org/r/14b8eb1d3f6bf6cc492833f183ac8c304e560484.1487965799.git.shli@fb.com
> > > Signed-off-by: Shaohua Li <shli@fb.com>
> > > Acked-by: Minchan Kim <minchan@kernel.org>
> > > Acked-by: Michal Hocko <mhocko@suse.com>
> > > Acked-by: Johannes Weiner <hannes@cmpxchg.org>
> > > Acked-by: Hillf Danton <hillf.zj@alibaba-inc.com>
> > > Cc: Hugh Dickins <hughd@google.com>
> > > Cc: Rik van Riel <riel@redhat.com>
> > > Cc: Mel Gorman <mgorman@techsingularity.net>
> > > Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
> > > ---
> > 
> > < snip >
> > 
> > > @@ -1419,11 +1413,21 @@ static int try_to_unmap_one(struct page
> > >  			VM_BUG_ON_PAGE(!PageSwapCache(page) && PageSwapBacked(page),
> > >  				page);
> > >  
> > > -			if (!PageDirty(page)) {
> > > +			/*
> > > +			 * swapin page could be clean, it has data stored in
> > > +			 * swap. We can't silently discard it without setting
> > > +			 * swap entry in the page table.
> > > +			 */
> > > +			if (!PageDirty(page) && !PageSwapCache(page)) {
> > >  				/* It's a freeable page by MADV_FREE */
> > >  				dec_mm_counter(mm, MM_ANONPAGES);
> > > -				rp->lazyfreed++;
> > >  				goto discard;
> > > +			} else if (!PageSwapBacked(page)) {
> > > +				/* dirty MADV_FREE page */
> > > +				set_pte_at(mm, address, pvmw.pte, pteval);
> > > +				ret = SWAP_DIRTY;
> > > +				page_vma_mapped_walk_done(&pvmw);
> > > +				break;
> > >  			}
> > 
> > There is no point to make this logic complicated with clean swapin-page.
> > 
> > Andrew,
> > Could you fold below patch into the mm-reclaim-madv_free-pages.patch
> > if others are not against?
> > 
> > Thanks.
> > 
> > From 0c28f6560fbc4e65da4f4a8cc4664ab9f7b11cf3 Mon Sep 17 00:00:00 2001
> > From: Minchan Kim <minchan@kernel.org>
> > Date: Fri, 3 Mar 2017 11:42:52 +0900
> > Subject: [PATCH] mm: clean up lazyfree page handling
> > 
> > We can make it simple to understand without need to be aware of
> > clean-swapin page.
> > This patch just clean up lazyfree page handling in try_to_unmap_one.
> > 
> > Signed-off-by: Minchan Kim <minchan@kernel.org>
> 
> Agreed, this is a litle easier to follow.
> 
> Acked-by: Johannes Weiner <hannes@cmpxchg.org>

Thanks, Johannes.

> 
> > ---
> >  mm/rmap.c | 22 +++++++++++-----------
> >  1 file changed, 11 insertions(+), 11 deletions(-)
> > 
> > diff --git a/mm/rmap.c b/mm/rmap.c
> > index bb45712..f7eab40 100644
> > --- a/mm/rmap.c
> > +++ b/mm/rmap.c
> > @@ -1413,17 +1413,17 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
> >  			VM_BUG_ON_PAGE(!PageSwapCache(page) && PageSwapBacked(page),
> >  				page);
> 
> Since you're removing the PageSwapCache() check and we're now assuming
> that !swapbacked is not in the swapcache, can you modify this to check
> PageSwapBacked(page) != PageSwapCache(page)?
> 
> Better yet, change it into a warning and SWAP_FAIL.

Maybe, what you wanted is

 !!PageSwapBacked(page) != !!PageSwapCache(page)

Personally, I prefer && style rather than equation expression
in this case.

How about this?
If others are not against, I will resend it to Andrew with
Acked|Reviewed-by all I got until now.

Thanks.

commit 118cfee42600
Author: Minchan Kim <minchan@kernel.org>
Date:   Sat Mar 4 01:01:38 2017 +0000

    mm: clean up lazyfree page handling
    
    We can make it simple to understand without need to be aware of
    clean-swapin page.
    This patch just clean up lazyfree page handling in try_to_unmap_one.
    
    Link: http://lkml.kernel.org/r/20170303025237.GB3503@bbox
    Signed-off-by: Minchan Kim <minchan@kernel.org>
    Cc: Shaohua Li <shli@fb.com>
    Cc: Michal Hocko <mhocko@suse.com>
    Cc: Johannes Weiner <hannes@cmpxchg.org>
    Cc: Hillf Danton <hillf.zj@alibaba-inc.com>
    Cc: Hugh Dickins <hughd@google.com>
    Cc: Rik van Riel <riel@redhat.com>
    Cc: Mel Gorman <mgorman@techsingularity.net>
    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

diff --git a/mm/rmap.c b/mm/rmap.c
index 3d86036d96ec..1377f7b0361e 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1413,20 +1413,24 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
 			 * Store the swap location in the pte.
 			 * See handle_pte_fault() ...
 			 */
-			VM_BUG_ON_PAGE(!PageSwapCache(page) && PageSwapBacked(page),
-				page);
+			if (VM_WARN_ON_ONCE(PageSwapBacked(page) &&
+						!PageSwapCache(page))) {
+				ret = SWAP_FAIL;
+				page_vma_mapped_walk_done(&pvmw);
+				break;
+			}
 
-			/*
-			 * swapin page could be clean, it has data stored in
-			 * swap. We can't silently discard it without setting
-			 * swap entry in the page table.
-			 */
-			if (!PageDirty(page) && !PageSwapCache(page)) {
-				/* It's a freeable page by MADV_FREE */
-				dec_mm_counter(mm, MM_ANONPAGES);
-				goto discard;
-			} else if (!PageSwapBacked(page)) {
-				/* dirty MADV_FREE page */
+			/* MADV_FREE page check */
+			if (!PageSwapBacked(page)) {
+				if (!PageDirty(page)) {
+					dec_mm_counter(mm, MM_ANONPAGES);
+					goto discard;
+				}
+
+				/*
+				 * If the page was redirtied, it cannot be
+				 * discarded. Remap the page to page table.
+				 */
 				set_pte_at(mm, address, pvmw.pte, pteval);
 				ret = SWAP_DIRTY;
 				page_vma_mapped_walk_done(&pvmw);

Re: + mm-reclaim-madv_free-pages.patch added to -mm tree

[Johannes Weiner]

On Mon, Mar 06, 2017 at 12:03:44PM +0900, Minchan Kim wrote:
> On Fri, Mar 03, 2017 at 10:18:51AM -0500, Johannes Weiner wrote:
> > On Fri, Mar 03, 2017 at 11:52:37AM +0900, Minchan Kim wrote:
> > > On Tue, Feb 28, 2017 at 04:32:38PM -0800, akpm@linux-foundation.org wrote:
> > > > 
> > > > The patch titled
> > > >      Subject: mm: reclaim MADV_FREE pages
> > > > has been added to the -mm tree.  Its filename is
> > > >      mm-reclaim-madv_free-pages.patch
> > > > 
> > > > This patch should soon appear at
> > > >     http://ozlabs.org/~akpm/mmots/broken-out/mm-reclaim-madv_free-pages.patch
> > > > and later at
> > > >     http://ozlabs.org/~akpm/mmotm/broken-out/mm-reclaim-madv_free-pages.patch
> > > > 
> > > > Before you just go and hit "reply", please:
> > > >    a) Consider who else should be cc'ed
> > > >    b) Prefer to cc a suitable mailing list as well
> > > >    c) Ideally: find the original patch on the mailing list and do a
> > > >       reply-to-all to that, adding suitable additional cc's
> > > > 
> > > > *** Remember to use Documentation/SubmitChecklist when testing your code ***
> > > > 
> > > > The -mm tree is included into linux-next and is updated
> > > > there every 3-4 working days
> > > > 
> > > > ------------------------------------------------------
> > > > From: Shaohua Li <shli@fb.com>
> > > > Subject: mm: reclaim MADV_FREE pages
> > > > 
> > > > When memory pressure is high, we free MADV_FREE pages.  If the pages are
> > > > not dirty in pte, the pages could be freed immediately.  Otherwise we
> > > > can't reclaim them.  We put the pages back to anonumous LRU list (by
> > > > setting SwapBacked flag) and the pages will be reclaimed in normal swapout
> > > > way.
> > > > 
> > > > We use normal page reclaim policy.  Since MADV_FREE pages are put into
> > > > inactive file list, such pages and inactive file pages are reclaimed
> > > > according to their age.  This is expected, because we don't want to
> > > > reclaim too many MADV_FREE pages before used once pages.
> > > > 
> > > > Based on Minchan's original patch
> > > > 
> > > > Link: http://lkml.kernel.org/r/14b8eb1d3f6bf6cc492833f183ac8c304e560484.1487965799.git.shli@fb.com
> > > > Signed-off-by: Shaohua Li <shli@fb.com>
> > > > Acked-by: Minchan Kim <minchan@kernel.org>
> > > > Acked-by: Michal Hocko <mhocko@suse.com>
> > > > Acked-by: Johannes Weiner <hannes@cmpxchg.org>
> > > > Acked-by: Hillf Danton <hillf.zj@alibaba-inc.com>
> > > > Cc: Hugh Dickins <hughd@google.com>
> > > > Cc: Rik van Riel <riel@redhat.com>
> > > > Cc: Mel Gorman <mgorman@techsingularity.net>
> > > > Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
> > > > ---
> > > 
> > > < snip >
> > > 
> > > > @@ -1419,11 +1413,21 @@ static int try_to_unmap_one(struct page
> > > >  			VM_BUG_ON_PAGE(!PageSwapCache(page) && PageSwapBacked(page),
> > > >  				page);
> > > >  
> > > > -			if (!PageDirty(page)) {
> > > > +			/*
> > > > +			 * swapin page could be clean, it has data stored in
> > > > +			 * swap. We can't silently discard it without setting
> > > > +			 * swap entry in the page table.
> > > > +			 */
> > > > +			if (!PageDirty(page) && !PageSwapCache(page)) {
> > > >  				/* It's a freeable page by MADV_FREE */
> > > >  				dec_mm_counter(mm, MM_ANONPAGES);
> > > > -				rp->lazyfreed++;
> > > >  				goto discard;
> > > > +			} else if (!PageSwapBacked(page)) {
> > > > +				/* dirty MADV_FREE page */
> > > > +				set_pte_at(mm, address, pvmw.pte, pteval);
> > > > +				ret = SWAP_DIRTY;
> > > > +				page_vma_mapped_walk_done(&pvmw);
> > > > +				break;
> > > >  			}
> > > 
> > > There is no point to make this logic complicated with clean swapin-page.
> > > 
> > > Andrew,
> > > Could you fold below patch into the mm-reclaim-madv_free-pages.patch
> > > if others are not against?
> > > 
> > > Thanks.
> > > 
> > > From 0c28f6560fbc4e65da4f4a8cc4664ab9f7b11cf3 Mon Sep 17 00:00:00 2001
> > > From: Minchan Kim <minchan@kernel.org>
> > > Date: Fri, 3 Mar 2017 11:42:52 +0900
> > > Subject: [PATCH] mm: clean up lazyfree page handling
> > > 
> > > We can make it simple to understand without need to be aware of
> > > clean-swapin page.
> > > This patch just clean up lazyfree page handling in try_to_unmap_one.
> > > 
> > > Signed-off-by: Minchan Kim <minchan@kernel.org>
> > 
> > Agreed, this is a litle easier to follow.
> > 
> > Acked-by: Johannes Weiner <hannes@cmpxchg.org>
> 
> Thanks, Johannes.
> 
> > 
> > > ---
> > >  mm/rmap.c | 22 +++++++++++-----------
> > >  1 file changed, 11 insertions(+), 11 deletions(-)
> > > 
> > > diff --git a/mm/rmap.c b/mm/rmap.c
> > > index bb45712..f7eab40 100644
> > > --- a/mm/rmap.c
> > > +++ b/mm/rmap.c
> > > @@ -1413,17 +1413,17 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
> > >  			VM_BUG_ON_PAGE(!PageSwapCache(page) && PageSwapBacked(page),
> > >  				page);
> > 
> > Since you're removing the PageSwapCache() check and we're now assuming
> > that !swapbacked is not in the swapcache, can you modify this to check
> > PageSwapBacked(page) != PageSwapCache(page)?
> > 
> > Better yet, change it into a warning and SWAP_FAIL.
> 
> Maybe, what you wanted is
> 
>  !!PageSwapBacked(page) != !!PageSwapCache(page)

Those testers return 0 or 1 ints, on x86 even bool.

> Personally, I prefer && style rather than equation expression
> in this case.

> @@ -1413,20 +1413,24 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
>  			 * Store the swap location in the pte.
>  			 * See handle_pte_fault() ...
>  			 */
> -			VM_BUG_ON_PAGE(!PageSwapCache(page) && PageSwapBacked(page),
> -				page);
> +			if (VM_WARN_ON_ONCE(PageSwapBacked(page) &&
> +						!PageSwapCache(page))) {
> +				ret = SWAP_FAIL;

But you're not adding the !swapbacked && swapcache case?

> +				page_vma_mapped_walk_done(&pvmw);
> +				break;
> +			}

[...]

> -			/*
> -			 * swapin page could be clean, it has data stored in
> -			 * swap. We can't silently discard it without setting
> -			 * swap entry in the page table.
> -			 */
> -			if (!PageDirty(page) && !PageSwapCache(page)) {
> -				/* It's a freeable page by MADV_FREE */
> -				dec_mm_counter(mm, MM_ANONPAGES);
> -				goto discard;
> -			} else if (!PageSwapBacked(page)) {
> -				/* dirty MADV_FREE page */
> +			/* MADV_FREE page check */
> +			if (!PageSwapBacked(page)) {
> +				if (!PageDirty(page)) {
> +					dec_mm_counter(mm, MM_ANONPAGES);
> +					goto discard;
> +				}

Andrew already has this, you might want to send the warning changes as
a separate patch on top of this one.

Re: + mm-reclaim-madv_free-pages.patch added to -mm tree

[Minchan Kim]

On Mon, Mar 06, 2017 at 10:49:06AM -0500, Johannes Weiner wrote:
> On Mon, Mar 06, 2017 at 12:03:44PM +0900, Minchan Kim wrote:
> > On Fri, Mar 03, 2017 at 10:18:51AM -0500, Johannes Weiner wrote:
> > > On Fri, Mar 03, 2017 at 11:52:37AM +0900, Minchan Kim wrote:
> > > > On Tue, Feb 28, 2017 at 04:32:38PM -0800, akpm@linux-foundation.org wrote:
> > > > > 
> > > > > The patch titled
> > > > >      Subject: mm: reclaim MADV_FREE pages
> > > > > has been added to the -mm tree.  Its filename is
> > > > >      mm-reclaim-madv_free-pages.patch
> > > > > 
> > > > > This patch should soon appear at
> > > > >     http://ozlabs.org/~akpm/mmots/broken-out/mm-reclaim-madv_free-pages.patch
> > > > > and later at
> > > > >     http://ozlabs.org/~akpm/mmotm/broken-out/mm-reclaim-madv_free-pages.patch
> > > > > 
> > > > > Before you just go and hit "reply", please:
> > > > >    a) Consider who else should be cc'ed
> > > > >    b) Prefer to cc a suitable mailing list as well
> > > > >    c) Ideally: find the original patch on the mailing list and do a
> > > > >       reply-to-all to that, adding suitable additional cc's
> > > > > 
> > > > > *** Remember to use Documentation/SubmitChecklist when testing your code ***
> > > > > 
> > > > > The -mm tree is included into linux-next and is updated
> > > > > there every 3-4 working days
> > > > > 
> > > > > ------------------------------------------------------
> > > > > From: Shaohua Li <shli@fb.com>
> > > > > Subject: mm: reclaim MADV_FREE pages
> > > > > 
> > > > > When memory pressure is high, we free MADV_FREE pages.  If the pages are
> > > > > not dirty in pte, the pages could be freed immediately.  Otherwise we
> > > > > can't reclaim them.  We put the pages back to anonumous LRU list (by
> > > > > setting SwapBacked flag) and the pages will be reclaimed in normal swapout
> > > > > way.
> > > > > 
> > > > > We use normal page reclaim policy.  Since MADV_FREE pages are put into
> > > > > inactive file list, such pages and inactive file pages are reclaimed
> > > > > according to their age.  This is expected, because we don't want to
> > > > > reclaim too many MADV_FREE pages before used once pages.
> > > > > 
> > > > > Based on Minchan's original patch
> > > > > 
> > > > > Link: http://lkml.kernel.org/r/14b8eb1d3f6bf6cc492833f183ac8c304e560484.1487965799.git.shli@fb.com
> > > > > Signed-off-by: Shaohua Li <shli@fb.com>
> > > > > Acked-by: Minchan Kim <minchan@kernel.org>
> > > > > Acked-by: Michal Hocko <mhocko@suse.com>
> > > > > Acked-by: Johannes Weiner <hannes@cmpxchg.org>
> > > > > Acked-by: Hillf Danton <hillf.zj@alibaba-inc.com>
> > > > > Cc: Hugh Dickins <hughd@google.com>
> > > > > Cc: Rik van Riel <riel@redhat.com>
> > > > > Cc: Mel Gorman <mgorman@techsingularity.net>
> > > > > Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
> > > > > ---
> > > > 
> > > > < snip >
> > > > 
> > > > > @@ -1419,11 +1413,21 @@ static int try_to_unmap_one(struct page
> > > > >  			VM_BUG_ON_PAGE(!PageSwapCache(page) && PageSwapBacked(page),
> > > > >  				page);
> > > > >  
> > > > > -			if (!PageDirty(page)) {
> > > > > +			/*
> > > > > +			 * swapin page could be clean, it has data stored in
> > > > > +			 * swap. We can't silently discard it without setting
> > > > > +			 * swap entry in the page table.
> > > > > +			 */
> > > > > +			if (!PageDirty(page) && !PageSwapCache(page)) {
> > > > >  				/* It's a freeable page by MADV_FREE */
> > > > >  				dec_mm_counter(mm, MM_ANONPAGES);
> > > > > -				rp->lazyfreed++;
> > > > >  				goto discard;
> > > > > +			} else if (!PageSwapBacked(page)) {
> > > > > +				/* dirty MADV_FREE page */
> > > > > +				set_pte_at(mm, address, pvmw.pte, pteval);
> > > > > +				ret = SWAP_DIRTY;
> > > > > +				page_vma_mapped_walk_done(&pvmw);
> > > > > +				break;
> > > > >  			}
> > > > 
> > > > There is no point to make this logic complicated with clean swapin-page.
> > > > 
> > > > Andrew,
> > > > Could you fold below patch into the mm-reclaim-madv_free-pages.patch
> > > > if others are not against?
> > > > 
> > > > Thanks.
> > > > 
> > > > From 0c28f6560fbc4e65da4f4a8cc4664ab9f7b11cf3 Mon Sep 17 00:00:00 2001
> > > > From: Minchan Kim <minchan@kernel.org>
> > > > Date: Fri, 3 Mar 2017 11:42:52 +0900
> > > > Subject: [PATCH] mm: clean up lazyfree page handling
> > > > 
> > > > We can make it simple to understand without need to be aware of
> > > > clean-swapin page.
> > > > This patch just clean up lazyfree page handling in try_to_unmap_one.
> > > > 
> > > > Signed-off-by: Minchan Kim <minchan@kernel.org>
> > > 
> > > Agreed, this is a litle easier to follow.
> > > 
> > > Acked-by: Johannes Weiner <hannes@cmpxchg.org>
> > 
> > Thanks, Johannes.
> > 
> > > 
> > > > ---
> > > >  mm/rmap.c | 22 +++++++++++-----------
> > > >  1 file changed, 11 insertions(+), 11 deletions(-)
> > > > 
> > > > diff --git a/mm/rmap.c b/mm/rmap.c
> > > > index bb45712..f7eab40 100644
> > > > --- a/mm/rmap.c
> > > > +++ b/mm/rmap.c
> > > > @@ -1413,17 +1413,17 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
> > > >  			VM_BUG_ON_PAGE(!PageSwapCache(page) && PageSwapBacked(page),
> > > >  				page);
> > > 
> > > Since you're removing the PageSwapCache() check and we're now assuming
> > > that !swapbacked is not in the swapcache, can you modify this to check
> > > PageSwapBacked(page) != PageSwapCache(page)?
> > > 
> > > Better yet, change it into a warning and SWAP_FAIL.
> > 
> > Maybe, what you wanted is
> > 
> >  !!PageSwapBacked(page) != !!PageSwapCache(page)
> 
> Those testers return 0 or 1 ints, on x86 even bool.

You're right.
It seems PageXXX() was not in old day. Hmm, but seem to have corrected.

> 
> > Personally, I prefer && style rather than equation expression
> > in this case.
> 
> > @@ -1413,20 +1413,24 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
> >  			 * Store the swap location in the pte.
> >  			 * See handle_pte_fault() ...
> >  			 */
> > -			VM_BUG_ON_PAGE(!PageSwapCache(page) && PageSwapBacked(page),
> > -				page);
> > +			if (VM_WARN_ON_ONCE(PageSwapBacked(page) &&
> > +						!PageSwapCache(page))) {
> > +				ret = SWAP_FAIL;
> 
> But you're not adding the !swapbacked && swapcache case?

PageSwapcache checks already includes PageSwapBacked check so practically,
it would be pointless at this moment. But yes, it would be bad on relying
on the current implemenation as considering the future.

As a bonus from side-effect, seeing the warning always only means
it is from PageSwapbacked && !swapcache so we don't need to introduce
VM_WARN_ON_ONCE_PAGE at this moment. :)

I will resend it.

Thanks for the review!

Re: + mm-reclaim-madv_free-pages.patch added to -mm tree

[Minchan Kim]

On Mon, Mar 06, 2017 at 10:49:06AM -0500, Johannes Weiner wrote:

< snip >

> > @@ -1413,20 +1413,24 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
> >  			 * Store the swap location in the pte.
> >  			 * See handle_pte_fault() ...
> >  			 */
> > -			VM_BUG_ON_PAGE(!PageSwapCache(page) && PageSwapBacked(page),
> > -				page);
> > +			if (VM_WARN_ON_ONCE(PageSwapBacked(page) &&
> > +						!PageSwapCache(page))) {
> > +				ret = SWAP_FAIL;
> 
> But you're not adding the !swapbacked && swapcache case?
> 
> > +				page_vma_mapped_walk_done(&pvmw);
> > +				break;
> > +			}
> 
> [...]
> 
> > -			/*
> > -			 * swapin page could be clean, it has data stored in
> > -			 * swap. We can't silently discard it without setting
> > -			 * swap entry in the page table.
> > -			 */
> > -			if (!PageDirty(page) && !PageSwapCache(page)) {
> > -				/* It's a freeable page by MADV_FREE */
> > -				dec_mm_counter(mm, MM_ANONPAGES);
> > -				goto discard;
> > -			} else if (!PageSwapBacked(page)) {
> > -				/* dirty MADV_FREE page */
> > +			/* MADV_FREE page check */
> > +			if (!PageSwapBacked(page)) {
> > +				if (!PageDirty(page)) {
> > +					dec_mm_counter(mm, MM_ANONPAGES);
> > +					goto discard;
> > +				}
> 
> Andrew already has this, you might want to send the warning changes as
> a separate patch on top of this one.

Here it goes.

>From d42d296950c3bbce74afddcff307fa18eef305fe Mon Sep 17 00:00:00 2001
From: Minchan Kim <minchan@kernel.org>
Date: Tue, 7 Mar 2017 14:48:37 +0900
Subject: [PATCH] mm: fix lazyfree bug on check in try_to_unmap_one

If a page is swapbacked, it means it should be in swapcache
in try_to_unmap_one's path.

If a page is !swapbacked, it mean it shouldn't be in swapcache
in try_to_unmap_one's path.

Check both two cases all at once and if it fails, warn and
return SWAP_FAIL. Such bug never mean we should shut down
the kernel.

Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Minchan Kim <minchan@kernel.org>
---
 mm/rmap.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/mm/rmap.c b/mm/rmap.c
index 35acb83..9925f32 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1413,8 +1413,13 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
 			 * Store the swap location in the pte.
 			 * See handle_pte_fault() ...
 			 */
-			VM_BUG_ON_PAGE(!PageSwapCache(page) && PageSwapBacked(page),
-				page);
+			if (VM_WARN_ON_ONCE(PageSwapBacked(page) !=
+						PageSwapCache(page))) {
+				ret = SWAP_FAIL;
+				page_vma_mapped_walk_done(&pvmw);
+				break;
+
+			}
 
 			/* MADV_FREE page check */
 			if (!PageSwapBacked(page)) {
-- 
2.7.4

Re: + mm-reclaim-madv_free-pages.patch added to -mm tree

[Michal Hocko]

On Tue 07-03-17 14:55:51, Minchan Kim wrote:
[...]
> >From d42d296950c3bbce74afddcff307fa18eef305fe Mon Sep 17 00:00:00 2001
> From: Minchan Kim <minchan@kernel.org>
> Date: Tue, 7 Mar 2017 14:48:37 +0900
> Subject: [PATCH] mm: fix lazyfree bug on check in try_to_unmap_one
> 
> If a page is swapbacked, it means it should be in swapcache
> in try_to_unmap_one's path.
> 
> If a page is !swapbacked, it mean it shouldn't be in swapcache
> in try_to_unmap_one's path.
> 
> Check both two cases all at once and if it fails, warn and
> return SWAP_FAIL. Such bug never mean we should shut down
> the kernel.
> 
> Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
> Signed-off-by: Minchan Kim <minchan@kernel.org>

looks good to me
Acked-by: Michal Hocko <mhocko@suse.com>

> ---
>  mm/rmap.c | 9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/rmap.c b/mm/rmap.c
> index 35acb83..9925f32 100644
> --- a/mm/rmap.c
> +++ b/mm/rmap.c
> @@ -1413,8 +1413,13 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
>  			 * Store the swap location in the pte.
>  			 * See handle_pte_fault() ...
>  			 */
> -			VM_BUG_ON_PAGE(!PageSwapCache(page) && PageSwapBacked(page),
> -				page);
> +			if (VM_WARN_ON_ONCE(PageSwapBacked(page) !=
> +						PageSwapCache(page))) {
> +				ret = SWAP_FAIL;
> +				page_vma_mapped_walk_done(&pvmw);
> +				break;
> +
> +			}
>  
>  			/* MADV_FREE page check */
>  			if (!PageSwapBacked(page)) {
> -- 
> 2.7.4
> 

-- 
Michal Hocko
SUSE Labs

Re: + mm-reclaim-madv_free-pages.patch added to -mm tree

[Johannes Weiner]

On Tue, Mar 07, 2017 at 02:55:51PM +0900, Minchan Kim wrote:
> From d42d296950c3bbce74afddcff307fa18eef305fe Mon Sep 17 00:00:00 2001
> From: Minchan Kim <minchan@kernel.org>
> Date: Tue, 7 Mar 2017 14:48:37 +0900
> Subject: [PATCH] mm: fix lazyfree bug on check in try_to_unmap_one
> 
> If a page is swapbacked, it means it should be in swapcache
> in try_to_unmap_one's path.
> 
> If a page is !swapbacked, it mean it shouldn't be in swapcache
> in try_to_unmap_one's path.
> 
> Check both two cases all at once and if it fails, warn and
> return SWAP_FAIL. Such bug never mean we should shut down
> the kernel.
> 
> Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
> Signed-off-by: Minchan Kim <minchan@kernel.org>

Acked-by: Johannes Weiner <hannes@cmpxchg.org>

Thanks Minchan

Re: + mm-reclaim-madv_free-pages.patch added to -mm tree

[Shaohua Li]

On Fri, Mar 03, 2017 at 11:52:37AM +0900, Minchan Kim wrote:
> Hi,
> 
> On Tue, Feb 28, 2017 at 04:32:38PM -0800, akpm@linux-foundation.org wrote:
> > 
> > The patch titled
> >      Subject: mm: reclaim MADV_FREE pages
> > has been added to the -mm tree.  Its filename is
> >      mm-reclaim-madv_free-pages.patch
> > 
> > This patch should soon appear at
> >     https://urldefense.proofpoint.com/v2/url?u=http-3A__ozlabs.org_-7Eakpm_mmots_broken-2Dout_mm-2Dreclaim-2Dmadv-5Ffree-2Dpages.patch&d=DwIBAg&c=5VD0RTtNlTh3ycd41b3MUw&r=X13hAPkxmvBro1Ug8vcKHw&m=oMKTke-r00qQPnpEwo0Cn43po669gckvribhR9LQWf0&s=wzX_7IDavzBkFEMl7HDTMwrFXo1skB35mu0CjZmfFOg&e= 
> > and later at
> >     https://urldefense.proofpoint.com/v2/url?u=http-3A__ozlabs.org_-7Eakpm_mmotm_broken-2Dout_mm-2Dreclaim-2Dmadv-5Ffree-2Dpages.patch&d=DwIBAg&c=5VD0RTtNlTh3ycd41b3MUw&r=X13hAPkxmvBro1Ug8vcKHw&m=oMKTke-r00qQPnpEwo0Cn43po669gckvribhR9LQWf0&s=eUYtR1qeasKBGvuNWsZP6jE4XwoKwMb4CdKQU7OXGYM&e= 
> > 
> > Before you just go and hit "reply", please:
> >    a) Consider who else should be cc'ed
> >    b) Prefer to cc a suitable mailing list as well
> >    c) Ideally: find the original patch on the mailing list and do a
> >       reply-to-all to that, adding suitable additional cc's
> > 
> > *** Remember to use Documentation/SubmitChecklist when testing your code ***
> > 
> > The -mm tree is included into linux-next and is updated
> > there every 3-4 working days
> > 
> > ------------------------------------------------------
> > From: Shaohua Li <shli@fb.com>
> > Subject: mm: reclaim MADV_FREE pages
> > 
> > When memory pressure is high, we free MADV_FREE pages.  If the pages are
> > not dirty in pte, the pages could be freed immediately.  Otherwise we
> > can't reclaim them.  We put the pages back to anonumous LRU list (by
> > setting SwapBacked flag) and the pages will be reclaimed in normal swapout
> > way.
> > 
> > We use normal page reclaim policy.  Since MADV_FREE pages are put into
> > inactive file list, such pages and inactive file pages are reclaimed
> > according to their age.  This is expected, because we don't want to
> > reclaim too many MADV_FREE pages before used once pages.
> > 
> > Based on Minchan's original patch
> > 
> > Link: http://lkml.kernel.org/r/14b8eb1d3f6bf6cc492833f183ac8c304e560484.1487965799.git.shli@fb.com
> > Signed-off-by: Shaohua Li <shli@fb.com>
> > Acked-by: Minchan Kim <minchan@kernel.org>
> > Acked-by: Michal Hocko <mhocko@suse.com>
> > Acked-by: Johannes Weiner <hannes@cmpxchg.org>
> > Acked-by: Hillf Danton <hillf.zj@alibaba-inc.com>
> > Cc: Hugh Dickins <hughd@google.com>
> > Cc: Rik van Riel <riel@redhat.com>
> > Cc: Mel Gorman <mgorman@techsingularity.net>
> > Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
> > ---
> 
> < snip >
> 
> > @@ -1419,11 +1413,21 @@ static int try_to_unmap_one(struct page
> >  			VM_BUG_ON_PAGE(!PageSwapCache(page) && PageSwapBacked(page),
> >  				page);
> >  
> > -			if (!PageDirty(page)) {
> > +			/*
> > +			 * swapin page could be clean, it has data stored in
> > +			 * swap. We can't silently discard it without setting
> > +			 * swap entry in the page table.
> > +			 */
> > +			if (!PageDirty(page) && !PageSwapCache(page)) {
> >  				/* It's a freeable page by MADV_FREE */
> >  				dec_mm_counter(mm, MM_ANONPAGES);
> > -				rp->lazyfreed++;
> >  				goto discard;
> > +			} else if (!PageSwapBacked(page)) {
> > +				/* dirty MADV_FREE page */
> > +				set_pte_at(mm, address, pvmw.pte, pteval);
> > +				ret = SWAP_DIRTY;
> > +				page_vma_mapped_walk_done(&pvmw);
> > +				break;
> >  			}
> 
> There is no point to make this logic complicated with clean swapin-page.
> 
> Andrew,
> Could you fold below patch into the mm-reclaim-madv_free-pages.patch
> if others are not against?
> 
> Thanks.
> 
> From 0c28f6560fbc4e65da4f4a8cc4664ab9f7b11cf3 Mon Sep 17 00:00:00 2001
> From: Minchan Kim <minchan@kernel.org>
> Date: Fri, 3 Mar 2017 11:42:52 +0900
> Subject: [PATCH] mm: clean up lazyfree page handling
> 
> We can make it simple to understand without need to be aware of
> clean-swapin page.
> This patch just clean up lazyfree page handling in try_to_unmap_one.

Looks good, thanks!

Reviewed-by: Shaohua Li <shli@fb.com>
 
> Signed-off-by: Minchan Kim <minchan@kernel.org>
> ---
>  mm/rmap.c | 22 +++++++++++-----------
>  1 file changed, 11 insertions(+), 11 deletions(-)
> 
> diff --git a/mm/rmap.c b/mm/rmap.c
> index bb45712..f7eab40 100644
> --- a/mm/rmap.c
> +++ b/mm/rmap.c
> @@ -1413,17 +1413,17 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
>  			VM_BUG_ON_PAGE(!PageSwapCache(page) && PageSwapBacked(page),
>  				page);
>  
> -			/*
> -			 * swapin page could be clean, it has data stored in
> -			 * swap. We can't silently discard it without setting
> -			 * swap entry in the page table.
> -			 */
> -			if (!PageDirty(page) && !PageSwapCache(page)) {
> -				/* It's a freeable page by MADV_FREE */
> -				dec_mm_counter(mm, MM_ANONPAGES);
> -				goto discard;
> -			} else if (!PageSwapBacked(page)) {
> -				/* dirty MADV_FREE page */
> +			/* MADV_FREE page check */
> +			if (!PageSwapBacked(page)) {
> +				if (!PageDirty(page)) {
> +					dec_mm_counter(mm, MM_ANONPAGES);
> +					goto discard;
> +				}
> +
> +				/*
> +				 * If the page was redirtied, it cannot be
> +				 * discarded. Remap the page to page table.
> +				 */
>  				set_pte_at(mm, address, pvmw.pte, pteval);
>  				ret = SWAP_DIRTY;
>  				page_vma_mapped_walk_done(&pvmw);
> -- 
> 2.7.4
>