[PATCH] thp: keep huge zero pinned until tlb flush

[PATCH] thp: keep huge zero pinned until tlb flush

[Kirill A. Shutemov]

Andrea has found[1] a race condition on MMU-gather based TLB flush vs
split_huge_page() or shrinker which frees huge zero under us (patch 1/2
and 2/2 respectively).

With new THP refcounting, we don't patch 1/2: mmu_gather keeps the page
page pinned until flush is complete and the pin prevent the page from
being split under us.

We sill need patch 2/2. This is simplified version of Andrea's patch.
We don't need fancy encoding.

[1] http://lkml.kernel.org/r/1447938052-22165-1-git-send-email-aarcange@redhat.com

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Reported-by: Andrea Arcangeli <aarcange@redhat.com>
---
 include/linux/huge_mm.h | 1 +
 mm/huge_memory.c        | 6 +++---
 mm/swap.c               | 5 +++++
 3 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index 7008623e24b1..8232e0b8a04f 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -152,6 +152,7 @@ static inline bool is_huge_zero_pmd(pmd_t pmd)
 }
 
 struct page *get_huge_zero_page(void);
+void put_huge_zero_page(void);
 
 #else /* CONFIG_TRANSPARENT_HUGEPAGE */
 #define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; })
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 820214137bc5..860c7dec197e 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -235,7 +235,7 @@ retry:
 	return READ_ONCE(huge_zero_page);
 }
 
-static void put_huge_zero_page(void)
+void put_huge_zero_page(void)
 {
 	/*
 	 * Counter should never go to zero here. Only shrinker can put
@@ -1715,12 +1715,12 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 	if (vma_is_dax(vma)) {
 		spin_unlock(ptl);
 		if (is_huge_zero_pmd(orig_pmd))
-			put_huge_zero_page();
+			tlb_remove_page(tlb, pmd_page(orig_pmd));
 	} else if (is_huge_zero_pmd(orig_pmd)) {
 		pte_free(tlb->mm, pgtable_trans_huge_withdraw(tlb->mm, pmd));
 		atomic_long_dec(&tlb->mm->nr_ptes);
 		spin_unlock(ptl);
-		put_huge_zero_page();
+		tlb_remove_page(tlb, pmd_page(orig_pmd));
 	} else {
 		struct page *page = pmd_page(orig_pmd);
 		page_remove_rmap(page, true);
diff --git a/mm/swap.c b/mm/swap.c
index 09fe5e97714a..11915bd0f047 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -728,6 +728,11 @@ void release_pages(struct page **pages, int nr, bool cold)
 			zone = NULL;
 		}
 
+		if (is_huge_zero_page(page)) {
+			put_huge_zero_page();
+			continue;
+		}
+
 		page = compound_head(page);
 		if (!put_page_testzero(page))
 			continue;
-- 
2.8.0.rc3

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] thp: keep huge zero pinned until tlb flush

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 1212 bytes --]

Hi Kirill,

[auto build test ERROR on v4.6-rc2]
[also build test ERROR on next-20160404]
[if your patch is applied to the wrong git tree, please drop us a note to help improving the system]

url:    https://github.com/0day-ci/linux/commits/Kirill-A-Shutemov/thp-keep-huge-zero-pinned-until-tlb-flush/20160405-075953
config: i386-tinyconfig (attached as .config)
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

All errors (new ones prefixed by >>):

   mm/swap.c: In function 'release_pages':
>> mm/swap.c:732:4: error: implicit declaration of function 'put_huge_zero_page' [-Werror=implicit-function-declaration]
       put_huge_zero_page();
       ^
   cc1: some warnings being treated as errors

vim +/put_huge_zero_page +732 mm/swap.c

   726			if (zone && ++lock_batch == SWAP_CLUSTER_MAX) {
   727				spin_unlock_irqrestore(&zone->lru_lock, flags);
   728				zone = NULL;
   729			}
   730	
   731			if (is_huge_zero_page(page)) {
 > 732				put_huge_zero_page();
   733				continue;
   734			}
   735	

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/octet-stream, Size: 6274 bytes --]

Re: [PATCH] thp: keep huge zero pinned until tlb flush

[Aneesh Kumar K.V]

"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> writes:

> [ text/plain ]
> Andrea has found[1] a race condition on MMU-gather based TLB flush vs
> split_huge_page() or shrinker which frees huge zero under us (patch 1/2
> and 2/2 respectively).
>
> With new THP refcounting, we don't patch 1/2: mmu_gather keeps the page
> page pinned until flush is complete and the pin prevent the page from
> being split under us.
>
> We sill need patch 2/2. This is simplified version of Andrea's patch.
> We don't need fancy encoding.
>
> [1] http://lkml.kernel.org/r/1447938052-22165-1-git-send-email-aarcange@redhat.com

May be we should add the below from the original mail

put_huge_zero_page() because it can actually lead to the almost
immediate freeing of the THP zero page, must be run in the MMU gather
and not before the TLB flush like it is happening right now.


Reviewed-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>


>
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> Reported-by: Andrea Arcangeli <aarcange@redhat.com>
> ---
>  include/linux/huge_mm.h | 1 +
>  mm/huge_memory.c        | 6 +++---
>  mm/swap.c               | 5 +++++
>  3 files changed, 9 insertions(+), 3 deletions(-)
>
> diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
> index 7008623e24b1..8232e0b8a04f 100644
> --- a/include/linux/huge_mm.h
> +++ b/include/linux/huge_mm.h
> @@ -152,6 +152,7 @@ static inline bool is_huge_zero_pmd(pmd_t pmd)
>  }
>
>  struct page *get_huge_zero_page(void);
> +void put_huge_zero_page(void);
>
>  #else /* CONFIG_TRANSPARENT_HUGEPAGE */
>  #define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; })
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index 820214137bc5..860c7dec197e 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -235,7 +235,7 @@ retry:
>  	return READ_ONCE(huge_zero_page);
>  }
>
> -static void put_huge_zero_page(void)
> +void put_huge_zero_page(void)
>  {
>  	/*
>  	 * Counter should never go to zero here. Only shrinker can put
> @@ -1715,12 +1715,12 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
>  	if (vma_is_dax(vma)) {
>  		spin_unlock(ptl);
>  		if (is_huge_zero_pmd(orig_pmd))
> -			put_huge_zero_page();
> +			tlb_remove_page(tlb, pmd_page(orig_pmd));
>  	} else if (is_huge_zero_pmd(orig_pmd)) {
>  		pte_free(tlb->mm, pgtable_trans_huge_withdraw(tlb->mm, pmd));
>  		atomic_long_dec(&tlb->mm->nr_ptes);
>  		spin_unlock(ptl);
> -		put_huge_zero_page();
> +		tlb_remove_page(tlb, pmd_page(orig_pmd));
>  	} else {
>  		struct page *page = pmd_page(orig_pmd);
>  		page_remove_rmap(page, true);
> diff --git a/mm/swap.c b/mm/swap.c
> index 09fe5e97714a..11915bd0f047 100644
> --- a/mm/swap.c
> +++ b/mm/swap.c
> @@ -728,6 +728,11 @@ void release_pages(struct page **pages, int nr, bool cold)
>  			zone = NULL;
>  		}
>
> +		if (is_huge_zero_page(page)) {
> +			put_huge_zero_page();
> +			continue;
> +		}
> +
>  		page = compound_head(page);
>  		if (!put_page_testzero(page))
>  			continue;
> -- 
> 2.8.0.rc3

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>