[PATCH v2] mm: remove VM_BUG_ON(PageSlab()) from page_mapcount()

[PATCH v2] mm: remove VM_BUG_ON(PageSlab()) from page_mapcount()

[Konstantin Khlebnikov]

Replace superfluous VM_BUG_ON() with comment about correct usage.

Technically reverts commit 1d148e218a0d0566b1c06f2f45f1436d53b049b2
("mm: add VM_BUG_ON_PAGE() to page_mapcount()"), but context have changed.

Function isolate_migratepages_block() runs some checks out of lru_lock
when choose pages for migration. After checking PageLRU() it checks extra
page references by comparing page_count() and page_mapcount(). Between
these two checks page could be removed from lru, freed and taken by slab.

As a result this race triggers VM_BUG_ON(PageSlab()) in page_mapcount().
Race window is tiny. For certain workload this happens around once a year.


 page:ffffea0105ca9380 count:1 mapcount:0 mapping:ffff88ff7712c180 index:0x0 compound_mapcount: 0
 flags: 0x500000000008100(slab|head)
 raw: 0500000000008100 dead000000000100 dead000000000200 ffff88ff7712c180
 raw: 0000000000000000 0000000080200020 00000001ffffffff 0000000000000000
 page dumped because: VM_BUG_ON_PAGE(PageSlab(page))
 ------------[ cut here ]------------
 kernel BUG at ./include/linux/mm.h:628!
 invalid opcode: 0000 [#1] SMP NOPTI
 CPU: 77 PID: 504 Comm: kcompactd1 Tainted: G        W         4.19.109-27 #1
 Hardware name: Yandex T175-N41-Y3N/MY81-EX0-Y3N, BIOS R05 06/20/2019
 RIP: 0010:isolate_migratepages_block+0x986/0x9b0


Code in isolate_migratepages_block() was added in commit 119d6d59dcc0
("mm, compaction: avoid isolating pinned pages") before adding VM_BUG_ON
into page_mapcount().

This race has been predicted in 2015 by Vlastimil Babka (see link below).

Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
Fixes: 1d148e218a0d ("mm: add VM_BUG_ON_PAGE() to page_mapcount()")
Link: https://lore.kernel.org/lkml/557710E1.6060103@suse.cz/
Link: https://lore.kernel.org/linux-mm/158937872515.474360.5066096871639561424.stgit@buzz/T/ (v1)
---
 include/linux/mm.h |   14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 5a323422d783..95f777f482ac 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -782,6 +782,11 @@ static inline void *kvcalloc(size_t n, size_t size, gfp_t flags)
 
 extern void kvfree(const void *addr);
 
+/*
+ * Mapcount of compound page as a whole, not includes mapped sub-pages.
+ *
+ * Must be called only for compound pages or any their tail sub-pages.
+ */
 static inline int compound_mapcount(struct page *page)
 {
 	VM_BUG_ON_PAGE(!PageCompound(page), page);
@@ -801,10 +806,15 @@ static inline void page_mapcount_reset(struct page *page)
 
 int __page_mapcount(struct page *page);
 
+/*
+ * Mapcount of 0-order page, for sub-page includes compound_mapcount().
+ *
+ * Result is undefined for pages which cannot be mapped into userspace.
+ * For example SLAB or special types of pages. See function page_has_type().
+ * They use this place in struct page differently.
+ */
 static inline int page_mapcount(struct page *page)
 {
-	VM_BUG_ON_PAGE(PageSlab(page), page);
-
 	if (unlikely(PageCompound(page)))
 		return __page_mapcount(page);
 	return atomic_read(&page->_mapcount) + 1;

Re: [PATCH v2] mm: remove VM_BUG_ON(PageSlab()) from page_mapcount()

[Hugh Dickins]

On Sun, 24 May 2020, Konstantin Khlebnikov wrote:

> Replace superfluous VM_BUG_ON() with comment about correct usage.
> 
> Technically reverts commit 1d148e218a0d0566b1c06f2f45f1436d53b049b2
> ("mm: add VM_BUG_ON_PAGE() to page_mapcount()"), but context have changed.
> 
> Function isolate_migratepages_block() runs some checks out of lru_lock
> when choose pages for migration. After checking PageLRU() it checks extra
> page references by comparing page_count() and page_mapcount(). Between
> these two checks page could be removed from lru, freed and taken by slab.
> 
> As a result this race triggers VM_BUG_ON(PageSlab()) in page_mapcount().
> Race window is tiny. For certain workload this happens around once a year.
> 
> 
>  page:ffffea0105ca9380 count:1 mapcount:0 mapping:ffff88ff7712c180 index:0x0 compound_mapcount: 0
>  flags: 0x500000000008100(slab|head)
>  raw: 0500000000008100 dead000000000100 dead000000000200 ffff88ff7712c180
>  raw: 0000000000000000 0000000080200020 00000001ffffffff 0000000000000000
>  page dumped because: VM_BUG_ON_PAGE(PageSlab(page))
>  ------------[ cut here ]------------
>  kernel BUG at ./include/linux/mm.h:628!
>  invalid opcode: 0000 [#1] SMP NOPTI
>  CPU: 77 PID: 504 Comm: kcompactd1 Tainted: G        W         4.19.109-27 #1
>  Hardware name: Yandex T175-N41-Y3N/MY81-EX0-Y3N, BIOS R05 06/20/2019
>  RIP: 0010:isolate_migratepages_block+0x986/0x9b0
> 
> 
> Code in isolate_migratepages_block() was added in commit 119d6d59dcc0
> ("mm, compaction: avoid isolating pinned pages") before adding VM_BUG_ON
> into page_mapcount().
> 
> This race has been predicted in 2015 by Vlastimil Babka (see link below).

Ah, kudos to Vlastimil.

> 
> Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>

Thanks,
Acked-by: Hugh Dickins <hughd@google.com>

For the BUG deletion.  I got a bit confused by the comments at first:
slight rewording suggested below.

I had also intended a comment in page_mapcount(), on being used in
that racy exceptional way by isolate_migratepages_block(): but now
that the BUG is gone, I think you're right not to add that - it's
not for the function to document its various callers, nor what's
not in it.  That kind of info belongs to the commit message,
as you have done.

> Fixes: 1d148e218a0d ("mm: add VM_BUG_ON_PAGE() to page_mapcount()")
> Link: https://lore.kernel.org/lkml/557710E1.6060103@suse.cz/
> Link: https://lore.kernel.org/linux-mm/158937872515.474360.5066096871639561424.stgit@buzz/T/ (v1)
> ---
>  include/linux/mm.h |   14 ++++++++++++--
>  1 file changed, 12 insertions(+), 2 deletions(-)
> 
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 5a323422d783..95f777f482ac 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -782,6 +782,11 @@ static inline void *kvcalloc(size_t n, size_t size, gfp_t flags)
>  
>  extern void kvfree(const void *addr);
>  
> +/*
> + * Mapcount of compound page as a whole, not includes mapped sub-pages.

s/not includes/does not include/

> + *
> + * Must be called only for compound pages or any their tail sub-pages.
> + */
>  static inline int compound_mapcount(struct page *page)
>  {
>  	VM_BUG_ON_PAGE(!PageCompound(page), page);
> @@ -801,10 +806,15 @@ static inline void page_mapcount_reset(struct page *page)
>  
>  int __page_mapcount(struct page *page);
>  
> +/*
> + * Mapcount of 0-order page, for sub-page includes compound_mapcount().

s/, for sub-page/; when compound sub-page,/

> + *
> + * Result is undefined for pages which cannot be mapped into userspace.
> + * For example SLAB or special types of pages. See function page_has_type().
> + * They use this place in struct page differently.
> + */
>  static inline int page_mapcount(struct page *page)
>  {
> -	VM_BUG_ON_PAGE(PageSlab(page), page);
> -
>  	if (unlikely(PageCompound(page)))
>  		return __page_mapcount(page);
>  	return atomic_read(&page->_mapcount) + 1;

Re: [PATCH v2] mm: remove VM_BUG_ON(PageSlab()) from page_mapcount()

[Hugh Dickins]

On Sun, 24 May 2020, Konstantin Khlebnikov wrote:

> Replace superfluous VM_BUG_ON() with comment about correct usage.
> 
> Technically reverts commit 1d148e218a0d0566b1c06f2f45f1436d53b049b2
> ("mm: add VM_BUG_ON_PAGE() to page_mapcount()"), but context have changed.
> 
> Function isolate_migratepages_block() runs some checks out of lru_lock
> when choose pages for migration. After checking PageLRU() it checks extra
> page references by comparing page_count() and page_mapcount(). Between
> these two checks page could be removed from lru, freed and taken by slab.
> 
> As a result this race triggers VM_BUG_ON(PageSlab()) in page_mapcount().
> Race window is tiny. For certain workload this happens around once a year.
> 
> 
>  page:ffffea0105ca9380 count:1 mapcount:0 mapping:ffff88ff7712c180 index:0x0 compound_mapcount: 0
>  flags: 0x500000000008100(slab|head)
>  raw: 0500000000008100 dead000000000100 dead000000000200 ffff88ff7712c180
>  raw: 0000000000000000 0000000080200020 00000001ffffffff 0000000000000000
>  page dumped because: VM_BUG_ON_PAGE(PageSlab(page))
>  ------------[ cut here ]------------
>  kernel BUG at ./include/linux/mm.h:628!
>  invalid opcode: 0000 [#1] SMP NOPTI
>  CPU: 77 PID: 504 Comm: kcompactd1 Tainted: G        W         4.19.109-27 #1
>  Hardware name: Yandex T175-N41-Y3N/MY81-EX0-Y3N, BIOS R05 06/20/2019
>  RIP: 0010:isolate_migratepages_block+0x986/0x9b0
> 
> 
> Code in isolate_migratepages_block() was added in commit 119d6d59dcc0
> ("mm, compaction: avoid isolating pinned pages") before adding VM_BUG_ON
> into page_mapcount().
> 
> This race has been predicted in 2015 by Vlastimil Babka (see link below).

Ah, kudos to Vlastimil.

> 
> Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>

Thanks,
Acked-by: Hugh Dickins <hughd@google.com>

For the BUG deletion.  I got a bit confused by the comments at first:
slight rewording suggested below.

I had also intended a comment in page_mapcount(), on being used in
that racy exceptional way by isolate_migratepages_block(): but now
that the BUG is gone, I think you're right not to add that - it's
not for the function to document its various callers, nor what's
not in it.  That kind of info belongs to the commit message,
as you have done.

> Fixes: 1d148e218a0d ("mm: add VM_BUG_ON_PAGE() to page_mapcount()")
> Link: https://lore.kernel.org/lkml/557710E1.6060103@suse.cz/
> Link: https://lore.kernel.org/linux-mm/158937872515.474360.5066096871639561424.stgit@buzz/T/ (v1)
> ---
>  include/linux/mm.h |   14 ++++++++++++--
>  1 file changed, 12 insertions(+), 2 deletions(-)
> 
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 5a323422d783..95f777f482ac 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -782,6 +782,11 @@ static inline void *kvcalloc(size_t n, size_t size, gfp_t flags)
>  
>  extern void kvfree(const void *addr);
>  
> +/*
> + * Mapcount of compound page as a whole, not includes mapped sub-pages.

s/not includes/does not include/

> + *
> + * Must be called only for compound pages or any their tail sub-pages.
> + */
>  static inline int compound_mapcount(struct page *page)
>  {
>  	VM_BUG_ON_PAGE(!PageCompound(page), page);
> @@ -801,10 +806,15 @@ static inline void page_mapcount_reset(struct page *page)
>  
>  int __page_mapcount(struct page *page);
>  
> +/*
> + * Mapcount of 0-order page, for sub-page includes compound_mapcount().

s/, for sub-page/; when compound sub-page,/

> + *
> + * Result is undefined for pages which cannot be mapped into userspace.
> + * For example SLAB or special types of pages. See function page_has_type().
> + * They use this place in struct page differently.
> + */
>  static inline int page_mapcount(struct page *page)
>  {
> -	VM_BUG_ON_PAGE(PageSlab(page), page);
> -
>  	if (unlikely(PageCompound(page)))
>  		return __page_mapcount(page);
>  	return atomic_read(&page->_mapcount) + 1;

Re: [PATCH v2] mm: remove VM_BUG_ON(PageSlab()) from page_mapcount()

[Kirill A. Shutemov]

On Sun, May 24, 2020 at 04:43:18PM +0300, Konstantin Khlebnikov wrote:
> Replace superfluous VM_BUG_ON() with comment about correct usage.
> 
> Technically reverts commit 1d148e218a0d0566b1c06f2f45f1436d53b049b2
> ("mm: add VM_BUG_ON_PAGE() to page_mapcount()"), but context have changed.
> 
> Function isolate_migratepages_block() runs some checks out of lru_lock
> when choose pages for migration. After checking PageLRU() it checks extra
> page references by comparing page_count() and page_mapcount(). Between
> these two checks page could be removed from lru, freed and taken by slab.
> 
> As a result this race triggers VM_BUG_ON(PageSlab()) in page_mapcount().
> Race window is tiny. For certain workload this happens around once a year.
> 
> 
>  page:ffffea0105ca9380 count:1 mapcount:0 mapping:ffff88ff7712c180 index:0x0 compound_mapcount: 0
>  flags: 0x500000000008100(slab|head)
>  raw: 0500000000008100 dead000000000100 dead000000000200 ffff88ff7712c180
>  raw: 0000000000000000 0000000080200020 00000001ffffffff 0000000000000000
>  page dumped because: VM_BUG_ON_PAGE(PageSlab(page))
>  ------------[ cut here ]------------
>  kernel BUG at ./include/linux/mm.h:628!
>  invalid opcode: 0000 [#1] SMP NOPTI
>  CPU: 77 PID: 504 Comm: kcompactd1 Tainted: G        W         4.19.109-27 #1
>  Hardware name: Yandex T175-N41-Y3N/MY81-EX0-Y3N, BIOS R05 06/20/2019
>  RIP: 0010:isolate_migratepages_block+0x986/0x9b0
> 
> 
> Code in isolate_migratepages_block() was added in commit 119d6d59dcc0
> ("mm, compaction: avoid isolating pinned pages") before adding VM_BUG_ON
> into page_mapcount().
> 
> This race has been predicted in 2015 by Vlastimil Babka (see link below).
> 
> Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
> Fixes: 1d148e218a0d ("mm: add VM_BUG_ON_PAGE() to page_mapcount()")
> Link: https://lore.kernel.org/lkml/557710E1.6060103@suse.cz/
> Link: https://lore.kernel.org/linux-mm/158937872515.474360.5066096871639561424.stgit@buzz/T/ (v1)

Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>

-- 
 Kirill A. Shutemov

Re: [PATCH v2] mm: remove VM_BUG_ON(PageSlab()) from page_mapcount()

[Vlastimil Babka]

On 5/24/20 3:43 PM, Konstantin Khlebnikov wrote:
> Replace superfluous VM_BUG_ON() with comment about correct usage.
> 
> Technically reverts commit 1d148e218a0d0566b1c06f2f45f1436d53b049b2
> ("mm: add VM_BUG_ON_PAGE() to page_mapcount()"), but context have changed.
> 
> Function isolate_migratepages_block() runs some checks out of lru_lock
> when choose pages for migration. After checking PageLRU() it checks extra
> page references by comparing page_count() and page_mapcount(). Between
> these two checks page could be removed from lru, freed and taken by slab.
> 
> As a result this race triggers VM_BUG_ON(PageSlab()) in page_mapcount().
> Race window is tiny. For certain workload this happens around once a year.
> 
> 
>  page:ffffea0105ca9380 count:1 mapcount:0 mapping:ffff88ff7712c180 index:0x0 compound_mapcount: 0
>  flags: 0x500000000008100(slab|head)
>  raw: 0500000000008100 dead000000000100 dead000000000200 ffff88ff7712c180
>  raw: 0000000000000000 0000000080200020 00000001ffffffff 0000000000000000
>  page dumped because: VM_BUG_ON_PAGE(PageSlab(page))
>  ------------[ cut here ]------------
>  kernel BUG at ./include/linux/mm.h:628!
>  invalid opcode: 0000 [#1] SMP NOPTI
>  CPU: 77 PID: 504 Comm: kcompactd1 Tainted: G        W         4.19.109-27 #1
>  Hardware name: Yandex T175-N41-Y3N/MY81-EX0-Y3N, BIOS R05 06/20/2019
>  RIP: 0010:isolate_migratepages_block+0x986/0x9b0
> 
> 
> Code in isolate_migratepages_block() was added in commit 119d6d59dcc0
> ("mm, compaction: avoid isolating pinned pages") before adding VM_BUG_ON
> into page_mapcount().
> 
> This race has been predicted in 2015 by Vlastimil Babka (see link below).

Huh, looks like I made that prediction only half year after that patch has been
posted. Now if only I remembered why... I hope it was just a code inspection
while chasing something else. I most likely didn't actually see the bug happen,
as we don't compile with DEBUG_VM.

> Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
> Fixes: 1d148e218a0d ("mm: add VM_BUG_ON_PAGE() to page_mapcount()")
> Link: https://lore.kernel.org/lkml/557710E1.6060103@suse.cz/
> Link: https://lore.kernel.org/linux-mm/158937872515.474360.5066096871639561424.stgit@buzz/T/ (v1)

With Hugh's wording tweaks,
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Thanks.