[PATCH 1/2] thp: Add compound tail page _mapcount when mapped

[PATCH 1/2] thp: Add compound tail page _mapcount when mapped

[Youquan Song]

With 3.2-rc kernel, the IOMMU 2M page in KVM works. While I try to us IOMMU
1GB page in KVM, I encounter a oops and 1GB page total fail to be used.
The root cause is that 1GB page allocation calls gup_huge_pud() while 2M
 page calls gup_huge_pmd. If compound pages are used and the page is tail page,
gup_huge_pmd increase _mapcount to record tail page are mapped while
gup_huge_pud does not include this process. So when the mapped page is relesed,
it will result in kernel oops because the page does not mark mapped.

This patch add tail process for compound page in 1GB huge page which keeps the
same process as 2M page.

Reproduce like:
1. Add grub boot option: hugepagesz=1G hugepages=8
2. mount -t hugetlbfs -o pagesize=1G hugetlbfs /dev/hugepages
3.qemu-kvm -m 2048 -hda os-kvm.img -cpu kvm64 -smp 4 -mem-path /dev/hugepages
 -net none -device pci-assign,host=07:00.1

kernel BUG at mm/swap.c:114!
invalid opcode: 0000 [#1] SMP
Call Trace:
 [<ffffffff81127482>] put_page+0x15/0x37
 [<ffffffff810067c4>] kvm_release_pfn_clean+0x31/0x36
 [<ffffffff8100b69c>] kvm_iommu_put_pages+0x94/0xb1
 [<ffffffff8100b739>] kvm_iommu_unmap_memslots+0x80/0xb6
 [<ffffffff8100b6b9>] ? kvm_iommu_put_pages+0xb1/0xb1
 [<ffffffff81425cf3>] ? intel_iommu_attach_device+0x13b/0x144
 [<ffffffff8100bc03>] kvm_assign_device+0xba/0x117
 [<ffffffff8100aec2>] kvm_vm_ioctl_assigned_device+0x301/0xa47
 [<ffffffff8100ac6d>] ? kvm_vm_ioctl_assigned_device+0xac/0xa47
 [<ffffffff8104f2a6>] ? native_sched_clock+0x32/0x6b
 [<ffffffff810b0be2>] ? sched_clock_cpu+0x45/0xd4
 [<ffffffff810bc430>] ? trace_hardirqs_off+0xd/0xf
 [<ffffffff810b0cb2>] ? local_clock+0x41/0x5a
 [<ffffffff810bc881>] ? lock_release_holdtime+0x2c/0x129
 [<ffffffff8115760d>] ? cmpxchg_double_slab+0xd0/0x12b
 [<ffffffff81248f27>] ? avc_has_perm_noaudit+0x388/0x399
 [<ffffffff8104f2a6>] ? native_sched_clock+0x32/0x6b
 [<ffffffff8104f2e8>] ? sched_clock+0x9/0xd
 [<ffffffff81007dcb>] kvm_vm_ioctl+0x36c/0x3a2
 [<ffffffff8104f2a6>] ? native_sched_clock+0x32/0x6b
 [<ffffffff8104f2e8>] ? sched_clock+0x9/0xd
 [<ffffffff81174af0>] do_vfs_ioctl+0x49e/0x4e4
 [<ffffffff81174b90>] sys_ioctl+0x5a/0x7c
 [<ffffffff81500dc2>] system_call_fastpath+0x16/0x1b
RIP  [<ffffffff811273d9>] put_compound_page+0xd4/0x168

Reviewed-by: Andrea Arcangeli <aarcange@redhat.com>
Signed-off-by: Youquan Song <youquan.song@intel.com>
---
 arch/x86/mm/gup.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/arch/x86/mm/gup.c b/arch/x86/mm/gup.c
index ea30585..dd74e46 100644
--- a/arch/x86/mm/gup.c
+++ b/arch/x86/mm/gup.c
@@ -201,6 +201,8 @@ static noinline int gup_huge_pud(pud_t pud, unsigned long addr,
 	do {
 		VM_BUG_ON(compound_head(page) != head);
 		pages[*nr] = page;
+		if (PageTail(page))
+			get_huge_page_tail(page);
 		(*nr)++;
 		page++;
 		refs++;
-- 
1.6.4.2

[PATCH 2/2] thp: Set compound tail page _count to zero

[Youquan Song]

At 70b50f94f1644e2aa7cb374819cfd93f3c28d725 "mm: thp: tail page refcounting fix"
 it keep all page_tail->_count zero at all times.
But current kernel, it does not set page_tail->_count to zero if 1GB page is
utilized.
So when IOMMU 1GB page is used at KVM, it wil result in kernel oops because
a tail page its _count does not equal zero.

kernel BUG at include/linux/mm.h:386!
invalid opcode: 0000 [#1] SMP
Call Trace:
 [<ffffffff81072f7f>] gup_pud_range+0xb8/0x19d
 [<ffffffff8107312f>] get_user_pages_fast+0xcb/0x192
 [<ffffffff810bc450>] ? trace_hardirqs_off+0xd/0xf
 [<ffffffff81006a24>] hva_to_pfn+0x119/0x2f2
 [<ffffffff81006c29>] gfn_to_pfn_memslot+0x2c/0x2e
 [<ffffffff8100b909>] kvm_iommu_map_pages+0xfd/0x1c1
 [<ffffffff8100ba49>] kvm_iommu_map_memslots+0x7c/0xbd
 [<ffffffff8100b9cd>] ? kvm_iommu_map_pages+0x1c1/0x1c1
 [<ffffffff8100bb34>] kvm_iommu_map_guest+0xaa/0xbf
 [<ffffffff8100aeb0>] kvm_vm_ioctl_assigned_device+0x2ef/0xa47
 [<ffffffff8100ac6d>] ? kvm_vm_ioctl_assigned_device+0xac/0xa47
 [<ffffffff8104f2a6>] ? native_sched_clock+0x32/0x6b
 [<ffffffff810b0c02>] ? sched_clock_cpu+0x45/0xd4
 [<ffffffff810bc450>] ? trace_hardirqs_off+0xd/0xf
 [<ffffffff810b0cd2>] ? local_clock+0x41/0x5a
 [<ffffffff810bc8a1>] ? lock_release_holdtime+0x2c/0x129
 [<ffffffff8115762d>] ? cmpxchg_double_slab+0xd0/0x12b
 [<ffffffff81248f47>] ? avc_has_perm_noaudit+0x388/0x399
 [<ffffffff8104f2a6>] ? native_sched_clock+0x32/0x6b
 [<ffffffff8104f2e8>] ? sched_clock+0x9/0xd
 [<ffffffff81007dcb>] kvm_vm_ioctl+0x36c/0x3a2
 [<ffffffff8104f2a6>] ? native_sched_clock+0x32/0x6b
 [<ffffffff8104f2e8>] ? sched_clock+0x9/0xd
 [<ffffffff81174b10>] do_vfs_ioctl+0x49e/0x4e4
 [<ffffffff81174bb0>] sys_ioctl+0x5a/0x7c
 [<ffffffff81500e02>] system_call_fastpath+0x16/0x1b
RIP  [<ffffffff81072d13>] gup_huge_pud+0xf2/0x159

Reviewed-by: Andrea Arcangeli <aarcange@redhat.com>
Signed-off-by: Youquan Song <youquan.song@intel.com>
---
 mm/hugetlb.c    |    1 +
 mm/page_alloc.c |    2 +-
 2 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index bb28a5f..73f17c0 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -576,6 +576,7 @@ static void prep_compound_gigantic_page(struct page *page, unsigned long order)
 	__SetPageHead(page);
 	for (i = 1; i < nr_pages; i++, p = mem_map_next(p, page, i)) {
 		__SetPageTail(p);
+		set_page_count(p, 0);
 		p->first_page = page;
 	}
 }
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 9dd443d..850009a 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -356,8 +356,8 @@ void prep_compound_page(struct page *page, unsigned long order)
 	__SetPageHead(page);
 	for (i = 1; i < nr_pages; i++) {
 		struct page *p = page + i;
-
 		__SetPageTail(p);
+		set_page_count(p, 0);
 		p->first_page = page;
 	}
 }
-- 
1.6.4.2

Re: [PATCH 1/2] thp: Add compound tail page _mapcount when mapped

[Andrea Arcangeli]

Both are needed to take care of CONFIG_DEBUG_VM false positives. Worst
case a false positive hits, but if it doesn't there was no risk because
the tail page refcounting isn't actively used by that code and the
refcounting on the head page was done right already.

Thanks,
Andrea

Re: [PATCH 1/2] thp: Add compound tail page _mapcount when mapped

[Andrew Morton]

On Fri, 25 Nov 2011 13:47:09 +0800
Youquan Song <youquan.song@intel.com> wrote:

> With 3.2-rc kernel, the IOMMU 2M page in KVM works. While I try to us IOMMU
> 1GB page in KVM, I encounter a oops and 1GB page total fail to be used.
> The root cause is that 1GB page allocation calls gup_huge_pud() while 2M
>  page calls gup_huge_pmd. If compound pages are used and the page is tail page,
> gup_huge_pmd increase _mapcount to record tail page are mapped while
> gup_huge_pud does not include this process. So when the mapped page is relesed,
> it will result in kernel oops because the page does not mark mapped.
> 
> This patch add tail process for compound page in 1GB huge page which keeps the
> same process as 2M page.
> 
> Reproduce like:
> 1. Add grub boot option: hugepagesz=1G hugepages=8
> 2. mount -t hugetlbfs -o pagesize=1G hugetlbfs /dev/hugepages
> 3.qemu-kvm -m 2048 -hda os-kvm.img -cpu kvm64 -smp 4 -mem-path /dev/hugepages
>  -net none -device pci-assign,host=07:00.1
> 
> kernel BUG at mm/swap.c:114!
>
> ...
>
> --- a/arch/x86/mm/gup.c
> +++ b/arch/x86/mm/gup.c
> @@ -201,6 +201,8 @@ static noinline int gup_huge_pud(pud_t pud, unsigned long addr,
>  	do {
>  		VM_BUG_ON(compound_head(page) != head);
>  		pages[*nr] = page;
> +		if (PageTail(page))
> +			get_huge_page_tail(page);
>  		(*nr)++;
>  		page++;
>  		refs++;

Is the patch also applicable to 3.1.x?

Re: [PATCH 1/2] thp: Add compound tail page _mapcount when mapped

[Andi Kleen]

> > +++ b/arch/x86/mm/gup.c
> > @@ -201,6 +201,8 @@ static noinline int gup_huge_pud(pud_t pud, unsigned long addr,
> >  	do {
> >  		VM_BUG_ON(compound_head(page) != head);
> >  		pages[*nr] = page;
> > +		if (PageTail(page))
> > +			get_huge_page_tail(page);
> >  		(*nr)++;
> >  		page++;
> >  		refs++;
> 
> Is the patch also applicable to 3.1.x?

I suspect it's broken since many kernels back, at least since THP 
was introduced, maybe earlier.

-Andi
-- 
ak@linux.intel.com -- Speaking for myself only.

Re: [PATCH 1/2] thp: Add compound tail page _mapcount when mapped

[Andrea Arcangeli]

On Tue, Nov 29, 2011 at 01:19:27AM +0100, Andi Kleen wrote:
> > > +++ b/arch/x86/mm/gup.c
> > > @@ -201,6 +201,8 @@ static noinline int gup_huge_pud(pud_t pud, unsigned long addr,
> > >  	do {
> > >  		VM_BUG_ON(compound_head(page) != head);
> > >  		pages[*nr] = page;
> > > +		if (PageTail(page))
> > > +			get_huge_page_tail(page);
> > >  		(*nr)++;
> > >  		page++;
> > >  		refs++;
> > 
> > Is the patch also applicable to 3.1.x?
> 
> I suspect it's broken since many kernels back, at least since THP 
> was introduced, maybe earlier.

Correct. And the other patch in this series if applied without the
above too, would make things worse for earlier releases (it'd trigger
the lack of above at first invocation instead of from the second by
having the _count start at 0 instead of 1, so it'd go negative at the
first put_page). The correct thing is to have _count start at 0, but
to increase it with the above during gup_fast (or to increase
_mapcount since 3.2-rc and leave _count at 0 at all times on tail
pages). Both patches should be ok for earlier releases too but I think
it's just a false positive that goes away with DEBUG_VM=n or we should
have noticed sooner (all production systems runs with DEBUG_VM=n of
course). If there is no problem with DEBUG_VM=n like it seems from the
source, I doubt it needs backporting. Also worst case a VM_BUG_ON
hits, there's no risk of memory corruption because the refcounting on
the head pages has always been correct, and that's all it matters
as far as hugetlbfs is concerned.

Re: [PATCH 1/2] thp: Add compound tail page _mapcount when mapped

[Youquan Song]

> > > Is the patch also applicable to 3.1.x?
> > 
> > I suspect it's broken since many kernels back, at least since THP 
> > was introduced, maybe earlier.
> 
> Correct. And the other patch in this series if applied without the
> above too, would make things worse for earlier releases (it'd trigger
> the lack of above at first invocation instead of from the second by
> having the _count start at 0 instead of 1, so it'd go negative at the
> first put_page). The correct thing is to have _count start at 0, but
> to increase it with the above during gup_fast (or to increase
> _mapcount since 3.2-rc and leave _count at 0 at all times on tail
> pages). Both patches should be ok for earlier releases too but I think
> it's just a false positive that goes away with DEBUG_VM=n or we should
> have noticed sooner (all production systems runs with DEBUG_VM=n of
> course). If there is no problem with DEBUG_VM=n like it seems from the
> source, I doubt it needs backporting. Also worst case a VM_BUG_ON
> hits, there's no risk of memory corruption because the refcounting on
> the head pages has always been correct, and that's all it matters
> as far as hugetlbfs is concerned.

I notice 70b50f94f1644e2aa7cb374819cfd93f3c28d725 "mm: thp: tail page
refcounting fix" is in kernel 3.1.1 stable, so this series patches can be also
applied to 3.1 stable. 

I expect the patch 70b50f94f1644e2aa7cb374819cfd93f3c28d725 also go to
3.0 stable, so this series patches also applied to 3.0 stable.
 
Thanks
-Youquan

[PATCH 2/2] thp: Set compound tail page _count to zero

[Youquan Song]

At 70b50f94f1644e2aa7cb374819cfd93f3c28d725 "mm: thp: tail page refcounting fix"
 it keep all page_tail->_count zero at all times.
But kernel with THP, it does not set page_tail->_count to zero if 1GiB page is
utilized.
So when IOMMU 1GiB page is used at KVM, it wil result in kernel oops because
a tail page its _count does not equal zero.

kernel BUG at include/linux/mm.h:386!
invalid opcode: 0000 [#1] SMP
Call Trace:
 [<ffffffff81072f7f>] gup_pud_range+0xb8/0x19d
 [<ffffffff8107312f>] get_user_pages_fast+0xcb/0x192
 [<ffffffff810bc450>] ? trace_hardirqs_off+0xd/0xf
 [<ffffffff81006a24>] hva_to_pfn+0x119/0x2f2
 [<ffffffff81006c29>] gfn_to_pfn_memslot+0x2c/0x2e
 [<ffffffff8100b909>] kvm_iommu_map_pages+0xfd/0x1c1
 [<ffffffff8100ba49>] kvm_iommu_map_memslots+0x7c/0xbd
 [<ffffffff8100b9cd>] ? kvm_iommu_map_pages+0x1c1/0x1c1
 [<ffffffff8100bb34>] kvm_iommu_map_guest+0xaa/0xbf
 [<ffffffff8100aeb0>] kvm_vm_ioctl_assigned_device+0x2ef/0xa47
 [<ffffffff8100ac6d>] ? kvm_vm_ioctl_assigned_device+0xac/0xa47
 [<ffffffff8104f2a6>] ? native_sched_clock+0x32/0x6b
 [<ffffffff810b0c02>] ? sched_clock_cpu+0x45/0xd4
 [<ffffffff810bc450>] ? trace_hardirqs_off+0xd/0xf
 [<ffffffff810b0cd2>] ? local_clock+0x41/0x5a
 [<ffffffff810bc8a1>] ? lock_release_holdtime+0x2c/0x129
 [<ffffffff8115762d>] ? cmpxchg_double_slab+0xd0/0x12b
 [<ffffffff81248f47>] ? avc_has_perm_noaudit+0x388/0x399
 [<ffffffff8104f2a6>] ? native_sched_clock+0x32/0x6b
 [<ffffffff8104f2e8>] ? sched_clock+0x9/0xd
 [<ffffffff81007dcb>] kvm_vm_ioctl+0x36c/0x3a2
 [<ffffffff8104f2a6>] ? native_sched_clock+0x32/0x6b
 [<ffffffff8104f2e8>] ? sched_clock+0x9/0xd
 [<ffffffff81174b10>] do_vfs_ioctl+0x49e/0x4e4
 [<ffffffff81174bb0>] sys_ioctl+0x5a/0x7c
 [<ffffffff81500e02>] system_call_fastpath+0x16/0x1b
RIP  [<ffffffff81072d13>] gup_huge_pud+0xf2/0x159

Reviewed-by: Andrea Arcangeli <aarcange@redhat.com>
Cc: <stable@vger.kernel.org> # 3.0.x
Signed-off-by: Youquan Song <youquan.song@intel.com>
---
 mm/hugetlb.c    |    1 +
 mm/page_alloc.c |    2 +-
 2 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index bb28a5f..73f17c0 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -576,6 +576,7 @@ static void prep_compound_gigantic_page(struct page *page, unsigned long order)
 	__SetPageHead(page);
 	for (i = 1; i < nr_pages; i++, p = mem_map_next(p, page, i)) {
 		__SetPageTail(p);
+		set_page_count(p, 0);
 		p->first_page = page;
 	}
 }
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 9dd443d..850009a 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -356,8 +356,8 @@ void prep_compound_page(struct page *page, unsigned long order)
 	__SetPageHead(page);
 	for (i = 1; i < nr_pages; i++) {
 		struct page *p = page + i;
-
 		__SetPageTail(p);
+		set_page_count(p, 0);
 		p->first_page = page;
 	}
 }
-- 
1.6.4.2