[PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Rui Teng]

Santhosh G reported that call traces occurs when memory-hotplug script is run
with 16Gb hugepages configured.

It was found that the page_hstate(page) will get 0 if the PageHead(page) return
false, Which will cause the bad area access.

Issue:
Call traces occurs when memory-hotplug script is run with 16Gb hugepages configured.

Environment:
ppc64le PowerVM Lpar

root@ltctuleta-lp1:~# uname -r
4.4.0-34-generic

root@ltctuleta-lp1:~# cat /proc/meminfo | grep -i huge
AnonHugePages:         0 kB
HugePages_Total:       2
HugePages_Free:        2
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:   16777216 kB

root@ltctuleta-lp1:~# free -h
              total        used        free      shared  buff/cache   available
Mem:            85G         32G         52G         16M        193M         52G
Swap:           43G          0B         43G

Steps to reproduce:
1 - Download kernel source and enter to the directory- tools/testing/selftests/memory-hotplug/
2 - Run  mem-on-off-test.sh script in it.

System gives call traces like:

offline_memory_expect_success 639: unexpected fail
online-offline 668
[   57.552964] Unable to handle kernel paging request for data at address 0x00000028
[   57.552977] Faulting instruction address: 0xc00000000029bc04
[   57.552987] Oops: Kernel access of bad area, sig: 11 [#1]
[   57.552992] SMP NR_CPUS=2048 NUMA pSeries
[   57.553002] Modules linked in: btrfs xor raid6_pq pseries_rng sunrpc autofs4 ses enclosure nouveau bnx2x i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm vxlan ip6_udp_tunnel ipr udp_tunnel rtc_generic mdio libcrc32c
[   57.553050] CPU: 44 PID: 6518 Comm: mem-on-off-test Not tainted 4.4.0-34-generic #53-Ubuntu
[   57.553059] task: c00000072773c8e0 ti: c000000727780000 task.ti: c000000727780000
[   57.553067] NIP: c00000000029bc04 LR: c00000000029bbdc CTR: c0000000001107f0
[   57.553076] REGS: c000000727783770 TRAP: 0300   Not tainted  (4.4.0-34-generic)
[   57.553083] MSR: 8000000100009033 <SF,EE,ME,IR,DR,RI,LE>  CR: 24242882  XER: 00000002
[   57.553104] CFAR: c000000000008468 DAR: 0000000000000028 DSISR: 40000000 SOFTE: 1
GPR00: c00000000029bbdc c0000007277839f0 c0000000015b5d00 0000000000000000
GPR04: 000000000029d000 0000000000000800 0000000000000000 f00000000a000001
GPR08: f00000000a700020 0000000000000008 c00000000185e270 c000000e7e000050
GPR12: 0000000000002200 c00000000e6ea200 000000000029d000 0000000022000000
GPR16: 1000000000000000 c0000000015e2200 000000000a700000 0000000000000000
GPR20: 0000000000010000 0000000000000100 0000000000000200 c0000000015f16d0
GPR24: c000000001876510 0000000000000000 0000000000000001 c000000001872a00
GPR28: 000000000029d000 f000000000000000 f00000000a700000 000000000029c000
[   57.553211] NIP [c00000000029bc04] dissolve_free_huge_pages+0x154/0x220
[   57.553219] LR [c00000000029bbdc] dissolve_free_huge_pages+0x12c/0x220
[   57.553226] Call Trace:
[   57.553231] [c0000007277839f0] [c00000000029bbdc] dissolve_free_huge_pages+0x12c/0x220 (unreliable)
[   57.553244] [c000000727783a80] [c0000000002dcbc8] __offline_pages.constprop.6+0x3f8/0x900
[   57.553254] [c000000727783bd0] [c0000000006fbb38] memory_subsys_offline+0xa8/0x110
[   57.553265] [c000000727783c00] [c0000000006d6424] device_offline+0x104/0x140
[   57.553274] [c000000727783c40] [c0000000006fba80] store_mem_state+0x180/0x190
[   57.553283] [c000000727783c80] [c0000000006d1e58] dev_attr_store+0x68/0xa0
[   57.553293] [c000000727783cc0] [c000000000398110] sysfs_kf_write+0x80/0xb0
[   57.553302] [c000000727783d00] [c000000000397028] kernfs_fop_write+0x188/0x200
[   57.553312] [c000000727783d50] [c0000000002e190c] __vfs_write+0x6c/0xe0
[   57.553321] [c000000727783d90] [c0000000002e2640] vfs_write+0xc0/0x230
[   57.553329] [c000000727783de0] [c0000000002e367c] SyS_write+0x6c/0x110
[   57.553339] [c000000727783e30] [c000000000009204] system_call+0x38/0xb4
[   57.553346] Instruction dump:
[   57.553351] 7e831836 4bfff991 e91e0028 e8fe0020 7d32e82a f9070008 f8e80000 fabe0020
[   57.553366] fade0028 79294620 79291764 7d234a14 <e9030028> 3908ffff f9030028 81091458
[   57.553383] ---[ end trace 617f7bdd75bcfc10 ]---
[   57.557133]
Segmentation fault

Reported-by: Santhosh G <santhog4@in.ibm.com>
Signed-off-by: Rui Teng <rui.teng@linux.vnet.ibm.com>
---
 mm/hugetlb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 87e11d8..64b5f81 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1442,7 +1442,7 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed,
 static void dissolve_free_huge_page(struct page *page)
 {
 	spin_lock(&hugetlb_lock);
-	if (PageHuge(page) && !page_count(page)) {
+	if (PageHuge(page) && !page_count(page) && PageHead(page)) {
 		struct hstate *h = page_hstate(page);
 		int nid = page_to_nid(page);
 		list_del(&page->lru);
-- 
2.7.4

[PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Rui Teng]

Santhosh G reported that call traces occurs when memory-hotplug script is run
with 16Gb hugepages configured.

It was found that the page_hstate(page) will get 0 if the PageHead(page) return
false, Which will cause the bad area access.

Issue:
Call traces occurs when memory-hotplug script is run with 16Gb hugepages configured.

Environment:
ppc64le PowerVM Lpar

root@ltctuleta-lp1:~# uname -r
4.4.0-34-generic

root@ltctuleta-lp1:~# cat /proc/meminfo | grep -i huge
AnonHugePages:         0 kB
HugePages_Total:       2
HugePages_Free:        2
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:   16777216 kB

root@ltctuleta-lp1:~# free -h
              total        used        free      shared  buff/cache   available
Mem:            85G         32G         52G         16M        193M         52G
Swap:           43G          0B         43G

Steps to reproduce:
1 - Download kernel source and enter to the directory- tools/testing/selftests/memory-hotplug/
2 - Run  mem-on-off-test.sh script in it.

System gives call traces like:

offline_memory_expect_success 639: unexpected fail
online-offline 668
[   57.552964] Unable to handle kernel paging request for data at address 0x00000028
[   57.552977] Faulting instruction address: 0xc00000000029bc04
[   57.552987] Oops: Kernel access of bad area, sig: 11 [#1]
[   57.552992] SMP NR_CPUS=2048 NUMA pSeries
[   57.553002] Modules linked in: btrfs xor raid6_pq pseries_rng sunrpc autofs4 ses enclosure nouveau bnx2x i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm vxlan ip6_udp_tunnel ipr udp_tunnel rtc_generic mdio libcrc32c
[   57.553050] CPU: 44 PID: 6518 Comm: mem-on-off-test Not tainted 4.4.0-34-generic #53-Ubuntu
[   57.553059] task: c00000072773c8e0 ti: c000000727780000 task.ti: c000000727780000
[   57.553067] NIP: c00000000029bc04 LR: c00000000029bbdc CTR: c0000000001107f0
[   57.553076] REGS: c000000727783770 TRAP: 0300   Not tainted  (4.4.0-34-generic)
[   57.553083] MSR: 8000000100009033 <SF,EE,ME,IR,DR,RI,LE>  CR: 24242882  XER: 00000002
[   57.553104] CFAR: c000000000008468 DAR: 0000000000000028 DSISR: 40000000 SOFTE: 1
GPR00: c00000000029bbdc c0000007277839f0 c0000000015b5d00 0000000000000000
GPR04: 000000000029d000 0000000000000800 0000000000000000 f00000000a000001
GPR08: f00000000a700020 0000000000000008 c00000000185e270 c000000e7e000050
GPR12: 0000000000002200 c00000000e6ea200 000000000029d000 0000000022000000
GPR16: 1000000000000000 c0000000015e2200 000000000a700000 0000000000000000
GPR20: 0000000000010000 0000000000000100 0000000000000200 c0000000015f16d0
GPR24: c000000001876510 0000000000000000 0000000000000001 c000000001872a00
GPR28: 000000000029d000 f000000000000000 f00000000a700000 000000000029c000
[   57.553211] NIP [c00000000029bc04] dissolve_free_huge_pages+0x154/0x220
[   57.553219] LR [c00000000029bbdc] dissolve_free_huge_pages+0x12c/0x220
[   57.553226] Call Trace:
[   57.553231] [c0000007277839f0] [c00000000029bbdc] dissolve_free_huge_pages+0x12c/0x220 (unreliable)
[   57.553244] [c000000727783a80] [c0000000002dcbc8] __offline_pages.constprop.6+0x3f8/0x900
[   57.553254] [c000000727783bd0] [c0000000006fbb38] memory_subsys_offline+0xa8/0x110
[   57.553265] [c000000727783c00] [c0000000006d6424] device_offline+0x104/0x140
[   57.553274] [c000000727783c40] [c0000000006fba80] store_mem_state+0x180/0x190
[   57.553283] [c000000727783c80] [c0000000006d1e58] dev_attr_store+0x68/0xa0
[   57.553293] [c000000727783cc0] [c000000000398110] sysfs_kf_write+0x80/0xb0
[   57.553302] [c000000727783d00] [c000000000397028] kernfs_fop_write+0x188/0x200
[   57.553312] [c000000727783d50] [c0000000002e190c] __vfs_write+0x6c/0xe0
[   57.553321] [c000000727783d90] [c0000000002e2640] vfs_write+0xc0/0x230
[   57.553329] [c000000727783de0] [c0000000002e367c] SyS_write+0x6c/0x110
[   57.553339] [c000000727783e30] [c000000000009204] system_call+0x38/0xb4
[   57.553346] Instruction dump:
[   57.553351] 7e831836 4bfff991 e91e0028 e8fe0020 7d32e82a f9070008 f8e80000 fabe0020
[   57.553366] fade0028 79294620 79291764 7d234a14 <e9030028> 3908ffff f9030028 81091458
[   57.553383] ---[ end trace 617f7bdd75bcfc10 ]---
[   57.557133]
Segmentation fault

Reported-by: Santhosh G <santhog4@in.ibm.com>
Signed-off-by: Rui Teng <rui.teng@linux.vnet.ibm.com>
---
 mm/hugetlb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 87e11d8..64b5f81 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1442,7 +1442,7 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed,
 static void dissolve_free_huge_page(struct page *page)
 {
 	spin_lock(&hugetlb_lock);
-	if (PageHuge(page) && !page_count(page)) {
+	if (PageHuge(page) && !page_count(page) && PageHead(page)) {
 		struct hstate *h = page_hstate(page);
 		int nid = page_to_nid(page);
 		list_del(&page->lru);
-- 
2.7.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Dave Hansen]

On 09/13/2016 01:39 AM, Rui Teng wrote:
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index 87e11d8..64b5f81 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -1442,7 +1442,7 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed,
>  static void dissolve_free_huge_page(struct page *page)
>  {
>  	spin_lock(&hugetlb_lock);
> -	if (PageHuge(page) && !page_count(page)) {
> +	if (PageHuge(page) && !page_count(page) && PageHead(page)) {
>  		struct hstate *h = page_hstate(page);
>  		int nid = page_to_nid(page);
>  		list_del(&page->lru);

This is goofy.  What is calling dissolve_free_huge_page() on a tail page?

Hmm:

>         for (pfn = start_pfn; pfn < end_pfn; pfn += 1 << minimum_order)
>                 dissolve_free_huge_page(pfn_to_page(pfn));

So, skip through the area being offlined at the smallest huge page size,
and try to dissolve a huge page in each place one might appear.  But,
after we dissolve a 16GB huge page, we continue looking through the
remaining 15.98GB tail area for huge pages in the area we just
dissolved.  The tail pages are still PageHuge() (how??), and we call
page_hstate() on the tail page whose head was just dissolved.

Note, even with the fix, this taking a (global) spinlock 1023 more times
that it doesn't have to.

This seems inefficient, and fails to fully explain what is going on, and
how tail pages still _look_ like PageHuge(), which seems pretty wrong.

I guess the patch _works_.  But, sheesh, it leaves a lot of room for
improvement.

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Dave Hansen]

On 09/13/2016 01:39 AM, Rui Teng wrote:
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index 87e11d8..64b5f81 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -1442,7 +1442,7 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed,
>  static void dissolve_free_huge_page(struct page *page)
>  {
>  	spin_lock(&hugetlb_lock);
> -	if (PageHuge(page) && !page_count(page)) {
> +	if (PageHuge(page) && !page_count(page) && PageHead(page)) {
>  		struct hstate *h = page_hstate(page);
>  		int nid = page_to_nid(page);
>  		list_del(&page->lru);

This is goofy.  What is calling dissolve_free_huge_page() on a tail page?

Hmm:

>         for (pfn = start_pfn; pfn < end_pfn; pfn += 1 << minimum_order)
>                 dissolve_free_huge_page(pfn_to_page(pfn));

So, skip through the area being offlined at the smallest huge page size,
and try to dissolve a huge page in each place one might appear.  But,
after we dissolve a 16GB huge page, we continue looking through the
remaining 15.98GB tail area for huge pages in the area we just
dissolved.  The tail pages are still PageHuge() (how??), and we call
page_hstate() on the tail page whose head was just dissolved.

Note, even with the fix, this taking a (global) spinlock 1023 more times
that it doesn't have to.

This seems inefficient, and fails to fully explain what is going on, and
how tail pages still _look_ like PageHuge(), which seems pretty wrong.

I guess the patch _works_.  But, sheesh, it leaves a lot of room for
improvement.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Rui Teng]

On 9/14/16 1:32 AM, Dave Hansen wrote:
> On 09/13/2016 01:39 AM, Rui Teng wrote:
>> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
>> index 87e11d8..64b5f81 100644
>> --- a/mm/hugetlb.c
>> +++ b/mm/hugetlb.c
>> @@ -1442,7 +1442,7 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed,
>>  static void dissolve_free_huge_page(struct page *page)
>>  {
>>  	spin_lock(&hugetlb_lock);
>> -	if (PageHuge(page) && !page_count(page)) {
>> +	if (PageHuge(page) && !page_count(page) && PageHead(page)) {
>>  		struct hstate *h = page_hstate(page);
>>  		int nid = page_to_nid(page);
>>  		list_del(&page->lru);
>
> This is goofy.  What is calling dissolve_free_huge_page() on a tail page?
>
> Hmm:
>
>>         for (pfn = start_pfn; pfn < end_pfn; pfn += 1 << minimum_order)
>>                 dissolve_free_huge_page(pfn_to_page(pfn));
>
> So, skip through the area being offlined at the smallest huge page size,
> and try to dissolve a huge page in each place one might appear.  But,
> after we dissolve a 16GB huge page, we continue looking through the
> remaining 15.98GB tail area for huge pages in the area we just
> dissolved.  The tail pages are still PageHuge() (how??), and we call
> page_hstate() on the tail page whose head was just dissolved.
>
> Note, even with the fix, this taking a (global) spinlock 1023 more times
> that it doesn't have to.
>
> This seems inefficient, and fails to fully explain what is going on, and
> how tail pages still _look_ like PageHuge(), which seems pretty wrong.
>
> I guess the patch _works_.  But, sheesh, it leaves a lot of room for
> improvement.
>
Thanks for your suggestion!
How about return the size of page freed from dissolve_free_huge_page(), 
and jump such step on pfn?

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Rui Teng]

On 9/14/16 1:32 AM, Dave Hansen wrote:
> On 09/13/2016 01:39 AM, Rui Teng wrote:
>> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
>> index 87e11d8..64b5f81 100644
>> --- a/mm/hugetlb.c
>> +++ b/mm/hugetlb.c
>> @@ -1442,7 +1442,7 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed,
>>  static void dissolve_free_huge_page(struct page *page)
>>  {
>>  	spin_lock(&hugetlb_lock);
>> -	if (PageHuge(page) && !page_count(page)) {
>> +	if (PageHuge(page) && !page_count(page) && PageHead(page)) {
>>  		struct hstate *h = page_hstate(page);
>>  		int nid = page_to_nid(page);
>>  		list_del(&page->lru);
>
> This is goofy.  What is calling dissolve_free_huge_page() on a tail page?
>
> Hmm:
>
>>         for (pfn = start_pfn; pfn < end_pfn; pfn += 1 << minimum_order)
>>                 dissolve_free_huge_page(pfn_to_page(pfn));
>
> So, skip through the area being offlined at the smallest huge page size,
> and try to dissolve a huge page in each place one might appear.  But,
> after we dissolve a 16GB huge page, we continue looking through the
> remaining 15.98GB tail area for huge pages in the area we just
> dissolved.  The tail pages are still PageHuge() (how??), and we call
> page_hstate() on the tail page whose head was just dissolved.
>
> Note, even with the fix, this taking a (global) spinlock 1023 more times
> that it doesn't have to.
>
> This seems inefficient, and fails to fully explain what is going on, and
> how tail pages still _look_ like PageHuge(), which seems pretty wrong.
>
> I guess the patch _works_.  But, sheesh, it leaves a lot of room for
> improvement.
>
Thanks for your suggestion!
How about return the size of page freed from dissolve_free_huge_page(), 
and jump such step on pfn?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Dave Hansen]

On 09/14/2016 09:33 AM, Rui Teng wrote:
> 
> How about return the size of page freed from dissolve_free_huge_page(),
> and jump such step on pfn?

That would be a nice improvement.

But, as far as describing the initial problem, can you explain how the
tail pages still ended up being PageHuge()?  Seems like dissolving the
huge page should have cleared that.

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Dave Hansen]

On 09/14/2016 09:33 AM, Rui Teng wrote:
> 
> How about return the size of page freed from dissolve_free_huge_page(),
> and jump such step on pfn?

That would be a nice improvement.

But, as far as describing the initial problem, can you explain how the
tail pages still ended up being PageHuge()?  Seems like dissolving the
huge page should have cleared that.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Rui Teng]

On 9/15/16 12:37 AM, Dave Hansen wrote:
> On 09/14/2016 09:33 AM, Rui Teng wrote:
>>
>> How about return the size of page freed from dissolve_free_huge_page(),
>> and jump such step on pfn?
>
> That would be a nice improvement.
>
> But, as far as describing the initial problem, can you explain how the
> tail pages still ended up being PageHuge()?  Seems like dissolving the
> huge page should have cleared that.
>
I use the scripts of tools/testing/selftests/memory-hotplug/mem-on-
off-test.sh to test and reproduce this bug. And I printed the pfn range
on dissolve_free_huge_pages(). The sizes of the pfn range are always
4096, and the ranges are separated.
[   72.362427] start_pfn: 204800, end_pfn: 208896
[   72.371677] start_pfn: 2162688, end_pfn: 2166784
[   72.373945] start_pfn: 217088, end_pfn: 221184
[   72.383218] start_pfn: 2170880, end_pfn: 2174976
[   72.385918] start_pfn: 2306048, end_pfn: 2310144
[   72.388254] start_pfn: 2326528, end_pfn: 2330624

Sometimes, it will report a failure:
[   72.371690] memory offlining [mem 0x2100000000-0x210fffffff] failed

And sometimes, it will report following:
[   72.373956] Offlined Pages 4096

Whether the start_pfn and end_pfn of dissolve_free_huge_pages could be
*random*? If so, the range may not include any page head and start from
tail page, right?

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Rui Teng]

On 9/15/16 12:37 AM, Dave Hansen wrote:
> On 09/14/2016 09:33 AM, Rui Teng wrote:
>>
>> How about return the size of page freed from dissolve_free_huge_page(),
>> and jump such step on pfn?
>
> That would be a nice improvement.
>
> But, as far as describing the initial problem, can you explain how the
> tail pages still ended up being PageHuge()?  Seems like dissolving the
> huge page should have cleared that.
>
I use the scripts of tools/testing/selftests/memory-hotplug/mem-on-
off-test.sh to test and reproduce this bug. And I printed the pfn range
on dissolve_free_huge_pages(). The sizes of the pfn range are always
4096, and the ranges are separated.
[   72.362427] start_pfn: 204800, end_pfn: 208896
[   72.371677] start_pfn: 2162688, end_pfn: 2166784
[   72.373945] start_pfn: 217088, end_pfn: 221184
[   72.383218] start_pfn: 2170880, end_pfn: 2174976
[   72.385918] start_pfn: 2306048, end_pfn: 2310144
[   72.388254] start_pfn: 2326528, end_pfn: 2330624

Sometimes, it will report a failure:
[   72.371690] memory offlining [mem 0x2100000000-0x210fffffff] failed

And sometimes, it will report following:
[   72.373956] Offlined Pages 4096

Whether the start_pfn and end_pfn of dissolve_free_huge_pages could be
*random*? If so, the range may not include any page head and start from
tail page, right?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Dave Hansen]

On 09/16/2016 06:58 AM, Rui Teng wrote:
> On 9/15/16 12:37 AM, Dave Hansen wrote:
>> On 09/14/2016 09:33 AM, Rui Teng wrote:
>> But, as far as describing the initial problem, can you explain how the
>> tail pages still ended up being PageHuge()?  Seems like dissolving the
>> huge page should have cleared that.
>>
> I use the scripts of tools/testing/selftests/memory-hotplug/mem-on-
> off-test.sh to test and reproduce this bug. And I printed the pfn range
> on dissolve_free_huge_pages(). The sizes of the pfn range are always
> 4096, and the ranges are separated.
> [   72.362427] start_pfn: 204800, end_pfn: 208896
> [   72.371677] start_pfn: 2162688, end_pfn: 2166784
> [   72.373945] start_pfn: 217088, end_pfn: 221184
> [   72.383218] start_pfn: 2170880, end_pfn: 2174976
> [   72.385918] start_pfn: 2306048, end_pfn: 2310144
> [   72.388254] start_pfn: 2326528, end_pfn: 2330624
> 
> Sometimes, it will report a failure:
> [   72.371690] memory offlining [mem 0x2100000000-0x210fffffff] failed
> 
> And sometimes, it will report following:
> [   72.373956] Offlined Pages 4096
> 
> Whether the start_pfn and end_pfn of dissolve_free_huge_pages could be
> *random*? If so, the range may not include any page head and start from
> tail page, right?

That's an interesting data point, but it still doesn't quite explain
what is going on.

It seems like there might be parts of gigantic pages that have
PageHuge() set on tail pages, while other parts don't.  If that's true,
we have another bug and your patch just papers over the issue.

I think you really need to find the root cause before we apply this patch.

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Dave Hansen]

On 09/16/2016 06:58 AM, Rui Teng wrote:
> On 9/15/16 12:37 AM, Dave Hansen wrote:
>> On 09/14/2016 09:33 AM, Rui Teng wrote:
>> But, as far as describing the initial problem, can you explain how the
>> tail pages still ended up being PageHuge()?  Seems like dissolving the
>> huge page should have cleared that.
>>
> I use the scripts of tools/testing/selftests/memory-hotplug/mem-on-
> off-test.sh to test and reproduce this bug. And I printed the pfn range
> on dissolve_free_huge_pages(). The sizes of the pfn range are always
> 4096, and the ranges are separated.
> [   72.362427] start_pfn: 204800, end_pfn: 208896
> [   72.371677] start_pfn: 2162688, end_pfn: 2166784
> [   72.373945] start_pfn: 217088, end_pfn: 221184
> [   72.383218] start_pfn: 2170880, end_pfn: 2174976
> [   72.385918] start_pfn: 2306048, end_pfn: 2310144
> [   72.388254] start_pfn: 2326528, end_pfn: 2330624
> 
> Sometimes, it will report a failure:
> [   72.371690] memory offlining [mem 0x2100000000-0x210fffffff] failed
> 
> And sometimes, it will report following:
> [   72.373956] Offlined Pages 4096
> 
> Whether the start_pfn and end_pfn of dissolve_free_huge_pages could be
> *random*? If so, the range may not include any page head and start from
> tail page, right?

That's an interesting data point, but it still doesn't quite explain
what is going on.

It seems like there might be parts of gigantic pages that have
PageHuge() set on tail pages, while other parts don't.  If that's true,
we have another bug and your patch just papers over the issue.

I think you really need to find the root cause before we apply this patch.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Rui Teng]

On 9/17/16 12:25 AM, Dave Hansen wrote:
>
> That's an interesting data point, but it still doesn't quite explain
> what is going on.
>
> It seems like there might be parts of gigantic pages that have
> PageHuge() set on tail pages, while other parts don't.  If that's true,
> we have another bug and your patch just papers over the issue.
>
> I think you really need to find the root cause before we apply this patch.
>
The root cause is the test scripts(tools/testing/selftests/memory-
hotplug/mem-on-off-test.sh) changes online/offline status on memory
blocks other than page header. It will *randomly* select 10% memory
blocks from /sys/devices/system/memory/memory*, and change their
online/offline status.

On my system, the memory block size is 0x10000000:
	[root@elvis-n01-kvm memory]# cat block_size_bytes
	10000000

But the huge page size(16G) is more than this memory block size. So one
huge page is composed by several memory blocks. For example, memory704,
memory705, memory706 and so on. Then memory704 will contain a head
page, but memory705 will *only* contain tail pages. So the problem will
happened on it, if we call:
	#echo offline > memory705/state

That's why we need a PageHead() check now, and why this problem does
not happened on systems with smaller huge page such as 16M.

As far as the PageHuge() set, I think PageHuge() will return true for
all tail pages. Because it will get the compound_head for tail page,
and then get its huge page flag.
	page = compound_head(page);

And as far as the failure message, if one memory block is in use, it
will return failure when offline it.

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Rui Teng]

On 9/17/16 12:25 AM, Dave Hansen wrote:
>
> That's an interesting data point, but it still doesn't quite explain
> what is going on.
>
> It seems like there might be parts of gigantic pages that have
> PageHuge() set on tail pages, while other parts don't.  If that's true,
> we have another bug and your patch just papers over the issue.
>
> I think you really need to find the root cause before we apply this patch.
>
The root cause is the test scripts(tools/testing/selftests/memory-
hotplug/mem-on-off-test.sh) changes online/offline status on memory
blocks other than page header. It will *randomly* select 10% memory
blocks from /sys/devices/system/memory/memory*, and change their
online/offline status.

On my system, the memory block size is 0x10000000:
	[root@elvis-n01-kvm memory]# cat block_size_bytes
	10000000

But the huge page size(16G) is more than this memory block size. So one
huge page is composed by several memory blocks. For example, memory704,
memory705, memory706 and so on. Then memory704 will contain a head
page, but memory705 will *only* contain tail pages. So the problem will
happened on it, if we call:
	#echo offline > memory705/state

That's why we need a PageHead() check now, and why this problem does
not happened on systems with smaller huge page such as 16M.

As far as the PageHuge() set, I think PageHuge() will return true for
all tail pages. Because it will get the compound_head for tail page,
and then get its huge page flag.
	page = compound_head(page);

And as far as the failure message, if one memory block is in use, it
will return failure when offline it.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Dave Hansen]

On 09/20/2016 07:45 AM, Rui Teng wrote:
> On 9/17/16 12:25 AM, Dave Hansen wrote:
>>
>> That's an interesting data point, but it still doesn't quite explain
>> what is going on.
>>
>> It seems like there might be parts of gigantic pages that have
>> PageHuge() set on tail pages, while other parts don't.  If that's true,
>> we have another bug and your patch just papers over the issue.
>>
>> I think you really need to find the root cause before we apply this
>> patch.
>>
> The root cause is the test scripts(tools/testing/selftests/memory-
> hotplug/mem-on-off-test.sh) changes online/offline status on memory
> blocks other than page header. It will *randomly* select 10% memory
> blocks from /sys/devices/system/memory/memory*, and change their
> online/offline status.

Ahh, that does explain it!  Thanks for digging into that!

> That's why we need a PageHead() check now, and why this problem does
> not happened on systems with smaller huge page such as 16M.
> 
> As far as the PageHuge() set, I think PageHuge() will return true for
> all tail pages. Because it will get the compound_head for tail page,
> and then get its huge page flag.
>     page = compound_head(page);
> 
> And as far as the failure message, if one memory block is in use, it
> will return failure when offline it.

That's good, but aren't we still left with a situation where we've
offlined and dissolved the _middle_ of a gigantic huge page while the
head page is still in place and online?

That seems bad.

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Dave Hansen]

On 09/20/2016 07:45 AM, Rui Teng wrote:
> On 9/17/16 12:25 AM, Dave Hansen wrote:
>>
>> That's an interesting data point, but it still doesn't quite explain
>> what is going on.
>>
>> It seems like there might be parts of gigantic pages that have
>> PageHuge() set on tail pages, while other parts don't.  If that's true,
>> we have another bug and your patch just papers over the issue.
>>
>> I think you really need to find the root cause before we apply this
>> patch.
>>
> The root cause is the test scripts(tools/testing/selftests/memory-
> hotplug/mem-on-off-test.sh) changes online/offline status on memory
> blocks other than page header. It will *randomly* select 10% memory
> blocks from /sys/devices/system/memory/memory*, and change their
> online/offline status.

Ahh, that does explain it!  Thanks for digging into that!

> That's why we need a PageHead() check now, and why this problem does
> not happened on systems with smaller huge page such as 16M.
> 
> As far as the PageHuge() set, I think PageHuge() will return true for
> all tail pages. Because it will get the compound_head for tail page,
> and then get its huge page flag.
>     page = compound_head(page);
> 
> And as far as the failure message, if one memory block is in use, it
> will return failure when offline it.

That's good, but aren't we still left with a situation where we've
offlined and dissolved the _middle_ of a gigantic huge page while the
head page is still in place and online?

That seems bad.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Rui Teng]

On 9/20/16 10:53 PM, Dave Hansen wrote:
> On 09/20/2016 07:45 AM, Rui Teng wrote:
>> On 9/17/16 12:25 AM, Dave Hansen wrote:
>>>
>>> That's an interesting data point, but it still doesn't quite explain
>>> what is going on.
>>>
>>> It seems like there might be parts of gigantic pages that have
>>> PageHuge() set on tail pages, while other parts don't.  If that's true,
>>> we have another bug and your patch just papers over the issue.
>>>
>>> I think you really need to find the root cause before we apply this
>>> patch.
>>>
>> The root cause is the test scripts(tools/testing/selftests/memory-
>> hotplug/mem-on-off-test.sh) changes online/offline status on memory
>> blocks other than page header. It will *randomly* select 10% memory
>> blocks from /sys/devices/system/memory/memory*, and change their
>> online/offline status.
>
> Ahh, that does explain it!  Thanks for digging into that!
>
>> That's why we need a PageHead() check now, and why this problem does
>> not happened on systems with smaller huge page such as 16M.
>>
>> As far as the PageHuge() set, I think PageHuge() will return true for
>> all tail pages. Because it will get the compound_head for tail page,
>> and then get its huge page flag.
>>     page = compound_head(page);
>>
>> And as far as the failure message, if one memory block is in use, it
>> will return failure when offline it.
>
> That's good, but aren't we still left with a situation where we've
> offlined and dissolved the _middle_ of a gigantic huge page while the
> head page is still in place and online?
>
> That seems bad.
>
What about refusing to change the status for such memory block, if it
contains a huge page which larger than itself? (function
memory_block_action())

I think it will not affect the hot-plug function too much. We can
change the nr_hugepages to zero first, if we really want to hot-plug a
memory.

And I also found that the __test_page_isolated_in_pageblock() function
can not handle a gigantic page well. It will cause a device busy error
later. I am still investigating on that.

Any suggestion?

Thanks!

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Rui Teng]

On 9/20/16 10:53 PM, Dave Hansen wrote:
> On 09/20/2016 07:45 AM, Rui Teng wrote:
>> On 9/17/16 12:25 AM, Dave Hansen wrote:
>>>
>>> That's an interesting data point, but it still doesn't quite explain
>>> what is going on.
>>>
>>> It seems like there might be parts of gigantic pages that have
>>> PageHuge() set on tail pages, while other parts don't.  If that's true,
>>> we have another bug and your patch just papers over the issue.
>>>
>>> I think you really need to find the root cause before we apply this
>>> patch.
>>>
>> The root cause is the test scripts(tools/testing/selftests/memory-
>> hotplug/mem-on-off-test.sh) changes online/offline status on memory
>> blocks other than page header. It will *randomly* select 10% memory
>> blocks from /sys/devices/system/memory/memory*, and change their
>> online/offline status.
>
> Ahh, that does explain it!  Thanks for digging into that!
>
>> That's why we need a PageHead() check now, and why this problem does
>> not happened on systems with smaller huge page such as 16M.
>>
>> As far as the PageHuge() set, I think PageHuge() will return true for
>> all tail pages. Because it will get the compound_head for tail page,
>> and then get its huge page flag.
>>     page = compound_head(page);
>>
>> And as far as the failure message, if one memory block is in use, it
>> will return failure when offline it.
>
> That's good, but aren't we still left with a situation where we've
> offlined and dissolved the _middle_ of a gigantic huge page while the
> head page is still in place and online?
>
> That seems bad.
>
What about refusing to change the status for such memory block, if it
contains a huge page which larger than itself? (function
memory_block_action())

I think it will not affect the hot-plug function too much. We can
change the nr_hugepages to zero first, if we really want to hot-plug a
memory.

And I also found that the __test_page_isolated_in_pageblock() function
can not handle a gigantic page well. It will cause a device busy error
later. I am still investigating on that.

Any suggestion?

Thanks!

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Dave Hansen]

On 09/20/2016 08:52 AM, Rui Teng wrote:
> On 9/20/16 10:53 PM, Dave Hansen wrote:
...
>> That's good, but aren't we still left with a situation where we've
>> offlined and dissolved the _middle_ of a gigantic huge page while the
>> head page is still in place and online?
>>
>> That seems bad.
>>
> What about refusing to change the status for such memory block, if it
> contains a huge page which larger than itself? (function
> memory_block_action())

How will this be visible to users, though?  That sounds like you simply
won't be able to offline memory with gigantic huge pages.

> I think it will not affect the hot-plug function too much. We can
> change the nr_hugepages to zero first, if we really want to hot-plug a
> memory.

Is that really feasible?  Suggest that folks stop using hugetlbfs before
offlining any memory?  Isn't the entire point of hotplug to keep the
system running while you change the memory present?  Doing this would
require that you stop your applications that are using huge pages.

With gigantic pages, you may also never get them back if you do this.

> And I also found that the __test_page_isolated_in_pageblock() function
> can not handle a gigantic page well. It will cause a device busy error
> later. I am still investigating on that.
> 
> Any suggestion?

It sounds like the _first_ offline operation needs to dissolve an
_entire_ page if that page has any portion in the section being
offlined.  I'm not quite sure where the page should live after that, but
I'm not sure of any other way to do this sanely.

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Dave Hansen]

On 09/20/2016 08:52 AM, Rui Teng wrote:
> On 9/20/16 10:53 PM, Dave Hansen wrote:
...
>> That's good, but aren't we still left with a situation where we've
>> offlined and dissolved the _middle_ of a gigantic huge page while the
>> head page is still in place and online?
>>
>> That seems bad.
>>
> What about refusing to change the status for such memory block, if it
> contains a huge page which larger than itself? (function
> memory_block_action())

How will this be visible to users, though?  That sounds like you simply
won't be able to offline memory with gigantic huge pages.

> I think it will not affect the hot-plug function too much. We can
> change the nr_hugepages to zero first, if we really want to hot-plug a
> memory.

Is that really feasible?  Suggest that folks stop using hugetlbfs before
offlining any memory?  Isn't the entire point of hotplug to keep the
system running while you change the memory present?  Doing this would
require that you stop your applications that are using huge pages.

With gigantic pages, you may also never get them back if you do this.

> And I also found that the __test_page_isolated_in_pageblock() function
> can not handle a gigantic page well. It will cause a device busy error
> later. I am still investigating on that.
> 
> Any suggestion?

It sounds like the _first_ offline operation needs to dissolve an
_entire_ page if that page has any portion in the section being
offlined.  I'm not quite sure where the page should live after that, but
I'm not sure of any other way to do this sanely.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Michal Hocko]

On Tue 20-09-16 10:43:13, Dave Hansen wrote:
> On 09/20/2016 08:52 AM, Rui Teng wrote:
> > On 9/20/16 10:53 PM, Dave Hansen wrote:
> ...
> >> That's good, but aren't we still left with a situation where we've
> >> offlined and dissolved the _middle_ of a gigantic huge page while the
> >> head page is still in place and online?
> >>
> >> That seems bad.
> >>
> > What about refusing to change the status for such memory block, if it
> > contains a huge page which larger than itself? (function
> > memory_block_action())
> 
> How will this be visible to users, though?  That sounds like you simply
> won't be able to offline memory with gigantic huge pages.

I might be missing something but Is this any different from a regular
failure when the memory cannot be freed? I mean
/sys/devices/system/memory/memory API doesn't give you any hint whether
the memory in the particular block is used and
unmigrateable.

-- 
Michal Hocko
SUSE Labs

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Michal Hocko]

On Tue 20-09-16 10:43:13, Dave Hansen wrote:
> On 09/20/2016 08:52 AM, Rui Teng wrote:
> > On 9/20/16 10:53 PM, Dave Hansen wrote:
> ...
> >> That's good, but aren't we still left with a situation where we've
> >> offlined and dissolved the _middle_ of a gigantic huge page while the
> >> head page is still in place and online?
> >>
> >> That seems bad.
> >>
> > What about refusing to change the status for such memory block, if it
> > contains a huge page which larger than itself? (function
> > memory_block_action())
> 
> How will this be visible to users, though?  That sounds like you simply
> won't be able to offline memory with gigantic huge pages.

I might be missing something but Is this any different from a regular
failure when the memory cannot be freed? I mean
/sys/devices/system/memory/memory API doesn't give you any hint whether
the memory in the particular block is used and
unmigrateable.

-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Dave Hansen]

On 09/21/2016 05:05 AM, Michal Hocko wrote:
> On Tue 20-09-16 10:43:13, Dave Hansen wrote:
>> On 09/20/2016 08:52 AM, Rui Teng wrote:
>>> On 9/20/16 10:53 PM, Dave Hansen wrote:
>> ...
>>>> That's good, but aren't we still left with a situation where we've
>>>> offlined and dissolved the _middle_ of a gigantic huge page while the
>>>> head page is still in place and online?
>>>>
>>>> That seems bad.
>>>>
>>> What about refusing to change the status for such memory block, if it
>>> contains a huge page which larger than itself? (function
>>> memory_block_action())
>>
>> How will this be visible to users, though?  That sounds like you simply
>> won't be able to offline memory with gigantic huge pages.
> 
> I might be missing something but Is this any different from a regular
> failure when the memory cannot be freed? I mean
> /sys/devices/system/memory/memory API doesn't give you any hint whether
> the memory in the particular block is used and
> unmigrateable.

It's OK to have free hugetlbfs pages in an area that's being offline'd.
 If we did that, it would not be OK to have a free gigantic hugetlbfs
page that's larger than the area being offlined.

It would be a wee bit goofy to have the requirement that userspace go
find all the gigantic pages and make them non-gigantic before trying to
offline something.

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Dave Hansen]

On 09/21/2016 05:05 AM, Michal Hocko wrote:
> On Tue 20-09-16 10:43:13, Dave Hansen wrote:
>> On 09/20/2016 08:52 AM, Rui Teng wrote:
>>> On 9/20/16 10:53 PM, Dave Hansen wrote:
>> ...
>>>> That's good, but aren't we still left with a situation where we've
>>>> offlined and dissolved the _middle_ of a gigantic huge page while the
>>>> head page is still in place and online?
>>>>
>>>> That seems bad.
>>>>
>>> What about refusing to change the status for such memory block, if it
>>> contains a huge page which larger than itself? (function
>>> memory_block_action())
>>
>> How will this be visible to users, though?  That sounds like you simply
>> won't be able to offline memory with gigantic huge pages.
> 
> I might be missing something but Is this any different from a regular
> failure when the memory cannot be freed? I mean
> /sys/devices/system/memory/memory API doesn't give you any hint whether
> the memory in the particular block is used and
> unmigrateable.

It's OK to have free hugetlbfs pages in an area that's being offline'd.
 If we did that, it would not be OK to have a free gigantic hugetlbfs
page that's larger than the area being offlined.

It would be a wee bit goofy to have the requirement that userspace go
find all the gigantic pages and make them non-gigantic before trying to
offline something.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Michal Hocko]

On Wed 21-09-16 09:04:31, Dave Hansen wrote:
> On 09/21/2016 05:05 AM, Michal Hocko wrote:
> > On Tue 20-09-16 10:43:13, Dave Hansen wrote:
> >> On 09/20/2016 08:52 AM, Rui Teng wrote:
> >>> On 9/20/16 10:53 PM, Dave Hansen wrote:
> >> ...
> >>>> That's good, but aren't we still left with a situation where we've
> >>>> offlined and dissolved the _middle_ of a gigantic huge page while the
> >>>> head page is still in place and online?
> >>>>
> >>>> That seems bad.
> >>>>
> >>> What about refusing to change the status for such memory block, if it
> >>> contains a huge page which larger than itself? (function
> >>> memory_block_action())
> >>
> >> How will this be visible to users, though?  That sounds like you simply
> >> won't be able to offline memory with gigantic huge pages.
> > 
> > I might be missing something but Is this any different from a regular
> > failure when the memory cannot be freed? I mean
> > /sys/devices/system/memory/memory API doesn't give you any hint whether
> > the memory in the particular block is used and
> > unmigrateable.
> 
> It's OK to have free hugetlbfs pages in an area that's being offline'd.
>  If we did that, it would not be OK to have a free gigantic hugetlbfs
> page that's larger than the area being offlined.

That was not my point. I wasn't very clear probably. Offlining can fail
which shouldn't be really surprising. There might be a kernel allocation
in the particular block which cannot be migrated so failures are to be
expected. I just do not see how offlining in the middle of a gigantic
page is any different from having any other unmovable allocation in a
block. That being said, why don't we simply refuse to offline a block
which is in the middle of a gigantic page.
 
> It would be a wee bit goofy to have the requirement that userspace go
> find all the gigantic pages and make them non-gigantic before trying to
> offline something.

yes

-- 
Michal Hocko
SUSE Labs

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Michal Hocko]

On Wed 21-09-16 09:04:31, Dave Hansen wrote:
> On 09/21/2016 05:05 AM, Michal Hocko wrote:
> > On Tue 20-09-16 10:43:13, Dave Hansen wrote:
> >> On 09/20/2016 08:52 AM, Rui Teng wrote:
> >>> On 9/20/16 10:53 PM, Dave Hansen wrote:
> >> ...
> >>>> That's good, but aren't we still left with a situation where we've
> >>>> offlined and dissolved the _middle_ of a gigantic huge page while the
> >>>> head page is still in place and online?
> >>>>
> >>>> That seems bad.
> >>>>
> >>> What about refusing to change the status for such memory block, if it
> >>> contains a huge page which larger than itself? (function
> >>> memory_block_action())
> >>
> >> How will this be visible to users, though?  That sounds like you simply
> >> won't be able to offline memory with gigantic huge pages.
> > 
> > I might be missing something but Is this any different from a regular
> > failure when the memory cannot be freed? I mean
> > /sys/devices/system/memory/memory API doesn't give you any hint whether
> > the memory in the particular block is used and
> > unmigrateable.
> 
> It's OK to have free hugetlbfs pages in an area that's being offline'd.
>  If we did that, it would not be OK to have a free gigantic hugetlbfs
> page that's larger than the area being offlined.

That was not my point. I wasn't very clear probably. Offlining can fail
which shouldn't be really surprising. There might be a kernel allocation
in the particular block which cannot be migrated so failures are to be
expected. I just do not see how offlining in the middle of a gigantic
page is any different from having any other unmovable allocation in a
block. That being said, why don't we simply refuse to offline a block
which is in the middle of a gigantic page.
 
> It would be a wee bit goofy to have the requirement that userspace go
> find all the gigantic pages and make them non-gigantic before trying to
> offline something.

yes

-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Dave Hansen]

On 09/21/2016 09:27 AM, Michal Hocko wrote:
> That was not my point. I wasn't very clear probably. Offlining can fail
> which shouldn't be really surprising. There might be a kernel allocation
> in the particular block which cannot be migrated so failures are to be
> expected. I just do not see how offlining in the middle of a gigantic
> page is any different from having any other unmovable allocation in a
> block. That being said, why don't we simply refuse to offline a block
> which is in the middle of a gigantic page.

Don't we want to minimize the things that can cause an offline to fail?
 The code to fix it here doesn't seem too bad.

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Dave Hansen]

On 09/21/2016 09:27 AM, Michal Hocko wrote:
> That was not my point. I wasn't very clear probably. Offlining can fail
> which shouldn't be really surprising. There might be a kernel allocation
> in the particular block which cannot be migrated so failures are to be
> expected. I just do not see how offlining in the middle of a gigantic
> page is any different from having any other unmovable allocation in a
> block. That being said, why don't we simply refuse to offline a block
> which is in the middle of a gigantic page.

Don't we want to minimize the things that can cause an offline to fail?
 The code to fix it here doesn't seem too bad.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Michal Hocko]

On Wed 21-09-16 09:32:10, Dave Hansen wrote:
> On 09/21/2016 09:27 AM, Michal Hocko wrote:
> > That was not my point. I wasn't very clear probably. Offlining can fail
> > which shouldn't be really surprising. There might be a kernel allocation
> > in the particular block which cannot be migrated so failures are to be
> > expected. I just do not see how offlining in the middle of a gigantic
> > page is any different from having any other unmovable allocation in a
> > block. That being said, why don't we simply refuse to offline a block
> > which is in the middle of a gigantic page.
> 
> Don't we want to minimize the things that can cause an offline to fail?
> The code to fix it here doesn't seem too bad.

I am not really sure. So say somebody wants to offline few blocks (does
offlining anything but whole nodes make any sense btw.?) and that
happens to be in the middle of a gigantic huge page which is not really
that easy to allocate, do we want to free it in order to do the offline?
To me it sounds like keeping the gigantic page should be preffered but
I have to admit I do not really understand the per-block offlining
though.

-- 
Michal Hocko
SUSE Labs

Re: [PATCH] memory-hotplug: Fix bad area access on dissolve_free_huge_pages()

[Michal Hocko]

On Wed 21-09-16 09:32:10, Dave Hansen wrote:
> On 09/21/2016 09:27 AM, Michal Hocko wrote:
> > That was not my point. I wasn't very clear probably. Offlining can fail
> > which shouldn't be really surprising. There might be a kernel allocation
> > in the particular block which cannot be migrated so failures are to be
> > expected. I just do not see how offlining in the middle of a gigantic
> > page is any different from having any other unmovable allocation in a
> > block. That being said, why don't we simply refuse to offline a block
> > which is in the middle of a gigantic page.
> 
> Don't we want to minimize the things that can cause an offline to fail?
> The code to fix it here doesn't seem too bad.

I am not really sure. So say somebody wants to offline few blocks (does
offlining anything but whole nodes make any sense btw.?) and that
happens to be in the middle of a gigantic huge page which is not really
that easy to allocate, do we want to free it in order to do the offline?
To me it sounds like keeping the gigantic page should be preffered but
I have to admit I do not really understand the per-block offlining
though.

-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>