* [PATCH] mm: fix oom work when memory is under pressure @ 2016-09-06 14:47 zhongjiang 2016-09-09 11:44 ` Michal Hocko 0 siblings, 1 reply; 26+ messages in thread From: zhongjiang @ 2016-09-06 14:47 UTC (permalink / raw) To: akpm; +Cc: vbabka, mhocko, rientjes, linux-mm From: zhong jiang <zhongjiang@huawei.com> Some hungtask come up when I run the trinity, and OOM occurs frequently. A task hold lock to allocate memory, due to the low memory, it will lead to oom. at the some time , it will retry because it find that oom is in progress. but it always allocate fails, the freed memory was taken away quickly. The patch fix it by limit times to avoid hungtask and livelock come up. Signed-off-by: zhong jiang <zhongjiang@huawei.com> --- mm/page_alloc.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a178b1d..0dcf08b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3457,6 +3457,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, enum compact_result compact_result; int compaction_retries = 0; int no_progress_loops = 0; + int oom_failed = 0; /* * In the slowpath, we sanity check order to avoid ever trying to @@ -3645,8 +3646,13 @@ retry: page = __alloc_pages_may_oom(gfp_mask, order, ac, &did_some_progress); if (page) goto got_pg; + else + oom_failed++; + + /* more than limited times will drop out */ + if (oom_failed > MAX_RECLAIM_RETRIES) + goto nopage; - /* Retry as long as the OOM killer is making progress */ if (did_some_progress) { no_progress_loops = 0; goto retry; -- 1.8.3.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-06 14:47 [PATCH] mm: fix oom work when memory is under pressure zhongjiang @ 2016-09-09 11:44 ` Michal Hocko 2016-09-12 9:51 ` zhong jiang 0 siblings, 1 reply; 26+ messages in thread From: Michal Hocko @ 2016-09-09 11:44 UTC (permalink / raw) To: zhongjiang; +Cc: akpm, vbabka, rientjes, linux-mm On Tue 06-09-16 22:47:06, zhongjiang wrote: > From: zhong jiang <zhongjiang@huawei.com> > > Some hungtask come up when I run the trinity, and OOM occurs > frequently. > A task hold lock to allocate memory, due to the low memory, > it will lead to oom. at the some time , it will retry because > it find that oom is in progress. but it always allocate fails, > the freed memory was taken away quickly. > The patch fix it by limit times to avoid hungtask and livelock > come up. Which kernel has shown this issue? Since 4.6 IIRC we have oom reaper responsible for the async memory reclaim from the oom victim and later changes should help to reduce oom lockups even further. That being said this is not a right approach. It is even incorrect because it allows __GFP_NOFAIL to fail now. So NAK to this patch. > Signed-off-by: zhong jiang <zhongjiang@huawei.com> > --- > mm/page_alloc.c | 8 +++++++- > 1 file changed, 7 insertions(+), 1 deletion(-) > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index a178b1d..0dcf08b 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -3457,6 +3457,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, > enum compact_result compact_result; > int compaction_retries = 0; > int no_progress_loops = 0; > + int oom_failed = 0; > > /* > * In the slowpath, we sanity check order to avoid ever trying to > @@ -3645,8 +3646,13 @@ retry: > page = __alloc_pages_may_oom(gfp_mask, order, ac, &did_some_progress); > if (page) > goto got_pg; > + else > + oom_failed++; > + > + /* more than limited times will drop out */ > + if (oom_failed > MAX_RECLAIM_RETRIES) > + goto nopage; > > - /* Retry as long as the OOM killer is making progress */ > if (did_some_progress) { > no_progress_loops = 0; > goto retry; > -- > 1.8.3.1 -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-09 11:44 ` Michal Hocko @ 2016-09-12 9:51 ` zhong jiang 2016-09-12 11:13 ` Michal Hocko 0 siblings, 1 reply; 26+ messages in thread From: zhong jiang @ 2016-09-12 9:51 UTC (permalink / raw) To: Michal Hocko; +Cc: akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo On 2016/9/9 19:44, Michal Hocko wrote: > On Tue 06-09-16 22:47:06, zhongjiang wrote: >> From: zhong jiang <zhongjiang@huawei.com> >> >> Some hungtask come up when I run the trinity, and OOM occurs >> frequently. >> A task hold lock to allocate memory, due to the low memory, >> it will lead to oom. at the some time , it will retry because >> it find that oom is in progress. but it always allocate fails, >> the freed memory was taken away quickly. >> The patch fix it by limit times to avoid hungtask and livelock >> come up. > Which kernel has shown this issue? Since 4.6 IIRC we have oom reaper > responsible for the async memory reclaim from the oom victim and later > changes should help to reduce oom lockups even further. > > That being said this is not a right approach. It is even incorrect > because it allows __GFP_NOFAIL to fail now. So NAK to this patch. > >> Signed-off-by: zhong jiang <zhongjiang@huawei.com> >> --- >> mm/page_alloc.c | 8 +++++++- >> 1 file changed, 7 insertions(+), 1 deletion(-) >> >> diff --git a/mm/page_alloc.c b/mm/page_alloc.c >> index a178b1d..0dcf08b 100644 >> --- a/mm/page_alloc.c >> +++ b/mm/page_alloc.c >> @@ -3457,6 +3457,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, >> enum compact_result compact_result; >> int compaction_retries = 0; >> int no_progress_loops = 0; >> + int oom_failed = 0; >> >> /* >> * In the slowpath, we sanity check order to avoid ever trying to >> @@ -3645,8 +3646,13 @@ retry: >> page = __alloc_pages_may_oom(gfp_mask, order, ac, &did_some_progress); >> if (page) >> goto got_pg; >> + else >> + oom_failed++; >> + >> + /* more than limited times will drop out */ >> + if (oom_failed > MAX_RECLAIM_RETRIES) >> + goto nopage; >> >> - /* Retry as long as the OOM killer is making progress */ >> if (did_some_progress) { >> no_progress_loops = 0; >> goto retry; >> -- >> 1.8.3.1 hi, Michal oom reaper indeed can accelerate the recovery of memory, but the patch solve the extreme scenario, I hit it by runing trinity. I think the scenario can happen whether oom reaper or not. The __GFP_NOFAIL should be considered. Thank you for reminding. The following patch is updated. Thanks zhongjiang diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a178b1d..47804c1 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3457,6 +3457,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, enum compact_result compact_result; int compaction_retries = 0; int no_progress_loops = 0; + int oom_failed = 0; /* * In the slowpath, we sanity check order to avoid ever trying to @@ -3645,8 +3646,15 @@ retry: page = __alloc_pages_may_oom(gfp_mask, order, ac, &did_some_progress); if (page) goto got_pg; + else + oom_failed++; + + /* more than limited times will drop out */ + if (oom_failed > MAX_RECLAIM_RETRIES) { + WARN_ON_ONCE(gfp_mask & __GFP_NOFAIL); + goto nopage; + } - /* Retry as long as the OOM killer is making progress */ if (did_some_progress) { no_progress_loops = 0; goto retry; -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-12 9:51 ` zhong jiang @ 2016-09-12 11:13 ` Michal Hocko 2016-09-12 13:42 ` zhong jiang 0 siblings, 1 reply; 26+ messages in thread From: Michal Hocko @ 2016-09-12 11:13 UTC (permalink / raw) To: zhong jiang; +Cc: akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo On Mon 12-09-16 17:51:06, zhong jiang wrote: [...] > hi, Michal > oom reaper indeed can accelerate the recovery of memory, but the patch > solve the extreme scenario, I hit it by runing trinity. I think the > scenario can happen whether oom reaper or not. could you be more specific about the case when the oom reaper and the current oom code led to the oom deadlock? -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-12 11:13 ` Michal Hocko @ 2016-09-12 13:42 ` zhong jiang 2016-09-12 17:44 ` Michal Hocko 0 siblings, 1 reply; 26+ messages in thread From: zhong jiang @ 2016-09-12 13:42 UTC (permalink / raw) To: Michal Hocko; +Cc: akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo On 2016/9/12 19:13, Michal Hocko wrote: > On Mon 12-09-16 17:51:06, zhong jiang wrote: > [...] >> hi, Michal >> oom reaper indeed can accelerate the recovery of memory, but the patch >> solve the extreme scenario, I hit it by runing trinity. I think the >> scenario can happen whether oom reaper or not. > could you be more specific about the case when the oom reaper and the > current oom code led to the oom deadlock? It is not the oom deadlock. It will lead to hungtask. The explain is as follows. process A occupy a resource and lock it. then A need to allocate memory when memory is very low. at the some time, oom will come up and return directly. because it find other process is freeing memory in same zone. however, the freed memory is taken away by another process. it will lead to A oom again and again. process B still wait some resource holded by A. so B will obtain the lock until A release the resource. therefor, if A spend much time to obtain memory, B will hungtask. Thanks zhongjiang -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-12 13:42 ` zhong jiang @ 2016-09-12 17:44 ` Michal Hocko 2016-09-13 13:13 ` zhong jiang 0 siblings, 1 reply; 26+ messages in thread From: Michal Hocko @ 2016-09-12 17:44 UTC (permalink / raw) To: zhong jiang; +Cc: akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo On Mon 12-09-16 21:42:28, zhong jiang wrote: > On 2016/9/12 19:13, Michal Hocko wrote: > > On Mon 12-09-16 17:51:06, zhong jiang wrote: > > [...] > >> hi, Michal > >> oom reaper indeed can accelerate the recovery of memory, but the patch > >> solve the extreme scenario, I hit it by runing trinity. I think the > >> scenario can happen whether oom reaper or not. > > could you be more specific about the case when the oom reaper and the > > current oom code led to the oom deadlock? > > It is not the oom deadlock. It will lead to hungtask. The explain is > as follows. > > process A occupy a resource and lock it. then A need to allocate > memory when memory is very low. at the some time, oom will come up and > return directly. because it find other process is freeing memory in > same zone. > > however, the freed memory is taken away by another process. > it will lead to A oom again and again. > > process B still wait some resource holded by A. so B will obtain the > lock until A release the resource. therefor, if A spend much time to > obtain memory, B will hungtask. OK, I see what you are aiming for. And indeed such a starvation and resulting priority inversion is possible. It is a hard problem to solve and your patch doesn't address it either. You can spend enough time reclaiming and retrying without ever getting to the oom path to trigger this hungtask warning. If you want to solve this problem properly then you would have to give tasks which are looping in the page allocator access to some portion of memory reserves. This is quite tricky to do right, though. Retry counters with the fail path have been proposed in the past and not accepted. -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-12 17:44 ` Michal Hocko @ 2016-09-13 13:13 ` zhong jiang 2016-09-13 13:28 ` Michal Hocko 0 siblings, 1 reply; 26+ messages in thread From: zhong jiang @ 2016-09-13 13:13 UTC (permalink / raw) To: Michal Hocko; +Cc: akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo On 2016/9/13 1:44, Michal Hocko wrote: > On Mon 12-09-16 21:42:28, zhong jiang wrote: >> On 2016/9/12 19:13, Michal Hocko wrote: >>> On Mon 12-09-16 17:51:06, zhong jiang wrote: >>> [...] >>>> hi, Michal >>>> oom reaper indeed can accelerate the recovery of memory, but the patch >>>> solve the extreme scenario, I hit it by runing trinity. I think the >>>> scenario can happen whether oom reaper or not. >>> could you be more specific about the case when the oom reaper and the >>> current oom code led to the oom deadlock? >> It is not the oom deadlock. It will lead to hungtask. The explain is >> as follows. >> >> process A occupy a resource and lock it. then A need to allocate >> memory when memory is very low. at the some time, oom will come up and >> return directly. because it find other process is freeing memory in >> same zone. >> >> however, the freed memory is taken away by another process. >> it will lead to A oom again and again. >> >> process B still wait some resource holded by A. so B will obtain the >> lock until A release the resource. therefor, if A spend much time to >> obtain memory, B will hungtask. > OK, I see what you are aiming for. And indeed such a starvation and > resulting priority inversion is possible. It is a hard problem to solve > and your patch doesn't address it either. You can spend enough time > reclaiming and retrying without ever getting to the oom path to trigger > this hungtask warning. Yes. > If you want to solve this problem properly then you would have to give > tasks which are looping in the page allocator access to some portion of > memory reserves. This is quite tricky to do right, though. To use some portion of memory reserves is almost no effect in a so starvation scenario. I think the hungtask still will occur. it can not solve the problem primarily. > Retry counters with the fail path have been proposed in the past and not > accepted. The above patch have been tested by runing the trinity. The question is fixed. Is there any reasonable reason oppose to the patch ? or it will bring in any side-effect. Thanks zhongjiang -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-13 13:13 ` zhong jiang @ 2016-09-13 13:28 ` Michal Hocko 2016-09-13 14:01 ` zhong jiang 2016-09-14 7:13 ` zhong jiang 0 siblings, 2 replies; 26+ messages in thread From: Michal Hocko @ 2016-09-13 13:28 UTC (permalink / raw) To: zhong jiang; +Cc: akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo On Tue 13-09-16 21:13:21, zhong jiang wrote: > On 2016/9/13 1:44, Michal Hocko wrote: [...] > > If you want to solve this problem properly then you would have to give > > tasks which are looping in the page allocator access to some portion of > > memory reserves. This is quite tricky to do right, though. > > To use some portion of memory reserves is almost no effect in a so > starvation scenario. I think the hungtask still will occur. it can > not solve the problem primarily. Granting an access to memory reserves is of course no full solution but it raises chances for a forward progress. Other solutions would have to guarantee that the memory reclaimed on behalf of the requester will be given to the requester. Not an easy task > > Retry counters with the fail path have been proposed in the past and not > > accepted. > > The above patch have been tested by runing the trinity. The question > is fixed. Is there any reasonable reason oppose to the patch ? or it > will bring in any side-effect. Sure there is. Low order allocations have been traditionally non failing and changing that behavior is a major obstacle because it opens up a door to many bugs. I've tried to do something similar in the past and there was a strong resistance against it. Believe me been there done that... -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-13 13:28 ` Michal Hocko @ 2016-09-13 14:01 ` zhong jiang 2016-09-14 7:13 ` zhong jiang 1 sibling, 0 replies; 26+ messages in thread From: zhong jiang @ 2016-09-13 14:01 UTC (permalink / raw) To: Michal Hocko; +Cc: akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo On 2016/9/13 21:28, Michal Hocko wrote: > On Tue 13-09-16 21:13:21, zhong jiang wrote: >> On 2016/9/13 1:44, Michal Hocko wrote: > [...] >>> If you want to solve this problem properly then you would have to give >>> tasks which are looping in the page allocator access to some portion of >>> memory reserves. This is quite tricky to do right, though. >> To use some portion of memory reserves is almost no effect in a so >> starvation scenario. I think the hungtask still will occur. it can >> not solve the problem primarily. > Granting an access to memory reserves is of course no full solution but > it raises chances for a forward progress. Other solutions would have to > guarantee that the memory reclaimed on behalf of the requester will be > given to the requester. Not an easy task > >>> Retry counters with the fail path have been proposed in the past and not >>> accepted. >> The above patch have been tested by runing the trinity. The question >> is fixed. Is there any reasonable reason oppose to the patch ? or it >> will bring in any side-effect. > Sure there is. Low order allocations have been traditionally non failing > and changing that behavior is a major obstacle because it opens up a > door to many bugs. I've tried to do something similar in the past and > there was a strong resistance against it. Believe me been there done > that... > That sounds resonable. but So starvation scenario should unavoidable failed. In any case you mean we need allow to allocate the low order. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-13 13:28 ` Michal Hocko 2016-09-13 14:01 ` zhong jiang @ 2016-09-14 7:13 ` zhong jiang 2016-09-14 8:42 ` Michal Hocko 1 sibling, 1 reply; 26+ messages in thread From: zhong jiang @ 2016-09-14 7:13 UTC (permalink / raw) To: Michal Hocko; +Cc: akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo On 2016/9/13 21:28, Michal Hocko wrote: > On Tue 13-09-16 21:13:21, zhong jiang wrote: >> On 2016/9/13 1:44, Michal Hocko wrote: > [...] >>> If you want to solve this problem properly then you would have to give >>> tasks which are looping in the page allocator access to some portion of >>> memory reserves. This is quite tricky to do right, though. >> To use some portion of memory reserves is almost no effect in a so >> starvation scenario. I think the hungtask still will occur. it can >> not solve the problem primarily. > Granting an access to memory reserves is of course no full solution but > it raises chances for a forward progress. Other solutions would have to > guarantee that the memory reclaimed on behalf of the requester will be > given to the requester. Not an easy task > >>> Retry counters with the fail path have been proposed in the past and not >>> accepted. >> The above patch have been tested by runing the trinity. The question >> is fixed. Is there any reasonable reason oppose to the patch ? or it >> will bring in any side-effect. > Sure there is. Low order allocations have been traditionally non failing > and changing that behavior is a major obstacle because it opens up a > door to many bugs. I've tried to do something similar in the past and > there was a strong resistance against it. Believe me been there done > that... > hi, Michal Recently, I hit the same issue when run a OOM case of the LTP and ksm enable. [ 601.937145] Call trace: [ 601.939600] [<ffffffc000086a88>] __switch_to+0x74/0x8c [ 601.944760] [<ffffffc000a1bae0>] __schedule+0x23c/0x7bc [ 601.950007] [<ffffffc000a1c09c>] schedule+0x3c/0x94 [ 601.954907] [<ffffffc000a1eb84>] rwsem_down_write_failed+0x214/0x350 [ 601.961289] [<ffffffc000a1e32c>] down_write+0x64/0x80 [ 601.966363] [<ffffffc00021f794>] __ksm_exit+0x90/0x19c [ 601.971523] [<ffffffc0000be650>] mmput+0x118/0x11c [ 601.976335] [<ffffffc0000c3ec4>] do_exit+0x2dc/0xa74 [ 601.981321] [<ffffffc0000c46f8>] do_group_exit+0x4c/0xe4 [ 601.986656] [<ffffffc0000d0f34>] get_signal+0x444/0x5e0 [ 601.991904] [<ffffffc000089fcc>] do_signal+0x1d8/0x450 [ 601.997065] [<ffffffc00008a35c>] do_notify_resume+0x70/0x78 The root case is that ksmd hold the read lock. and the lock is not released. scan_get_next_rmap_item down_read get_next_rmap_item alloc_rmap_item #ksmd will loop permanently. How do you see this kind of situation ? or let the issue alone. Thanks zhongjiang -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-14 7:13 ` zhong jiang @ 2016-09-14 8:42 ` Michal Hocko 2016-09-14 8:50 ` zhong jiang 2016-09-14 8:52 ` Michal Hocko 0 siblings, 2 replies; 26+ messages in thread From: Michal Hocko @ 2016-09-14 8:42 UTC (permalink / raw) To: zhong jiang; +Cc: akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo [Let's CC Hugh] On Wed 14-09-16 15:13:50, zhong jiang wrote: [...] > hi, Michal > > Recently, I hit the same issue when run a OOM case of the LTP and ksm enable. > > [ 601.937145] Call trace: > [ 601.939600] [<ffffffc000086a88>] __switch_to+0x74/0x8c > [ 601.944760] [<ffffffc000a1bae0>] __schedule+0x23c/0x7bc > [ 601.950007] [<ffffffc000a1c09c>] schedule+0x3c/0x94 > [ 601.954907] [<ffffffc000a1eb84>] rwsem_down_write_failed+0x214/0x350 > [ 601.961289] [<ffffffc000a1e32c>] down_write+0x64/0x80 > [ 601.966363] [<ffffffc00021f794>] __ksm_exit+0x90/0x19c > [ 601.971523] [<ffffffc0000be650>] mmput+0x118/0x11c > [ 601.976335] [<ffffffc0000c3ec4>] do_exit+0x2dc/0xa74 > [ 601.981321] [<ffffffc0000c46f8>] do_group_exit+0x4c/0xe4 > [ 601.986656] [<ffffffc0000d0f34>] get_signal+0x444/0x5e0 > [ 601.991904] [<ffffffc000089fcc>] do_signal+0x1d8/0x450 > [ 601.997065] [<ffffffc00008a35c>] do_notify_resume+0x70/0x78 So this is a hung task triggering because the exiting task cannot get the mmap sem for write because the ksmd holds it for read while allocating memory which just takes ages to complete, right? > > The root case is that ksmd hold the read lock. and the lock is not released. > scan_get_next_rmap_item > down_read > get_next_rmap_item > alloc_rmap_item #ksmd will loop permanently. > > How do you see this kind of situation ? or let the issue alone. I am not familiar with the ksmd code so it is hard for me to judge but one thing to do would be __GFP_NORETRY which would force a bail out from the allocation rather than looping for ever. A quick look tells me that the allocation failure here is quite easy to handle. There might be others... -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-14 8:42 ` Michal Hocko @ 2016-09-14 8:50 ` zhong jiang 2016-09-14 9:05 ` Michal Hocko 2016-09-14 8:52 ` Michal Hocko 1 sibling, 1 reply; 26+ messages in thread From: zhong jiang @ 2016-09-14 8:50 UTC (permalink / raw) To: Michal Hocko; +Cc: akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo On 2016/9/14 16:42, Michal Hocko wrote: > [Let's CC Hugh] > > On Wed 14-09-16 15:13:50, zhong jiang wrote: > [...] >> hi, Michal >> >> Recently, I hit the same issue when run a OOM case of the LTP and ksm enable. >> >> [ 601.937145] Call trace: >> [ 601.939600] [<ffffffc000086a88>] __switch_to+0x74/0x8c >> [ 601.944760] [<ffffffc000a1bae0>] __schedule+0x23c/0x7bc >> [ 601.950007] [<ffffffc000a1c09c>] schedule+0x3c/0x94 >> [ 601.954907] [<ffffffc000a1eb84>] rwsem_down_write_failed+0x214/0x350 >> [ 601.961289] [<ffffffc000a1e32c>] down_write+0x64/0x80 >> [ 601.966363] [<ffffffc00021f794>] __ksm_exit+0x90/0x19c >> [ 601.971523] [<ffffffc0000be650>] mmput+0x118/0x11c >> [ 601.976335] [<ffffffc0000c3ec4>] do_exit+0x2dc/0xa74 >> [ 601.981321] [<ffffffc0000c46f8>] do_group_exit+0x4c/0xe4 >> [ 601.986656] [<ffffffc0000d0f34>] get_signal+0x444/0x5e0 >> [ 601.991904] [<ffffffc000089fcc>] do_signal+0x1d8/0x450 >> [ 601.997065] [<ffffffc00008a35c>] do_notify_resume+0x70/0x78 > So this is a hung task triggering because the exiting task cannot get > the mmap sem for write because the ksmd holds it for read while > allocating memory which just takes ages to complete, right? Yes >> The root case is that ksmd hold the read lock. and the lock is not released. >> scan_get_next_rmap_item >> down_read >> get_next_rmap_item >> alloc_rmap_item #ksmd will loop permanently. >> >> How do you see this kind of situation ? or let the issue alone. > I am not familiar with the ksmd code so it is hard for me to judge but > one thing to do would be __GFP_NORETRY which would force a bail out from > the allocation rather than looping for ever. A quick look tells me that > the allocation failure here is quite easy to handle. There might be > others... > by adding my patch , The question is fixed. They are same issue. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-14 8:50 ` zhong jiang @ 2016-09-14 9:05 ` Michal Hocko 0 siblings, 0 replies; 26+ messages in thread From: Michal Hocko @ 2016-09-14 9:05 UTC (permalink / raw) To: zhong jiang; +Cc: akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo On Wed 14-09-16 16:50:48, zhong jiang wrote: > On 2016/9/14 16:42, Michal Hocko wrote: > > [Let's CC Hugh] > > > > On Wed 14-09-16 15:13:50, zhong jiang wrote: > > [...] > >> hi, Michal > >> > >> Recently, I hit the same issue when run a OOM case of the LTP and ksm enable. > >> > >> [ 601.937145] Call trace: > >> [ 601.939600] [<ffffffc000086a88>] __switch_to+0x74/0x8c > >> [ 601.944760] [<ffffffc000a1bae0>] __schedule+0x23c/0x7bc > >> [ 601.950007] [<ffffffc000a1c09c>] schedule+0x3c/0x94 > >> [ 601.954907] [<ffffffc000a1eb84>] rwsem_down_write_failed+0x214/0x350 > >> [ 601.961289] [<ffffffc000a1e32c>] down_write+0x64/0x80 > >> [ 601.966363] [<ffffffc00021f794>] __ksm_exit+0x90/0x19c > >> [ 601.971523] [<ffffffc0000be650>] mmput+0x118/0x11c > >> [ 601.976335] [<ffffffc0000c3ec4>] do_exit+0x2dc/0xa74 > >> [ 601.981321] [<ffffffc0000c46f8>] do_group_exit+0x4c/0xe4 > >> [ 601.986656] [<ffffffc0000d0f34>] get_signal+0x444/0x5e0 > >> [ 601.991904] [<ffffffc000089fcc>] do_signal+0x1d8/0x450 > >> [ 601.997065] [<ffffffc00008a35c>] do_notify_resume+0x70/0x78 > > So this is a hung task triggering because the exiting task cannot get > > the mmap sem for write because the ksmd holds it for read while > > allocating memory which just takes ages to complete, right? > Yes > >> The root case is that ksmd hold the read lock. and the lock is not released. > >> scan_get_next_rmap_item > >> down_read > >> get_next_rmap_item > >> alloc_rmap_item #ksmd will loop permanently. > >> > >> How do you see this kind of situation ? or let the issue alone. > > I am not familiar with the ksmd code so it is hard for me to judge but > > one thing to do would be __GFP_NORETRY which would force a bail out from > > the allocation rather than looping for ever. A quick look tells me that > > the allocation failure here is quite easy to handle. There might be > > others... > > > by adding my patch , The question is fixed. They are same issue. No, it's not, as I've alreade mentioned before. -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-14 8:42 ` Michal Hocko 2016-09-14 8:50 ` zhong jiang @ 2016-09-14 8:52 ` Michal Hocko 2016-09-14 9:25 ` zhong jiang 2016-09-16 22:13 ` Hugh Dickins 1 sibling, 2 replies; 26+ messages in thread From: Michal Hocko @ 2016-09-14 8:52 UTC (permalink / raw) To: zhong jiang Cc: akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo, Hugh Dickins On Wed 14-09-16 10:42:19, Michal Hocko wrote: > [Let's CC Hugh] now for real... > > On Wed 14-09-16 15:13:50, zhong jiang wrote: > [...] > > hi, Michal > > > > Recently, I hit the same issue when run a OOM case of the LTP and ksm enable. > > > > [ 601.937145] Call trace: > > [ 601.939600] [<ffffffc000086a88>] __switch_to+0x74/0x8c > > [ 601.944760] [<ffffffc000a1bae0>] __schedule+0x23c/0x7bc > > [ 601.950007] [<ffffffc000a1c09c>] schedule+0x3c/0x94 > > [ 601.954907] [<ffffffc000a1eb84>] rwsem_down_write_failed+0x214/0x350 > > [ 601.961289] [<ffffffc000a1e32c>] down_write+0x64/0x80 > > [ 601.966363] [<ffffffc00021f794>] __ksm_exit+0x90/0x19c > > [ 601.971523] [<ffffffc0000be650>] mmput+0x118/0x11c > > [ 601.976335] [<ffffffc0000c3ec4>] do_exit+0x2dc/0xa74 > > [ 601.981321] [<ffffffc0000c46f8>] do_group_exit+0x4c/0xe4 > > [ 601.986656] [<ffffffc0000d0f34>] get_signal+0x444/0x5e0 > > [ 601.991904] [<ffffffc000089fcc>] do_signal+0x1d8/0x450 > > [ 601.997065] [<ffffffc00008a35c>] do_notify_resume+0x70/0x78 > > So this is a hung task triggering because the exiting task cannot get > the mmap sem for write because the ksmd holds it for read while > allocating memory which just takes ages to complete, right? > > > > > The root case is that ksmd hold the read lock. and the lock is not released. > > scan_get_next_rmap_item > > down_read > > get_next_rmap_item > > alloc_rmap_item #ksmd will loop permanently. > > > > How do you see this kind of situation ? or let the issue alone. > > I am not familiar with the ksmd code so it is hard for me to judge but > one thing to do would be __GFP_NORETRY which would force a bail out from > the allocation rather than looping for ever. A quick look tells me that > the allocation failure here is quite easy to handle. There might be > others... > > -- > Michal Hocko > SUSE Labs -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-14 8:52 ` Michal Hocko @ 2016-09-14 9:25 ` zhong jiang 2016-09-14 11:29 ` Tetsuo Handa 2016-09-16 22:13 ` Hugh Dickins 1 sibling, 1 reply; 26+ messages in thread From: zhong jiang @ 2016-09-14 9:25 UTC (permalink / raw) To: Michal Hocko Cc: akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo, Hugh Dickins On 2016/9/14 16:52, Michal Hocko wrote: > On Wed 14-09-16 10:42:19, Michal Hocko wrote: >> [Let's CC Hugh] > now for real... > >> On Wed 14-09-16 15:13:50, zhong jiang wrote: >> [...] >>> hi, Michal >>> >>> Recently, I hit the same issue when run a OOM case of the LTP and ksm enable. >>> >>> [ 601.937145] Call trace: >>> [ 601.939600] [<ffffffc000086a88>] __switch_to+0x74/0x8c >>> [ 601.944760] [<ffffffc000a1bae0>] __schedule+0x23c/0x7bc >>> [ 601.950007] [<ffffffc000a1c09c>] schedule+0x3c/0x94 >>> [ 601.954907] [<ffffffc000a1eb84>] rwsem_down_write_failed+0x214/0x350 >>> [ 601.961289] [<ffffffc000a1e32c>] down_write+0x64/0x80 >>> [ 601.966363] [<ffffffc00021f794>] __ksm_exit+0x90/0x19c >>> [ 601.971523] [<ffffffc0000be650>] mmput+0x118/0x11c >>> [ 601.976335] [<ffffffc0000c3ec4>] do_exit+0x2dc/0xa74 >>> [ 601.981321] [<ffffffc0000c46f8>] do_group_exit+0x4c/0xe4 >>> [ 601.986656] [<ffffffc0000d0f34>] get_signal+0x444/0x5e0 >>> [ 601.991904] [<ffffffc000089fcc>] do_signal+0x1d8/0x450 >>> [ 601.997065] [<ffffffc00008a35c>] do_notify_resume+0x70/0x78 >> So this is a hung task triggering because the exiting task cannot get >> the mmap sem for write because the ksmd holds it for read while >> allocating memory which just takes ages to complete, right? >> >>> The root case is that ksmd hold the read lock. and the lock is not released. >>> scan_get_next_rmap_item >>> down_read >>> get_next_rmap_item >>> alloc_rmap_item #ksmd will loop permanently. >>> >>> How do you see this kind of situation ? or let the issue alone. >> I am not familiar with the ksmd code so it is hard for me to judge but >> one thing to do would be __GFP_NORETRY which would force a bail out from >> the allocation rather than looping for ever. A quick look tells me that >> the allocation failure here is quite easy to handle. There might be >> others... >> >> -- >> Michal Hocko >> SUSE Labs Adding the __GFP_NORETRY, the issue also can fixed. Therefore, we can assure that the case of LTP will leads to the endless looping. index d45a0a1..03fb67b 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -283,7 +283,7 @@ static inline struct rmap_item *alloc_rmap_item(void) { struct rmap_item *rmap_item; - rmap_item = kmem_cache_zalloc(rmap_item_cache, GFP_KERNEL); + rmap_item = kmem_cache_zalloc(rmap_item_cache, GFP_KERNEL | __GFP_NORETRY); if (rmap_item) ksm_rmap_items++; return rmap_item; -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-14 9:25 ` zhong jiang @ 2016-09-14 11:29 ` Tetsuo Handa 2016-09-14 13:52 ` zhong jiang 0 siblings, 1 reply; 26+ messages in thread From: Tetsuo Handa @ 2016-09-14 11:29 UTC (permalink / raw) To: zhong jiang, Michal Hocko Cc: akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo, Hugh Dickins On 2016/09/14 18:25, zhong jiang wrote: > On 2016/9/14 16:52, Michal Hocko wrote: >> On Wed 14-09-16 10:42:19, Michal Hocko wrote: >>> [Let's CC Hugh] >> now for real... >> >>> On Wed 14-09-16 15:13:50, zhong jiang wrote: >>> [...] >>>> hi, Michal >>>> >>>> Recently, I hit the same issue when run a OOM case of the LTP and ksm enable. >>>> >>>> [ 601.937145] Call trace: >>>> [ 601.939600] [<ffffffc000086a88>] __switch_to+0x74/0x8c >>>> [ 601.944760] [<ffffffc000a1bae0>] __schedule+0x23c/0x7bc >>>> [ 601.950007] [<ffffffc000a1c09c>] schedule+0x3c/0x94 >>>> [ 601.954907] [<ffffffc000a1eb84>] rwsem_down_write_failed+0x214/0x350 >>>> [ 601.961289] [<ffffffc000a1e32c>] down_write+0x64/0x80 >>>> [ 601.966363] [<ffffffc00021f794>] __ksm_exit+0x90/0x19c >>>> [ 601.971523] [<ffffffc0000be650>] mmput+0x118/0x11c >>>> [ 601.976335] [<ffffffc0000c3ec4>] do_exit+0x2dc/0xa74 >>>> [ 601.981321] [<ffffffc0000c46f8>] do_group_exit+0x4c/0xe4 >>>> [ 601.986656] [<ffffffc0000d0f34>] get_signal+0x444/0x5e0 >>>> [ 601.991904] [<ffffffc000089fcc>] do_signal+0x1d8/0x450 >>>> [ 601.997065] [<ffffffc00008a35c>] do_notify_resume+0x70/0x78 Please be sure to include exact kernel version (e.g. "uname -r", "cat /proc/version") when reporting. You are reporting a bug in 4.1-stable kernel, which was prone to OOM livelock because the OOM reaper is not available. ( http://lkml.kernel.org/r/57D8012F.7080508@huawei.com ) I think we no longer can reproduce this bug using 4.8-rc6 (or linux-next), but it will be a nice thing to backport __GFP_NORETRY patch to stable kernels which do not have the OOM reaper. > Adding the __GFP_NORETRY, the issue also can fixed. > Therefore, we can assure that the case of LTP will leads to the endless looping. > > index d45a0a1..03fb67b 100644 > --- a/mm/ksm.c > +++ b/mm/ksm.c > @@ -283,7 +283,7 @@ static inline struct rmap_item *alloc_rmap_item(void) > { > struct rmap_item *rmap_item; > > - rmap_item = kmem_cache_zalloc(rmap_item_cache, GFP_KERNEL); > + rmap_item = kmem_cache_zalloc(rmap_item_cache, GFP_KERNEL | __GFP_NORETRY); > if (rmap_item) > ksm_rmap_items++; > return rmap_item; -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-14 11:29 ` Tetsuo Handa @ 2016-09-14 13:52 ` zhong jiang 2016-09-18 6:00 ` Tetsuo Handa 0 siblings, 1 reply; 26+ messages in thread From: zhong jiang @ 2016-09-14 13:52 UTC (permalink / raw) To: Tetsuo Handa Cc: Michal Hocko, akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo, Hugh Dickins On 2016/9/14 19:29, Tetsuo Handa wrote: > On 2016/09/14 18:25, zhong jiang wrote: >> On 2016/9/14 16:52, Michal Hocko wrote: >>> On Wed 14-09-16 10:42:19, Michal Hocko wrote: >>>> [Let's CC Hugh] >>> now for real... >>> >>>> On Wed 14-09-16 15:13:50, zhong jiang wrote: >>>> [...] >>>>> hi, Michal >>>>> >>>>> Recently, I hit the same issue when run a OOM case of the LTP and ksm enable. >>>>> >>>>> [ 601.937145] Call trace: >>>>> [ 601.939600] [<ffffffc000086a88>] __switch_to+0x74/0x8c >>>>> [ 601.944760] [<ffffffc000a1bae0>] __schedule+0x23c/0x7bc >>>>> [ 601.950007] [<ffffffc000a1c09c>] schedule+0x3c/0x94 >>>>> [ 601.954907] [<ffffffc000a1eb84>] rwsem_down_write_failed+0x214/0x350 >>>>> [ 601.961289] [<ffffffc000a1e32c>] down_write+0x64/0x80 >>>>> [ 601.966363] [<ffffffc00021f794>] __ksm_exit+0x90/0x19c >>>>> [ 601.971523] [<ffffffc0000be650>] mmput+0x118/0x11c >>>>> [ 601.976335] [<ffffffc0000c3ec4>] do_exit+0x2dc/0xa74 >>>>> [ 601.981321] [<ffffffc0000c46f8>] do_group_exit+0x4c/0xe4 >>>>> [ 601.986656] [<ffffffc0000d0f34>] get_signal+0x444/0x5e0 >>>>> [ 601.991904] [<ffffffc000089fcc>] do_signal+0x1d8/0x450 >>>>> [ 601.997065] [<ffffffc00008a35c>] do_notify_resume+0x70/0x78 > Please be sure to include exact kernel version (e.g. "uname -r", > "cat /proc/version") when reporting. > > You are reporting a bug in 4.1-stable kernel, which was prone to > OOM livelock because the OOM reaper is not available. > ( http://lkml.kernel.org/r/57D8012F.7080508@huawei.com ) > > I think we no longer can reproduce this bug using 4.8-rc6 (or linux-next), > but it will be a nice thing to backport __GFP_NORETRY patch to stable > kernels which do not have the OOM reaper. No, OOM reaper can not solve the issue completely , As had disscussed with Michal. The conclusion is that we need come up with a better method to fix it. Thanks zhongjiang >> Adding the __GFP_NORETRY, the issue also can fixed. >> Therefore, we can assure that the case of LTP will leads to the endless looping. >> >> index d45a0a1..03fb67b 100644 >> --- a/mm/ksm.c >> +++ b/mm/ksm.c >> @@ -283,7 +283,7 @@ static inline struct rmap_item *alloc_rmap_item(void) >> { >> struct rmap_item *rmap_item; >> >> - rmap_item = kmem_cache_zalloc(rmap_item_cache, GFP_KERNEL); >> + rmap_item = kmem_cache_zalloc(rmap_item_cache, GFP_KERNEL | __GFP_NORETRY); >> if (rmap_item) >> ksm_rmap_items++; >> return rmap_item; > > . > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-14 13:52 ` zhong jiang @ 2016-09-18 6:00 ` Tetsuo Handa 2016-09-18 6:13 ` Tetsuo Handa 2016-09-19 7:15 ` zhong jiang 0 siblings, 2 replies; 26+ messages in thread From: Tetsuo Handa @ 2016-09-18 6:00 UTC (permalink / raw) To: zhongjiang Cc: mhocko, akpm, vbabka, rientjes, linux-mm, qiuxishi, guohanjun, hughd zhong jiang wrote: > On 2016/9/14 19:29, Tetsuo Handa wrote: > > On 2016/09/14 18:25, zhong jiang wrote: > >> On 2016/9/14 16:52, Michal Hocko wrote: > >>> On Wed 14-09-16 10:42:19, Michal Hocko wrote: > >>>> [Let's CC Hugh] > >>> now for real... > >>> > >>>> On Wed 14-09-16 15:13:50, zhong jiang wrote: > >>>> [...] > >>>>> hi, Michal > >>>>> > >>>>> Recently, I hit the same issue when run a OOM case of the LTP and ksm enable. > >>>>> > >>>>> [ 601.937145] Call trace: > >>>>> [ 601.939600] [<ffffffc000086a88>] __switch_to+0x74/0x8c > >>>>> [ 601.944760] [<ffffffc000a1bae0>] __schedule+0x23c/0x7bc > >>>>> [ 601.950007] [<ffffffc000a1c09c>] schedule+0x3c/0x94 > >>>>> [ 601.954907] [<ffffffc000a1eb84>] rwsem_down_write_failed+0x214/0x350 > >>>>> [ 601.961289] [<ffffffc000a1e32c>] down_write+0x64/0x80 > >>>>> [ 601.966363] [<ffffffc00021f794>] __ksm_exit+0x90/0x19c > >>>>> [ 601.971523] [<ffffffc0000be650>] mmput+0x118/0x11c > >>>>> [ 601.976335] [<ffffffc0000c3ec4>] do_exit+0x2dc/0xa74 > >>>>> [ 601.981321] [<ffffffc0000c46f8>] do_group_exit+0x4c/0xe4 > >>>>> [ 601.986656] [<ffffffc0000d0f34>] get_signal+0x444/0x5e0 > >>>>> [ 601.991904] [<ffffffc000089fcc>] do_signal+0x1d8/0x450 > >>>>> [ 601.997065] [<ffffffc00008a35c>] do_notify_resume+0x70/0x78 > > Please be sure to include exact kernel version (e.g. "uname -r", > > "cat /proc/version") when reporting. > > > > You are reporting a bug in 4.1-stable kernel, which was prone to > > OOM livelock because the OOM reaper is not available. > > ( http://lkml.kernel.org/r/57D8012F.7080508@huawei.com ) > > > > I think we no longer can reproduce this bug using 4.8-rc6 (or linux-next), > > but it will be a nice thing to backport __GFP_NORETRY patch to stable > > kernels which do not have the OOM reaper. > No, OOM reaper can not solve the issue completely , As had disscussed with Michal. > The conclusion is that we need come up with a better method to fix it. > > Thanks > zhongjiang I still think we no longer can reproduce this bug using 4.8-rc6 (or linux-next). As of 4.1-stable, this bug caused OOM livelock situation because TIF_MEMDIE was cleared only after returning from mmput() from exit_mm() from do_exit(). Since there is a TIF_MEMDIE thread waiting at mmput() exists, the OOM killer does not select next OOM victim because oom_scan_process_thread() returns OOM_SCAN_ABORT, although ksmd is waking up the OOM killer via a __GFP_FS allocation request. As of 4.8-rc6, the OOM reaper cannot take mmap_sem for read at __oom_reap_task() because of TIF_MEMDIE thread waiting at ksm_exit() from __mmput() from mmput() from exit_mm() from do_exit(). Thus, __oom_reap_task() returns false and oom_reap_task() will emit "oom_reaper: unable to reap pid:%d (%s)\n" message. Then, oom_reap_task() clears TIF_MEMDIE from that thread, which in turn makes oom_scan_process_thread() not to return OOM_SCAN_ABORT because atomic_read(&task->signal->oom_victims) becomes 0 due to exit_oom_victim() by the OOM reaper. Then, the OOM killer selects next OOM victim because ksmd is waking up the OOM killer via a __GFP_FS allocation request. Thus, this bug will be completely solved (at the cost of selecting next OOM victim) as of 4.8-rc6. > >> Adding the __GFP_NORETRY, the issue also can fixed. > >> Therefore, we can assure that the case of LTP will leads to the endless looping. > >> > >> index d45a0a1..03fb67b 100644 > >> --- a/mm/ksm.c > >> +++ b/mm/ksm.c > >> @@ -283,7 +283,7 @@ static inline struct rmap_item *alloc_rmap_item(void) > >> { > >> struct rmap_item *rmap_item; > >> > >> - rmap_item = kmem_cache_zalloc(rmap_item_cache, GFP_KERNEL); > >> + rmap_item = kmem_cache_zalloc(rmap_item_cache, GFP_KERNEL | __GFP_NORETRY); > >> if (rmap_item) > >> ksm_rmap_items++; > >> return rmap_item; > > Your patch to add __GFP_NORETRY and __GFP_NOWARN is OK. But please explicitly state that you hit this bug in 4.1-stable. Also, your trace is not only a hung task but also an OOM livelock; the kernel as of 4.1-stable is silent when OOM livelock situation occurred. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-18 6:00 ` Tetsuo Handa @ 2016-09-18 6:13 ` Tetsuo Handa 2016-09-19 4:44 ` zhong jiang 2016-09-19 7:15 ` zhong jiang 1 sibling, 1 reply; 26+ messages in thread From: Tetsuo Handa @ 2016-09-18 6:13 UTC (permalink / raw) To: zhongjiang Cc: mhocko, akpm, vbabka, rientjes, linux-mm, qiuxishi, guohanjun, hughd Tetsuo Handa wrote: > As of 4.8-rc6, the OOM reaper cannot take mmap_sem for read at __oom_reap_task() > because of TIF_MEMDIE thread waiting at ksm_exit() from __mmput() from mmput() > from exit_mm() from do_exit(). Thus, __oom_reap_task() returns false and > oom_reap_task() will emit "oom_reaper: unable to reap pid:%d (%s)\n" message. > Then, oom_reap_task() clears TIF_MEMDIE from that thread, which in turn > makes oom_scan_process_thread() not to return OOM_SCAN_ABORT because > atomic_read(&task->signal->oom_victims) becomes 0 due to exit_oom_victim() > by the OOM reaper. Then, the OOM killer selects next OOM victim because > ksmd is waking up the OOM killer via a __GFP_FS allocation request. Oops. As of 4.8-rc6, __oom_reap_task() returns true because of find_lock_task_mm() returning NULL. Thus, oom_reap_task() clears TIF_MEMDIE without emitting "oom_reaper: unable to reap pid:%d (%s)\n" message. > > Thus, this bug will be completely solved (at the cost of selecting next > OOM victim) as of 4.8-rc6. The conclusion is same. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-18 6:13 ` Tetsuo Handa @ 2016-09-19 4:44 ` zhong jiang 0 siblings, 0 replies; 26+ messages in thread From: zhong jiang @ 2016-09-19 4:44 UTC (permalink / raw) To: Tetsuo Handa Cc: mhocko, akpm, vbabka, rientjes, linux-mm, qiuxishi, guohanjun, hughd On 2016/9/18 14:13, Tetsuo Handa wrote: > Tetsuo Handa wrote: >> As of 4.8-rc6, the OOM reaper cannot take mmap_sem for read at __oom_reap_task() >> because of TIF_MEMDIE thread waiting at ksm_exit() from __mmput() from mmput() >> from exit_mm() from do_exit(). Thus, __oom_reap_task() returns false and >> oom_reap_task() will emit "oom_reaper: unable to reap pid:%d (%s)\n" message. >> Then, oom_reap_task() clears TIF_MEMDIE from that thread, which in turn >> makes oom_scan_process_thread() not to return OOM_SCAN_ABORT because >> atomic_read(&task->signal->oom_victims) becomes 0 due to exit_oom_victim() >> by the OOM reaper. Then, the OOM killer selects next OOM victim because >> ksmd is waking up the OOM killer via a __GFP_FS allocation request. hi, Tetsuo OOM reaper indeed relieve the issue, as had discussed with Michal, but it is not completely solved. and OOM livelock had been solved by backport the patch from Michal. The key is that ksmd enter into the OOM and bail out quickly. because other process implement a OOM in the same zone. therefore, ksmd can not obtain the memory. Thanks zhongjiang > Oops. As of 4.8-rc6, __oom_reap_task() returns true because of > find_lock_task_mm() returning NULL. Thus, oom_reap_task() clears TIF_MEMDIE > without emitting "oom_reaper: unable to reap pid:%d (%s)\n" message. > >> Thus, this bug will be completely solved (at the cost of selecting next >> OOM victim) as of 4.8-rc6. > The conclusion is same. > > . > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-18 6:00 ` Tetsuo Handa 2016-09-18 6:13 ` Tetsuo Handa @ 2016-09-19 7:15 ` zhong jiang 1 sibling, 0 replies; 26+ messages in thread From: zhong jiang @ 2016-09-19 7:15 UTC (permalink / raw) To: Tetsuo Handa Cc: mhocko, akpm, vbabka, rientjes, linux-mm, qiuxishi, guohanjun, hughd On 2016/9/18 14:00, Tetsuo Handa wrote: > zhong jiang wrote: >> On 2016/9/14 19:29, Tetsuo Handa wrote: >>> On 2016/09/14 18:25, zhong jiang wrote: >>>> On 2016/9/14 16:52, Michal Hocko wrote: >>>>> On Wed 14-09-16 10:42:19, Michal Hocko wrote: >>>>>> [Let's CC Hugh] >>>>> now for real... >>>>> >>>>>> On Wed 14-09-16 15:13:50, zhong jiang wrote: >>>>>> [...] >>>>>>> hi, Michal >>>>>>> >>>>>>> Recently, I hit the same issue when run a OOM case of the LTP and ksm enable. >>>>>>> >>>>>>> [ 601.937145] Call trace: >>>>>>> [ 601.939600] [<ffffffc000086a88>] __switch_to+0x74/0x8c >>>>>>> [ 601.944760] [<ffffffc000a1bae0>] __schedule+0x23c/0x7bc >>>>>>> [ 601.950007] [<ffffffc000a1c09c>] schedule+0x3c/0x94 >>>>>>> [ 601.954907] [<ffffffc000a1eb84>] rwsem_down_write_failed+0x214/0x350 >>>>>>> [ 601.961289] [<ffffffc000a1e32c>] down_write+0x64/0x80 >>>>>>> [ 601.966363] [<ffffffc00021f794>] __ksm_exit+0x90/0x19c >>>>>>> [ 601.971523] [<ffffffc0000be650>] mmput+0x118/0x11c >>>>>>> [ 601.976335] [<ffffffc0000c3ec4>] do_exit+0x2dc/0xa74 >>>>>>> [ 601.981321] [<ffffffc0000c46f8>] do_group_exit+0x4c/0xe4 >>>>>>> [ 601.986656] [<ffffffc0000d0f34>] get_signal+0x444/0x5e0 >>>>>>> [ 601.991904] [<ffffffc000089fcc>] do_signal+0x1d8/0x450 >>>>>>> [ 601.997065] [<ffffffc00008a35c>] do_notify_resume+0x70/0x78 >>> Please be sure to include exact kernel version (e.g. "uname -r", >>> "cat /proc/version") when reporting. >>> >>> You are reporting a bug in 4.1-stable kernel, which was prone to >>> OOM livelock because the OOM reaper is not available. >>> ( http://lkml.kernel.org/r/57D8012F.7080508@huawei.com ) >>> >>> I think we no longer can reproduce this bug using 4.8-rc6 (or linux-next), >>> but it will be a nice thing to backport __GFP_NORETRY patch to stable >>> kernels which do not have the OOM reaper. >> No, OOM reaper can not solve the issue completely , As had disscussed with Michal. >> The conclusion is that we need come up with a better method to fix it. >> >> Thanks >> zhongjiang > I still think we no longer can reproduce this bug using 4.8-rc6 (or linux-next). > > As of 4.1-stable, this bug caused OOM livelock situation because TIF_MEMDIE was > cleared only after returning from mmput() from exit_mm() from do_exit(). Since > there is a TIF_MEMDIE thread waiting at mmput() exists, the OOM killer does not > select next OOM victim because oom_scan_process_thread() returns OOM_SCAN_ABORT, > although ksmd is waking up the OOM killer via a __GFP_FS allocation request. > > As of 4.8-rc6, the OOM reaper cannot take mmap_sem for read at __oom_reap_task() > because of TIF_MEMDIE thread waiting at ksm_exit() from __mmput() from mmput() > from exit_mm() from do_exit(). Thus, __oom_reap_task() returns false and > oom_reap_task() will emit "oom_reaper: unable to reap pid:%d (%s)\n" message. > Then, oom_reap_task() clears TIF_MEMDIE from that thread, which in turn > makes oom_scan_process_thread() not to return OOM_SCAN_ABORT because > atomic_read(&task->signal->oom_victims) becomes 0 due to exit_oom_victim() > by the OOM reaper. Then, the OOM killer selects next OOM victim because > ksmd is waking up the OOM killer via a __GFP_FS allocation request. The issue will come up just in arm64, x86 is not. > Thus, this bug will be completely solved (at the cost of selecting next > OOM victim) as of 4.8-rc6. > >>>> Adding the __GFP_NORETRY, the issue also can fixed. >>>> Therefore, we can assure that the case of LTP will leads to the endless looping. >>>> >>>> index d45a0a1..03fb67b 100644 >>>> --- a/mm/ksm.c >>>> +++ b/mm/ksm.c >>>> @@ -283,7 +283,7 @@ static inline struct rmap_item *alloc_rmap_item(void) >>>> { >>>> struct rmap_item *rmap_item; >>>> >>>> - rmap_item = kmem_cache_zalloc(rmap_item_cache, GFP_KERNEL); >>>> + rmap_item = kmem_cache_zalloc(rmap_item_cache, GFP_KERNEL | __GFP_NORETRY); >>>> if (rmap_item) >>>> ksm_rmap_items++; >>>> return rmap_item; > Your patch to add __GFP_NORETRY and __GFP_NOWARN is OK. But please explicitly state > that you hit this bug in 4.1-stable. Also, your trace is not only a hung task but also > an OOM livelock; the kernel as of 4.1-stable is silent when OOM livelock situation > occurred. > > . > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-14 8:52 ` Michal Hocko 2016-09-14 9:25 ` zhong jiang @ 2016-09-16 22:13 ` Hugh Dickins 2016-09-17 15:56 ` Michal Hocko 2016-09-18 4:04 ` zhong jiang 1 sibling, 2 replies; 26+ messages in thread From: Hugh Dickins @ 2016-09-16 22:13 UTC (permalink / raw) To: Michal Hocko Cc: zhong jiang, akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo, Hugh Dickins On Wed, 14 Sep 2016, Michal Hocko wrote: > On Wed 14-09-16 10:42:19, Michal Hocko wrote: > > [Let's CC Hugh] > > now for real... > > > > > On Wed 14-09-16 15:13:50, zhong jiang wrote: > > [...] > > > hi, Michal > > > > > > Recently, I hit the same issue when run a OOM case of the LTP and ksm enable. > > > > > > [ 601.937145] Call trace: > > > [ 601.939600] [<ffffffc000086a88>] __switch_to+0x74/0x8c > > > [ 601.944760] [<ffffffc000a1bae0>] __schedule+0x23c/0x7bc > > > [ 601.950007] [<ffffffc000a1c09c>] schedule+0x3c/0x94 > > > [ 601.954907] [<ffffffc000a1eb84>] rwsem_down_write_failed+0x214/0x350 > > > [ 601.961289] [<ffffffc000a1e32c>] down_write+0x64/0x80 > > > [ 601.966363] [<ffffffc00021f794>] __ksm_exit+0x90/0x19c > > > [ 601.971523] [<ffffffc0000be650>] mmput+0x118/0x11c > > > [ 601.976335] [<ffffffc0000c3ec4>] do_exit+0x2dc/0xa74 > > > [ 601.981321] [<ffffffc0000c46f8>] do_group_exit+0x4c/0xe4 > > > [ 601.986656] [<ffffffc0000d0f34>] get_signal+0x444/0x5e0 > > > [ 601.991904] [<ffffffc000089fcc>] do_signal+0x1d8/0x450 > > > [ 601.997065] [<ffffffc00008a35c>] do_notify_resume+0x70/0x78 > > > > So this is a hung task triggering because the exiting task cannot get > > the mmap sem for write because the ksmd holds it for read while > > allocating memory which just takes ages to complete, right? > > > > > > > > The root case is that ksmd hold the read lock. and the lock is not released. > > > scan_get_next_rmap_item > > > down_read > > > get_next_rmap_item > > > alloc_rmap_item #ksmd will loop permanently. > > > > > > How do you see this kind of situation ? or let the issue alone. > > > > I am not familiar with the ksmd code so it is hard for me to judge but > > one thing to do would be __GFP_NORETRY which would force a bail out from > > the allocation rather than looping for ever. A quick look tells me that > > the allocation failure here is quite easy to handle. There might be > > others... Yes, very good suggestion in this case: the ksmd code does exactly the right thing when that allocation fails, but was too stupid to use an allocation mode which might fail - and it can allocate rather a lot of slots along that path, so it will be good to let it break out there. Thank you, Zhongjiang, please send akpm a fully signed-off patch, tagged for stable, with your explanation above (which was a lot more helpful to me than what you wrote in your other mail of Sept 13th). But please make it GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN (and break that line before 80 cols): the allocation will sometimes fail, and we're not at all interested in hearing about that. Michal, how would you feel about this or a separate patch adding __GFP_HIGH to the allocation in ksm's alloc_stable_node()? That allocation could cause the same problem, but it is much less common (so less important to do anything about it), and differs from the rmap_item case in that if it succeeds, it will usually free a page; whereas if it fails, the fallback (two break_cow()s) may want to allocate a couple of pages. So __GFP_HIGH makes more sense for it than __GFP_NORETRY: but perhaps we prefer not to add __GFP_HIGHs? Hugh -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-16 22:13 ` Hugh Dickins @ 2016-09-17 15:56 ` Michal Hocko 2016-09-18 4:04 ` zhong jiang 1 sibling, 0 replies; 26+ messages in thread From: Michal Hocko @ 2016-09-17 15:56 UTC (permalink / raw) To: Hugh Dickins Cc: zhong jiang, akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo On Fri 16-09-16 15:13:56, Hugh Dickins wrote: > On Wed, 14 Sep 2016, Michal Hocko wrote: > > On Wed 14-09-16 10:42:19, Michal Hocko wrote: > > > [Let's CC Hugh] > > > > now for real... > > > > > > > > On Wed 14-09-16 15:13:50, zhong jiang wrote: > > > [...] > > > > hi, Michal > > > > > > > > Recently, I hit the same issue when run a OOM case of the LTP and ksm enable. > > > > > > > > [ 601.937145] Call trace: > > > > [ 601.939600] [<ffffffc000086a88>] __switch_to+0x74/0x8c > > > > [ 601.944760] [<ffffffc000a1bae0>] __schedule+0x23c/0x7bc > > > > [ 601.950007] [<ffffffc000a1c09c>] schedule+0x3c/0x94 > > > > [ 601.954907] [<ffffffc000a1eb84>] rwsem_down_write_failed+0x214/0x350 > > > > [ 601.961289] [<ffffffc000a1e32c>] down_write+0x64/0x80 > > > > [ 601.966363] [<ffffffc00021f794>] __ksm_exit+0x90/0x19c > > > > [ 601.971523] [<ffffffc0000be650>] mmput+0x118/0x11c > > > > [ 601.976335] [<ffffffc0000c3ec4>] do_exit+0x2dc/0xa74 > > > > [ 601.981321] [<ffffffc0000c46f8>] do_group_exit+0x4c/0xe4 > > > > [ 601.986656] [<ffffffc0000d0f34>] get_signal+0x444/0x5e0 > > > > [ 601.991904] [<ffffffc000089fcc>] do_signal+0x1d8/0x450 > > > > [ 601.997065] [<ffffffc00008a35c>] do_notify_resume+0x70/0x78 > > > > > > So this is a hung task triggering because the exiting task cannot get > > > the mmap sem for write because the ksmd holds it for read while > > > allocating memory which just takes ages to complete, right? > > > > > > > > > > > The root case is that ksmd hold the read lock. and the lock is not released. > > > > scan_get_next_rmap_item > > > > down_read > > > > get_next_rmap_item > > > > alloc_rmap_item #ksmd will loop permanently. > > > > > > > > How do you see this kind of situation ? or let the issue alone. > > > > > > I am not familiar with the ksmd code so it is hard for me to judge but > > > one thing to do would be __GFP_NORETRY which would force a bail out from > > > the allocation rather than looping for ever. A quick look tells me that > > > the allocation failure here is quite easy to handle. There might be > > > others... > > Yes, very good suggestion in this case: the ksmd code does exactly the > right thing when that allocation fails, but was too stupid to use an > allocation mode which might fail - and it can allocate rather a lot of > slots along that path, so it will be good to let it break out there. > > Thank you, Zhongjiang, please send akpm a fully signed-off patch, tagged > for stable, with your explanation above (which was a lot more helpful > to me than what you wrote in your other mail of Sept 13th). But please > make it GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN (and break that line agreed > before 80 cols): the allocation will sometimes fail, and we're not at > all interested in hearing about that. > > Michal, how would you feel about this or a separate patch adding > __GFP_HIGH to the allocation in ksm's alloc_stable_node()? That > allocation could cause the same problem, but it is much less common > (so less important to do anything about it), and differs from the > rmap_item case in that if it succeeds, it will usually free a page; > whereas if it fails, the fallback (two break_cow()s) may want to > allocate a couple of pages. So __GFP_HIGH makes more sense for it > than __GFP_NORETRY: but perhaps we prefer not to add __GFP_HIGHs? I am not familiar with the ksmd code enough to have a strong opinion here. __GFP_HIGH should be imho used only when really necessary but as you point out and comment in cmp_and_merge_page explain /* * If we fail to insert the page into the stable tree, * we will have 2 virtual addresses that are pointing * to a ksm page left outside the stable tree, * in which case we need to break_cow on both. */ this can actually save some memory if succeed. So I will leave the decision to you. I have no experience in how much this path can actually eat and whether the flag actually makes much difference. Thanks! -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-16 22:13 ` Hugh Dickins 2016-09-17 15:56 ` Michal Hocko @ 2016-09-18 4:04 ` zhong jiang 2016-09-18 14:42 ` Michal Hocko 1 sibling, 1 reply; 26+ messages in thread From: zhong jiang @ 2016-09-18 4:04 UTC (permalink / raw) To: Hugh Dickins Cc: Michal Hocko, akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo On 2016/9/17 6:13, Hugh Dickins wrote: > On Wed, 14 Sep 2016, Michal Hocko wrote: >> On Wed 14-09-16 10:42:19, Michal Hocko wrote: >>> [Let's CC Hugh] >> now for real... >> >>> On Wed 14-09-16 15:13:50, zhong jiang wrote: >>> [...] >>>> hi, Michal >>>> >>>> Recently, I hit the same issue when run a OOM case of the LTP and ksm enable. >>>> >>>> [ 601.937145] Call trace: >>>> [ 601.939600] [<ffffffc000086a88>] __switch_to+0x74/0x8c >>>> [ 601.944760] [<ffffffc000a1bae0>] __schedule+0x23c/0x7bc >>>> [ 601.950007] [<ffffffc000a1c09c>] schedule+0x3c/0x94 >>>> [ 601.954907] [<ffffffc000a1eb84>] rwsem_down_write_failed+0x214/0x350 >>>> [ 601.961289] [<ffffffc000a1e32c>] down_write+0x64/0x80 >>>> [ 601.966363] [<ffffffc00021f794>] __ksm_exit+0x90/0x19c >>>> [ 601.971523] [<ffffffc0000be650>] mmput+0x118/0x11c >>>> [ 601.976335] [<ffffffc0000c3ec4>] do_exit+0x2dc/0xa74 >>>> [ 601.981321] [<ffffffc0000c46f8>] do_group_exit+0x4c/0xe4 >>>> [ 601.986656] [<ffffffc0000d0f34>] get_signal+0x444/0x5e0 >>>> [ 601.991904] [<ffffffc000089fcc>] do_signal+0x1d8/0x450 >>>> [ 601.997065] [<ffffffc00008a35c>] do_notify_resume+0x70/0x78 >>> So this is a hung task triggering because the exiting task cannot get >>> the mmap sem for write because the ksmd holds it for read while >>> allocating memory which just takes ages to complete, right? >>> >>>> The root case is that ksmd hold the read lock. and the lock is not released. >>>> scan_get_next_rmap_item >>>> down_read >>>> get_next_rmap_item >>>> alloc_rmap_item #ksmd will loop permanently. >>>> >>>> How do you see this kind of situation ? or let the issue alone. >>> I am not familiar with the ksmd code so it is hard for me to judge but >>> one thing to do would be __GFP_NORETRY which would force a bail out from >>> the allocation rather than looping for ever. A quick look tells me that >>> the allocation failure here is quite easy to handle. There might be >>> others... > Yes, very good suggestion in this case: the ksmd code does exactly the > right thing when that allocation fails, but was too stupid to use an > allocation mode which might fail - and it can allocate rather a lot of > slots along that path, so it will be good to let it break out there. > > Thank you, Zhongjiang, please send akpm a fully signed-off patch, tagged > for stable, with your explanation above (which was a lot more helpful > to me than what you wrote in your other mail of Sept 13th). But please > make it GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN (and break that line > before 80 cols): the allocation will sometimes fail, and we're not at > all interested in hearing about that. > > Michal, how would you feel about this or a separate patch adding > __GFP_HIGH to the allocation in ksm's alloc_stable_node()? That > allocation could cause the same problem, but it is much less common > (so less important to do anything about it), and differs from the > rmap_item case in that if it succeeds, it will usually free a page; > whereas if it fails, the fallback (two break_cow()s) may want to > allocate a couple of pages. So __GFP_HIGH makes more sense for it > than __GFP_NORETRY: but perhaps we prefer not to add __GFP_HIGHs? > > Hugh > > . > I agree. it indeed make progress. if alloc_stable_node fails to allocate memory, some memory need to obtain from kernel at same time. the pressure suddenly will increase. index 5048083..72dc475 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -299,7 +299,7 @@ static inline void free_rmap_item(struct rmap_item *rmap_item) static inline struct stable_node *alloc_stable_node(void) { - return kmem_cache_alloc(stable_node_cache, GFP_KERNEL); + return kmem_cache_alloc(stable_node_cache, __GFP_HIGH); } -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-18 4:04 ` zhong jiang @ 2016-09-18 14:42 ` Michal Hocko 2016-09-19 17:27 ` Hugh Dickins 0 siblings, 1 reply; 26+ messages in thread From: Michal Hocko @ 2016-09-18 14:42 UTC (permalink / raw) To: zhong jiang Cc: Hugh Dickins, akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo On Sun 18-09-16 12:04:47, zhong jiang wrote: [...] > index 5048083..72dc475 100644 > --- a/mm/ksm.c > +++ b/mm/ksm.c > @@ -299,7 +299,7 @@ static inline void free_rmap_item(struct rmap_item *rmap_item) > > static inline struct stable_node *alloc_stable_node(void) > { > - return kmem_cache_alloc(stable_node_cache, GFP_KERNEL); > + return kmem_cache_alloc(stable_node_cache, __GFP_HIGH); > } I do not want to speak for Hugh but I believe he meant something different. The above will grant access to memory reserves but it doesn't wake kswapd nor the direct reclaim. I guess he meant GFP_KERNEL | __GFP_HIGH -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] mm: fix oom work when memory is under pressure 2016-09-18 14:42 ` Michal Hocko @ 2016-09-19 17:27 ` Hugh Dickins 0 siblings, 0 replies; 26+ messages in thread From: Hugh Dickins @ 2016-09-19 17:27 UTC (permalink / raw) To: zhong jiang Cc: Michal Hocko, Hugh Dickins, akpm, vbabka, rientjes, linux-mm, Xishi Qiu, Hanjun Guo On Sun, 18 Sep 2016, Michal Hocko wrote: > On Sun 18-09-16 12:04:47, zhong jiang wrote: > [...] > > index 5048083..72dc475 100644 > > --- a/mm/ksm.c > > +++ b/mm/ksm.c > > @@ -299,7 +299,7 @@ static inline void free_rmap_item(struct rmap_item *rmap_item) > > > > static inline struct stable_node *alloc_stable_node(void) > > { > > - return kmem_cache_alloc(stable_node_cache, GFP_KERNEL); > > + return kmem_cache_alloc(stable_node_cache, __GFP_HIGH); > > } > > I do not want to speak for Hugh but I believe he meant something > different. The above will grant access to memory reserves but it doesn't > wake kswapd nor the direct reclaim. I guess he meant GFP_KERNEL | __GFP_HIGH You speak for me correctly, Michal: sorry I wasn't clear, Zhongjiang, yes, I meant __GFP_HIGH as a modifier for GFP_KERNEL: GFP_KERNEL | __GFP_HIGH And after running it past Michal and thinking on it some more, I do still think that it's the right thing to do for alloc_stable_node(). But please only include that change in your patch if you yourself are comfortable with it: it is very definitely a much lower order issue than the alloc_rmap_item() issue, and can always be added later. (I have no view on whether and how this problem still occurs with OOM reaper: I leave the thinking on that to you and the experts.) Hugh -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2016-09-19 17:27 UTC | newest] Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-09-06 14:47 [PATCH] mm: fix oom work when memory is under pressure zhongjiang 2016-09-09 11:44 ` Michal Hocko 2016-09-12 9:51 ` zhong jiang 2016-09-12 11:13 ` Michal Hocko 2016-09-12 13:42 ` zhong jiang 2016-09-12 17:44 ` Michal Hocko 2016-09-13 13:13 ` zhong jiang 2016-09-13 13:28 ` Michal Hocko 2016-09-13 14:01 ` zhong jiang 2016-09-14 7:13 ` zhong jiang 2016-09-14 8:42 ` Michal Hocko 2016-09-14 8:50 ` zhong jiang 2016-09-14 9:05 ` Michal Hocko 2016-09-14 8:52 ` Michal Hocko 2016-09-14 9:25 ` zhong jiang 2016-09-14 11:29 ` Tetsuo Handa 2016-09-14 13:52 ` zhong jiang 2016-09-18 6:00 ` Tetsuo Handa 2016-09-18 6:13 ` Tetsuo Handa 2016-09-19 4:44 ` zhong jiang 2016-09-19 7:15 ` zhong jiang 2016-09-16 22:13 ` Hugh Dickins 2016-09-17 15:56 ` Michal Hocko 2016-09-18 4:04 ` zhong jiang 2016-09-18 14:42 ` Michal Hocko 2016-09-19 17:27 ` Hugh Dickins
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.