Low memory killer problem

Low memory killer problem

[zhiyuan_zhu]

[-- Attachment #1: Type: text/plain, Size: 1487 bytes --]

Loop Linux-mm, thanks.
I want to know, why lowmemory killer not included ION free memory?
Thnanks.


From: Zhiyuan Zhu(朱志遠)
Sent: Friday, May 12, 2017 4:56 PM
To: vinmenon@codeaurora.org
Cc: Kyle Lin(林昭凱); Zhiyuan Zhu(朱志遠)
Subject: Low memory killer problem

Dear Vinmenon,

I found a part of ION memory will be return to system in android platform,
But these memorys  can’t accounted in low-memory-killer strategy.
…
And I also found ION memory comes from, kmalloc/vmalloc/alloc pages/reserved memory.
What affect if account these memorys for free?
Many thanks.

…
Lowmemory killer strategy
af6c02b83 (Vinayak Menon      2015-08-27 16:29:37 +0530 418)            global_page_state(NR_FILE_PAGES) + zcache_pages())
af6c02b83 (Vinayak Menon      2015-08-27 16:29:37 +0530 419)            other_file = global_page_state(NR_FILE_PAGES) + zcache_pages() -
058dbde92 (Vinayak Menon      2014-02-27 00:36:22 +0530 420)                                            global_page_state(NR_SHMEM) -
058dbde92 (Vinayak Menon      2014-02-27 00:36:22 +0530 422)                                            total_swapcache_pages();



Meminfo example
$ adb shell cat /proc/meminfo
MemTotal:        3805312 kB
MemFree:         1446220 kB
MemAvailable:    2388384 kB
Buffers:           16796 kB
Cached:          1190868 kB
…
IonTotal:         224252 kB
IonInUse:         199108 kB



Thanks
BR
Zhiyuan zhu

[-- Attachment #2: Type: text/html, Size: 8056 bytes --]

RE: Low memory killer problem

[zhiyuan_zhu]

[-- Attachment #1: Type: text/plain, Size: 1746 bytes --]

Loop MM maintainers,

Dear All,

Who can talk about this problem? Thanks.
Maybe this is lowmemory killer’s bug ?
ION memory is complex now, we need to have a breakdown for them, I think.

Thanks a lot
Zhiyuan zhu
From: Zhiyuan Zhu(朱志遠)
Sent: Friday, May 12, 2017 4:56 PM
To: vinmenon@codeaurora.org
Cc: Kyle Lin(林昭凱); Zhiyuan Zhu(朱志遠)
Subject: Low memory killer problem

Dear Vinmenon,

I found a part of ION memory will be return to system in android platform,
But these memorys  can’t accounted in low-memory-killer strategy.
…
And I also found ION memory comes from, kmalloc/vmalloc/alloc pages/reserved memory.
What affect if account these memorys for free?
Many thanks.

…
Lowmemory killer strategy
af6c02b83 (Vinayak Menon      2015-08-27 16:29:37 +0530 418)            global_page_state(NR_FILE_PAGES) + zcache_pages())
af6c02b83 (Vinayak Menon      2015-08-27 16:29:37 +0530 419)            other_file = global_page_state(NR_FILE_PAGES) + zcache_pages() -
058dbde92 (Vinayak Menon      2014-02-27 00:36:22 +0530 420)                                            global_page_state(NR_SHMEM) -
5c4698e38 (kyle_lin           2015-04-16 16:04:22 +0800 421)                                            global_page_state(NR_MLOCK) -
058dbde92 (Vinayak Menon      2014-02-27 00:36:22 +0530 422)                                            total_swapcache_pages();



Meminfo example
$ adb shell cat /proc/meminfo
MemTotal:        3805312 kB
MemFree:         1446220 kB
MemAvailable:    2388384 kB
Buffers:           16796 kB
Cached:          1190868 kB
…
IonTotal:         224252 kB
IonInUse:         199108 kB



Thanks
BR
Zhiyuan zhu

[-- Attachment #2: Type: text/html, Size: 9170 bytes --]

Re: Low memory killer problem

[Greg KH]

On Mon, May 15, 2017 at 07:25:20AM +0000, zhiyuan_zhu@htc.com wrote:
> Loop MM maintainers,
> 
>  
> 
> Dear All,
> 
>  
> 
> Who can talk about this problem? Thanks.

What problem?

> Maybe this is lowmemory killera??s bug ?

This code is now removed from the kernel, so I doubt there could be a
bug in it :)

> ION memory is complex now, we need to have a breakdown for them, I think.

What do you mean by this?

thanks,

greg k-h

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: Low memory killer problem

[zhiyuan_zhu]

Dear Greg, 

Very sorry my mail history is lost.

I found a part of ION memory will be return to system in android platform,
But these memorys  can’t accounted in low-memory-killer strategy.
…
And I also found ION memory comes from,  kmalloc/vmalloc/alloc pages/reserved memory.
I understand reserved memory shouldn't accounted to free memory.
But the memory which alloced by kmalloc/vmalloc/alloc pages, can be reclaimed.

But the low-memory killer can't accounted this part,
Many thanks.

Code location, 
   ---> drivers/staging/android/lowmemorykiller.c   -> lowmem_scan

Code review,
415         other_free = global_page_state(NR_FREE_PAGES);
416 
417         if (global_page_state(NR_SHMEM) + global_page_state(NR_MLOCK) + total_swapcache_pages() <
418                 global_page_state(NR_FILE_PAGES) + zcache_pages())
419                 other_file = global_page_state(NR_FILE_PAGES) + zcache_pages() -
420                                                 global_page_state(NR_SHMEM) -
422                                                 total_swapcache_pages();

So, other_free means:  MemFree,
other_file means: Buffers + Cached - ShMEM - SwapCache.
And part of  ( IonInUse - IonTotal ) should be reclaimed but not accounted.
Thanks.

Meminfo example
$ adb shell cat /proc/meminfo
MemTotal:        3805312 kB
MemFree:         1446220 kB
MemAvailable:    2388384 kB
Buffers:           16796 kB
Cached:          1190868 kB
…
IonTotal:         224252 kB
IonInUse:         199108 kB


Thanks
BR
Zhiyuan zhu

-----Original Message-----
From: Greg KH [mailto:gregkh@linuxfoundation.org] 
Sent: Monday, May 15, 2017 4:06 PM
To: Zhiyuan Zhu(朱志遠)
Cc: vinmenon@codeaurora.org; linux-mm@kvack.org; skhiani@codeaurora.org; torvalds@linux-foundation.org
Subject: Re: Low memory killer problem

On Mon, May 15, 2017 at 07:25:20AM +0000, zhiyuan_zhu@htc.com wrote:
> Loop MM maintainers,
> 
>  
> 
> Dear All,
> 
>  
> 
> Who can talk about this problem? Thanks.

What problem?

> Maybe this is lowmemory killer’s bug ?

This code is now removed from the kernel, so I doubt there could be a bug in it :)

> ION memory is complex now, we need to have a breakdown for them, I think.

What do you mean by this?

thanks,

greg k-h

Re: Low memory killer problem

[Greg KH]

On Mon, May 15, 2017 at 08:22:38AM +0000, zhiyuan_zhu@htc.com wrote:
> Dear Greg, 
> 
> Very sorry my mail history is lost.
> 
> I found a part of ION memory will be return to system in android platform,
> But these memorys  cana??t accounted in low-memory-killer strategy.
> a?|
> And I also found ION memory comes from,  kmalloc/vmalloc/alloc pages/reserved memory.
> I understand reserved memory shouldn't accounted to free memory.
> But the memory which alloced by kmalloc/vmalloc/alloc pages, can be reclaimed.
> 
> But the low-memory killer can't accounted this part,
> Many thanks.
> 
> Code location, 
>    ---> drivers/staging/android/lowmemorykiller.c  A -> lowmem_scan

That file is gone from the latest kernel release, sorry.  So there's not
much we can do about this code anymore.

See the mailing list archives for what should be used instead of this
code, there is a plan for what to do.

Also note that the ION code has had a lot of reworks lately as well.

good luck!

greg k-h

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: Low memory killer problem

[zhiyuan_zhu]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1887 bytes --]

Thanks for your remind,
I found lowmemorykiller.c have been removed, and ION module still exist since v4.12-rc1.
I will pay attention to ION module.

But I still have 3 questions,
Is there any substitute for low-memory-killer after kernel v4.12-rc1 ?
Can I accounted the ION free to free memory?
Is there any different from ION free and the normal system memory free?

ION free means:   IonTotal - IonInUse  - ION reserved memory.
Thanks a lot.

-----Original Message-----
From: Greg KH [mailto:gregkh@linuxfoundation.org] 
Sent: Monday, May 15, 2017 5:00 PM
To: Zhiyuan Zhu(朱志遠)
Cc: vinmenon@codeaurora.org; linux-mm@kvack.org; skhiani@codeaurora.org; torvalds@linux-foundation.org; Jet Li(李發傑)
Subject: Re: Low memory killer problem

On Mon, May 15, 2017 at 08:22:38AM +0000, zhiyuan_zhu@htc.com wrote:
> Dear Greg,
> 
> Very sorry my mail history is lost.
> 
> I found a part of ION memory will be return to system in android 
> platform, But these memorys  can’t accounted in low-memory-killer strategy.
> …
> And I also found ION memory comes from,  kmalloc/vmalloc/alloc pages/reserved memory.
> I understand reserved memory shouldn't accounted to free memory.
> But the memory which alloced by kmalloc/vmalloc/alloc pages, can be reclaimed.
> 
> But the low-memory killer can't accounted this part, Many thanks.
> 
> Code location, 
>    ---> drivers/staging/android/lowmemorykiller.c   -> lowmem_scan

That file is gone from the latest kernel release, sorry.  So there's not much we can do about this code anymore.

See the mailing list archives for what should be used instead of this code, there is a plan for what to do.

Also note that the ION code has had a lot of reworks lately as well.

good luck!

greg k-h

N‹§²æìr¸›zǧu©ž²Æ {\b­†éì¹»\x1c®&Þ–)îÆi¢žØ^n‡r¶‰šŽŠÝ¢j$½§$¢¸\x05¢¹¨­è§~Š'.)îÄÃ,yèm¶ŸÿÃ\f%Š{±šj+ƒðèž×¦j)Z†·Ÿ

Re: Low memory killer problem

[Greg KH]

On Tue, May 16, 2017 at 03:41:31AM +0000, zhiyuan_zhu@htc.com wrote:
> Thanks for your remind,
> I found lowmemorykiller.c have been removed, and ION module still exist since v4.12-rc1.
> I will pay attention to ION module.
> 
> But I still have 3 questions,
> Is there any substitute for low-memory-killer after kernel v4.12-rc1 ?

See the email thread when it was removed, there was some proposals on
how to do this "correctly".  I know someone at Google is currently
working on this, hopefully they have something to post soon about it.

> Can I accounted the ION free to free memory?
> Is there any different from ION free and the normal system memory free?

No idea, try asking the ION developers :)

good luck!

greg k-h

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Low memory killer problem

[Laura Abbott]

On 05/15/2017 08:41 PM, zhiyuan_zhu@htc.com wrote:
> Thanks for your remind,
> I found lowmemorykiller.c have been removed, and ION module still exist since v4.12-rc1.
> I will pay attention to ION module.
> 
> But I still have 3 questions,
> Is there any substitute for low-memory-killer after kernel v4.12-rc1 ?
> Can I accounted the ION free to free memory?
> Is there any different from ION free and the normal system memory free?
> 
> ION free means:   IonTotal - IonInUse  - ION reserved memory.
> Thanks a lot.
> 

Issues like this are exactly why the LMK was deleted. The problem
is the LMK is hooked up as a shrinker so it runs in parallel to
any other shrinker. The short answer is yes if you want the LMK
to do anything reasonable you probably need to tweak it to account
for other memory that may be held in the system (Ion, zswap etc.).
There never seemed to be one universal heuristic that worked for
everyone which was part of the reason why most changes exist downstream.
Using some combination of the Ion variables above would work if
you experiment. If this sounds like a non-answer, that's because it is.

Thanks,
Laura

> -----Original Message-----
> From: Greg KH [mailto:gregkh@linuxfoundation.org] 
> Sent: Monday, May 15, 2017 5:00 PM
> To: Zhiyuan Zhu(ae?+-a??e? )
> Cc: vinmenon@codeaurora.org; linux-mm@kvack.org; skhiani@codeaurora.org; torvalds@linux-foundation.org; Jet Li(ae??c? 1/4 a??)
> Subject: Re: Low memory killer problem
> 
> On Mon, May 15, 2017 at 08:22:38AM +0000, zhiyuan_zhu@htc.com wrote:
>> Dear Greg,
>>
>> Very sorry my mail history is lost.
>>
>> I found a part of ION memory will be return to system in android 
>> platform, But these memorys  cana??t accounted in low-memory-killer strategy.
>> a?|
>> And I also found ION memory comes from,  kmalloc/vmalloc/alloc pages/reserved memory.
>> I understand reserved memory shouldn't accounted to free memory.
>> But the memory which alloced by kmalloc/vmalloc/alloc pages, can be reclaimed.
>>
>> But the low-memory killer can't accounted this part, Many thanks.
>>
>> Code location, 
>>    ---> drivers/staging/android/lowmemorykiller.c   -> lowmem_scan
> 
> That file is gone from the latest kernel release, sorry.  So there's not much we can do about this code anymore.
> 
> See the mailing list archives for what should be used instead of this code, there is a plan for what to do.
> 
> Also note that the ION code has had a lot of reworks lately as well.
> 
> good luck!
> 
> greg k-h
> 
> Ni? 1/2 i? 1/2 i? 1/2 i? 1/2 i? 1/2 ri? 1/2 i? 1/2 zC?ui? 1/2 i? 1/2 i? 1/2 AE {\bi? 1/2 i? 1/2 i? 1/2 i1>>\x1ci? 1/2 &TH?)i? 1/2 i? 1/2 ii? 1/2 i? 1/2 i? 1/2 ^ni? 1/2 ri? 1/2 i? 1/2 i? 1/2 i? 1/2 i? 1/2 Ycj$i? 1/2 i? 1/2 $i? 1/2 i? 1/2 \x05i? 1/2 i? 1/2 i? 1/2 i? 1/2 i? 1/2 ~i? 1/2 '.)i? 1/2 i? 1/2 i? 1/2 ,yi? 1/2 mi? 1/2 i? 1/2 i? 1/2 i? 1/2 \f%i? 1/2 {i? 1/2 i? 1/2 j+i? 1/2 i? 1/2 i? 1/2 x|j)Zi? 1/2 i? 1/2 i? 1/2 \x02fi? 1/2 i? 1/2 \x1di? 1/2 {di? 1/2 i? 1/2 $i? 1/2 i? 1/2 \x1ei? 1/2 i? 1/2 i? 1/2 i? 1/2 i? 1/2 i? 1/2 i? 1/2 i? 1/2 i? 1/2 /a==
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: Low memory killer problem

[zhiyuan_zhu]

Thanks Laura,
Indeed, some part of ION memory have registered shrinker.

/proc/meminfo have no account for the memory which ION can be shrinked part, but just have IonTotal and IonInUse.
And another problem is: ION memory is special, maybe hard to reclaim.
So, wating for Google's solution now.

By the way, which E-mail client do you use?  :)
Thanks
Zhiyuan zhu

-----Original Message-----
From: Laura Abbott [mailto:labbott@redhat.com] 
Sent: Wednesday, May 17, 2017 6:37 AM
To: Zhiyuan Zhu(朱志遠); gregkh@linuxfoundation.org
Cc: vinmenon@codeaurora.org; linux-mm@kvack.org; skhiani@codeaurora.org; torvalds@linux-foundation.org; Jet Li(李發傑)
Subject: Re: Low memory killer problem

On 05/15/2017 08:41 PM, zhiyuan_zhu@htc.com wrote:
> Thanks for your remind,
> I found lowmemorykiller.c have been removed, and ION module still exist since v4.12-rc1.
> I will pay attention to ION module.
> 
> But I still have 3 questions,
> Is there any substitute for low-memory-killer after kernel v4.12-rc1 ?
> Can I accounted the ION free to free memory?
> Is there any different from ION free and the normal system memory free?
> 
> ION free means:   IonTotal - IonInUse  - ION reserved memory.
> Thanks a lot.
> 

Issues like this are exactly why the LMK was deleted. The problem is the LMK is hooked up as a shrinker so it runs in parallel to any other shrinker. The short answer is yes if you want the LMK to do anything reasonable you probably need to tweak it to account for other memory that may be held in the system (Ion, zswap etc.).
There never seemed to be one universal heuristic that worked for everyone which was part of the reason why most changes exist downstream.
Using some combination of the Ion variables above would work if you experiment. If this sounds like a non-answer, that's because it is.

Thanks,
Laura

> -----Original Message-----
> From: Greg KH [mailto:gregkh@linuxfoundation.org]
> Sent: Monday, May 15, 2017 5:00 PM
> To: Zhiyuan Zhu(朱志遠)
> Cc: vinmenon@codeaurora.org; linux-mm@kvack.org; 
> skhiani@codeaurora.org; torvalds@linux-foundation.org; Jet Li(李發傑)
> Subject: Re: Low memory killer problem
> 
> On Mon, May 15, 2017 at 08:22:38AM +0000, zhiyuan_zhu@htc.com wrote:
>> Dear Greg,
>>
>> Very sorry my mail history is lost.
>>
>> I found a part of ION memory will be return to system in android 
>> platform, But these memorys  can’t accounted in low-memory-killer strategy.
>> …
>> And I also found ION memory comes from,  kmalloc/vmalloc/alloc pages/reserved memory.
>> I understand reserved memory shouldn't accounted to free memory.
>> But the memory which alloced by kmalloc/vmalloc/alloc pages, can be reclaimed.
>>
>> But the low-memory killer can't accounted this part, Many thanks.
>>
>> Code location, 
>>    ---> drivers/staging/android/lowmemorykiller.c   -> lowmem_scan
> 
> That file is gone from the latest kernel release, sorry.  So there's not much we can do about this code anymore.
> 
> See the mailing list archives for what should be used instead of this code, there is a plan for what to do.
> 
> Also note that the ION code has had a lot of reworks lately as well.
> 
> good luck!
> 
> greg k-h
> 
> N     r  zǧu   Ơ{\b   칻\x1c &ޖ)  i   ^n r     ݢj$  $  \x05     ~ '.)   ,y m  
>   
% {  j+   צj)Z   \x02f  \x1d {d  $  \x1e         /a==
>