CMA related memory questions

CMA related memory questions

[Jun Nie]

On 2015?01?30? 10:36, Jun Nie wrote:
> Hi Marek & Arnd,
>
> Did you ever know issue that free CMA memory is high, but system is
> hungry for memory and page cache is very low? I am enabling CMA in
> Android on my board with 512MB memory and see FreeMem in /proc/meminfo
> increase a lot with CMA comparing the reservation solution on boot. But
> I find system is not borrowing memory from CMA pool when running 3dmark
> (high webkit workload at start). Because the FreeMem size is high, but
> cache size decreasing significantly to several MB during benchmark run,
> I suppose system is trying to reclaim memory from pagecache for new
> allocation. My question is that what API that page cache and webkit
> related functionality are using to allocate memory. Maybe page cache
> require memory that is not movable/reclaimable memory, where we may have
> optimization to go thru dma_alloc_xxx to borrow CMA memory? I suppose
> app level memory allocation shall be movable/reclaimable memory and can
> borrow from CMA pool, but not sure whether the flag match the
> movable/reclaimable memory and go thru the right path.
>
> Could you help share your experience/thoughts on this? Thanks!
>
>
> B.R.
> Jun

Add more people.

Re: CMA related memory questions

[Heesub Shin]

On 01/30/2015 11:57 AM, Jun Nie wrote:
> On 2015a1'01ae??30ae?JPY 10:36, Jun Nie wrote:
>> Hi Marek & Arnd,
>>
>> Did you ever know issue that free CMA memory is high, but system is
>> hungry for memory and page cache is very low? I am enabling CMA in
>> Android on my board with 512MB memory and see FreeMem in /proc/meminfo
>> increase a lot with CMA comparing the reservation solution on boot. But
>> I find system is not borrowing memory from CMA pool when running 3dmark
>> (high webkit workload at start). Because the FreeMem size is high, but
>> cache size decreasing significantly to several MB during benchmark run,
>> I suppose system is trying to reclaim memory from pagecache for new
>> allocation. My question is that what API that page cache and webkit
>> related functionality are using to allocate memory. Maybe page cache
>> require memory that is not movable/reclaimable memory, where we may have
>> optimization to go thru dma_alloc_xxx to borrow CMA memory? I suppose
>> app level memory allocation shall be movable/reclaimable memory and can
>> borrow from CMA pool, but not sure whether the flag match the
>> movable/reclaimable memory and go thru the right path.
>>
>> Could you help share your experience/thoughts on this? Thanks!

CC'ed linux-mm@kvack.org

__zone_watermark_ok() assumes that free pages from CMA pageblock are not 
free when ALLOC_CMA is not set on alloc_flags. The main goal was to 
force core mm to keep some non-CMA always free and thus let kernel to 
allocate a few unmovable pages from any context (including atomic, irq, 
etc.). However, this behavior may cause excessive page reclamation as it 
is sometimes very hard to satisfy the high wmark + balance_gap with only 
non-CMA pages and reclaiming CMA pages does not help at all.

It is observed that page cache pages are excessively reclaimed and 
entire system falls into thrashing even though the amount of free pages 
are much higer than the high wmark. In this case, majority of the free 
pages were from CMA page block (and about 30% pages in highmem zone were 
from CMA pageblock). Therefore, kswapd kept running and reclaiming too 
many pages. Although it is relatively rare and only observed on a 
specific workload, the device gets in an unresponsive state for a while 
(up to 10 secs), once it happens.

regards,
heesub

>>
>>
>> B.R.
>> Jun
>
> Add more people.
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

CMA related memory questions

[Heesub Shin]

On 01/30/2015 11:57 AM, Jun Nie wrote:
> On 2015?01?30? 10:36, Jun Nie wrote:
>> Hi Marek & Arnd,
>>
>> Did you ever know issue that free CMA memory is high, but system is
>> hungry for memory and page cache is very low? I am enabling CMA in
>> Android on my board with 512MB memory and see FreeMem in /proc/meminfo
>> increase a lot with CMA comparing the reservation solution on boot. But
>> I find system is not borrowing memory from CMA pool when running 3dmark
>> (high webkit workload at start). Because the FreeMem size is high, but
>> cache size decreasing significantly to several MB during benchmark run,
>> I suppose system is trying to reclaim memory from pagecache for new
>> allocation. My question is that what API that page cache and webkit
>> related functionality are using to allocate memory. Maybe page cache
>> require memory that is not movable/reclaimable memory, where we may have
>> optimization to go thru dma_alloc_xxx to borrow CMA memory? I suppose
>> app level memory allocation shall be movable/reclaimable memory and can
>> borrow from CMA pool, but not sure whether the flag match the
>> movable/reclaimable memory and go thru the right path.
>>
>> Could you help share your experience/thoughts on this? Thanks!

CC'ed linux-mm at kvack.org

__zone_watermark_ok() assumes that free pages from CMA pageblock are not 
free when ALLOC_CMA is not set on alloc_flags. The main goal was to 
force core mm to keep some non-CMA always free and thus let kernel to 
allocate a few unmovable pages from any context (including atomic, irq, 
etc.). However, this behavior may cause excessive page reclamation as it 
is sometimes very hard to satisfy the high wmark + balance_gap with only 
non-CMA pages and reclaiming CMA pages does not help at all.

It is observed that page cache pages are excessively reclaimed and 
entire system falls into thrashing even though the amount of free pages 
are much higer than the high wmark. In this case, majority of the free 
pages were from CMA page block (and about 30% pages in highmem zone were 
from CMA pageblock). Therefore, kswapd kept running and reclaiming too 
many pages. Although it is relatively rare and only observed on a 
specific workload, the device gets in an unresponsive state for a while 
(up to 10 secs), once it happens.

regards,
heesub

>>
>>
>> B.R.
>> Jun
>
> Add more people.
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: CMA related memory questions

[Jun Nie]

On 2015a1'01ae??30ae?JPY 11:25, Heesub Shin wrote:
>
>
> On 01/30/2015 11:57 AM, Jun Nie wrote:
>> On 2015a1'01ae??30ae?JPY 10:36, Jun Nie wrote:
>>> Hi Marek & Arnd,
>>>
>>> Did you ever know issue that free CMA memory is high, but system is
>>> hungry for memory and page cache is very low? I am enabling CMA in
>>> Android on my board with 512MB memory and see FreeMem in /proc/meminfo
>>> increase a lot with CMA comparing the reservation solution on boot. But
>>> I find system is not borrowing memory from CMA pool when running 3dmark
>>> (high webkit workload at start). Because the FreeMem size is high, but
>>> cache size decreasing significantly to several MB during benchmark run,
>>> I suppose system is trying to reclaim memory from pagecache for new
>>> allocation. My question is that what API that page cache and webkit
>>> related functionality are using to allocate memory. Maybe page cache
>>> require memory that is not movable/reclaimable memory, where we may have
>>> optimization to go thru dma_alloc_xxx to borrow CMA memory? I suppose
>>> app level memory allocation shall be movable/reclaimable memory and can
>>> borrow from CMA pool, but not sure whether the flag match the
>>> movable/reclaimable memory and go thru the right path.
>>>
>>> Could you help share your experience/thoughts on this? Thanks!
>
> CC'ed linux-mm@kvack.org
>
> __zone_watermark_ok() assumes that free pages from CMA pageblock are not
> free when ALLOC_CMA is not set on alloc_flags. The main goal was to
> force core mm to keep some non-CMA always free and thus let kernel to
> allocate a few unmovable pages from any context (including atomic, irq,
> etc.). However, this behavior may cause excessive page reclamation as it
> is sometimes very hard to satisfy the high wmark + balance_gap with only
> non-CMA pages and reclaiming CMA pages does not help at all.
Seems it is tricky to tune it. Could you help share some experience on 
this, how to change the parameters, what's pro/con? Thanks!
>
> It is observed that page cache pages are excessively reclaimed and
> entire system falls into thrashing even though the amount of free pages
> are much higer than the high wmark. In this case, majority of the free
> pages were from CMA page block (and about 30% pages in highmem zone were
> from CMA pageblock). Therefore, kswapd kept running and reclaiming too
> many pages. Although it is relatively rare and only observed on a
> specific workload, the device gets in an unresponsive state for a while
> (up to 10 secs), once it happens.
>
I am in this situation. kswapd is busy and most FreeMem is from CMA 
because I have 192MB CMA memblock and most of them are free.
> regards,
> heesub
>
>>>
>>>
>>> B.R.
>>> Jun
>>
>> Add more people.
>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> linux-arm-kernel@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

CMA related memory questions

[Jun Nie]

On 2015?01?30? 11:25, Heesub Shin wrote:
>
>
> On 01/30/2015 11:57 AM, Jun Nie wrote:
>> On 2015?01?30? 10:36, Jun Nie wrote:
>>> Hi Marek & Arnd,
>>>
>>> Did you ever know issue that free CMA memory is high, but system is
>>> hungry for memory and page cache is very low? I am enabling CMA in
>>> Android on my board with 512MB memory and see FreeMem in /proc/meminfo
>>> increase a lot with CMA comparing the reservation solution on boot. But
>>> I find system is not borrowing memory from CMA pool when running 3dmark
>>> (high webkit workload at start). Because the FreeMem size is high, but
>>> cache size decreasing significantly to several MB during benchmark run,
>>> I suppose system is trying to reclaim memory from pagecache for new
>>> allocation. My question is that what API that page cache and webkit
>>> related functionality are using to allocate memory. Maybe page cache
>>> require memory that is not movable/reclaimable memory, where we may have
>>> optimization to go thru dma_alloc_xxx to borrow CMA memory? I suppose
>>> app level memory allocation shall be movable/reclaimable memory and can
>>> borrow from CMA pool, but not sure whether the flag match the
>>> movable/reclaimable memory and go thru the right path.
>>>
>>> Could you help share your experience/thoughts on this? Thanks!
>
> CC'ed linux-mm at kvack.org
>
> __zone_watermark_ok() assumes that free pages from CMA pageblock are not
> free when ALLOC_CMA is not set on alloc_flags. The main goal was to
> force core mm to keep some non-CMA always free and thus let kernel to
> allocate a few unmovable pages from any context (including atomic, irq,
> etc.). However, this behavior may cause excessive page reclamation as it
> is sometimes very hard to satisfy the high wmark + balance_gap with only
> non-CMA pages and reclaiming CMA pages does not help at all.
Seems it is tricky to tune it. Could you help share some experience on 
this, how to change the parameters, what's pro/con? Thanks!
>
> It is observed that page cache pages are excessively reclaimed and
> entire system falls into thrashing even though the amount of free pages
> are much higer than the high wmark. In this case, majority of the free
> pages were from CMA page block (and about 30% pages in highmem zone were
> from CMA pageblock). Therefore, kswapd kept running and reclaiming too
> many pages. Although it is relatively rare and only observed on a
> specific workload, the device gets in an unresponsive state for a while
> (up to 10 secs), once it happens.
>
I am in this situation. kswapd is busy and most FreeMem is from CMA 
because I have 192MB CMA memblock and most of them are free.
> regards,
> heesub
>
>>>
>>>
>>> B.R.
>>> Jun
>>
>> Add more people.
>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> linux-arm-kernel at lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: CMA related memory questions

[Heesub Shin]

On 01/30/2015 12:43 PM, Jun Nie wrote:
> On 2015a1'01ae??30ae?JPY 11:25, Heesub Shin wrote:
>>
>>
>> On 01/30/2015 11:57 AM, Jun Nie wrote:
>>> On 2015a1'01ae??30ae?JPY 10:36, Jun Nie wrote:
>>>> Hi Marek & Arnd,
>>>>
>>>> Did you ever know issue that free CMA memory is high, but system is
>>>> hungry for memory and page cache is very low? I am enabling CMA in
>>>> Android on my board with 512MB memory and see FreeMem in /proc/meminfo
>>>> increase a lot with CMA comparing the reservation solution on boot. But
>>>> I find system is not borrowing memory from CMA pool when running 3dmark
>>>> (high webkit workload at start). Because the FreeMem size is high, but
>>>> cache size decreasing significantly to several MB during benchmark run,
>>>> I suppose system is trying to reclaim memory from pagecache for new
>>>> allocation. My question is that what API that page cache and webkit
>>>> related functionality are using to allocate memory. Maybe page cache
>>>> require memory that is not movable/reclaimable memory, where we may
>>>> have
>>>> optimization to go thru dma_alloc_xxx to borrow CMA memory? I suppose
>>>> app level memory allocation shall be movable/reclaimable memory and can
>>>> borrow from CMA pool, but not sure whether the flag match the
>>>> movable/reclaimable memory and go thru the right path.
>>>>
>>>> Could you help share your experience/thoughts on this? Thanks!
>>
>> CC'ed linux-mm@kvack.org
>>
>> __zone_watermark_ok() assumes that free pages from CMA pageblock are not
>> free when ALLOC_CMA is not set on alloc_flags. The main goal was to
>> force core mm to keep some non-CMA always free and thus let kernel to
>> allocate a few unmovable pages from any context (including atomic, irq,
>> etc.). However, this behavior may cause excessive page reclamation as it
>> is sometimes very hard to satisfy the high wmark + balance_gap with only
>> non-CMA pages and reclaiming CMA pages does not help at all.
> Seems it is tricky to tune it. Could you help share some experience on
> this, how to change the parameters, what's pro/con? Thanks!

AFAIK, unfortunately there's no other way rather than reducing the 
number of CMA pageblocks which are making anomalies. Selectively 
ignoring CMA pages when we isolate pages from LRU could be an 
alternative, but it has another side effect. I also want to know how to 
handle this problem nicely.

>>
>> It is observed that page cache pages are excessively reclaimed and
>> entire system falls into thrashing even though the amount of free pages
>> are much higer than the high wmark. In this case, majority of the free
>> pages were from CMA page block (and about 30% pages in highmem zone were
>> from CMA pageblock). Therefore, kswapd kept running and reclaiming too
>> many pages. Although it is relatively rare and only observed on a
>> specific workload, the device gets in an unresponsive state for a while
>> (up to 10 secs), once it happens.
>>
> I am in this situation. kswapd is busy and most FreeMem is from CMA
> because I have 192MB CMA memblock and most of them are free.
>> regards,
>> heesub
>>
>>>>
>>>>
>>>> B.R.
>>>> Jun
>>>
>>> Add more people.
>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

CMA related memory questions

[Heesub Shin]

On 01/30/2015 12:43 PM, Jun Nie wrote:
> On 2015?01?30? 11:25, Heesub Shin wrote:
>>
>>
>> On 01/30/2015 11:57 AM, Jun Nie wrote:
>>> On 2015?01?30? 10:36, Jun Nie wrote:
>>>> Hi Marek & Arnd,
>>>>
>>>> Did you ever know issue that free CMA memory is high, but system is
>>>> hungry for memory and page cache is very low? I am enabling CMA in
>>>> Android on my board with 512MB memory and see FreeMem in /proc/meminfo
>>>> increase a lot with CMA comparing the reservation solution on boot. But
>>>> I find system is not borrowing memory from CMA pool when running 3dmark
>>>> (high webkit workload at start). Because the FreeMem size is high, but
>>>> cache size decreasing significantly to several MB during benchmark run,
>>>> I suppose system is trying to reclaim memory from pagecache for new
>>>> allocation. My question is that what API that page cache and webkit
>>>> related functionality are using to allocate memory. Maybe page cache
>>>> require memory that is not movable/reclaimable memory, where we may
>>>> have
>>>> optimization to go thru dma_alloc_xxx to borrow CMA memory? I suppose
>>>> app level memory allocation shall be movable/reclaimable memory and can
>>>> borrow from CMA pool, but not sure whether the flag match the
>>>> movable/reclaimable memory and go thru the right path.
>>>>
>>>> Could you help share your experience/thoughts on this? Thanks!
>>
>> CC'ed linux-mm at kvack.org
>>
>> __zone_watermark_ok() assumes that free pages from CMA pageblock are not
>> free when ALLOC_CMA is not set on alloc_flags. The main goal was to
>> force core mm to keep some non-CMA always free and thus let kernel to
>> allocate a few unmovable pages from any context (including atomic, irq,
>> etc.). However, this behavior may cause excessive page reclamation as it
>> is sometimes very hard to satisfy the high wmark + balance_gap with only
>> non-CMA pages and reclaiming CMA pages does not help at all.
> Seems it is tricky to tune it. Could you help share some experience on
> this, how to change the parameters, what's pro/con? Thanks!

AFAIK, unfortunately there's no other way rather than reducing the 
number of CMA pageblocks which are making anomalies. Selectively 
ignoring CMA pages when we isolate pages from LRU could be an 
alternative, but it has another side effect. I also want to know how to 
handle this problem nicely.

>>
>> It is observed that page cache pages are excessively reclaimed and
>> entire system falls into thrashing even though the amount of free pages
>> are much higer than the high wmark. In this case, majority of the free
>> pages were from CMA page block (and about 30% pages in highmem zone were
>> from CMA pageblock). Therefore, kswapd kept running and reclaiming too
>> many pages. Although it is relatively rare and only observed on a
>> specific workload, the device gets in an unresponsive state for a while
>> (up to 10 secs), once it happens.
>>
> I am in this situation. kswapd is busy and most FreeMem is from CMA
> because I have 192MB CMA memblock and most of them are free.
>> regards,
>> heesub
>>
>>>>
>>>>
>>>> B.R.
>>>> Jun
>>>
>>> Add more people.
>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel at lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo at kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email at kvack.org </a>
>

Re: CMA related memory questions

[Vlastimil Babka]

[CC some usual CMA suspects]

On 01/30/2015 06:14 AM, Heesub Shin wrote:
> 
> 
> On 01/30/2015 12:43 PM, Jun Nie wrote:
>> On 2015a1'01ae??30ae?JPY 11:25, Heesub Shin wrote:
>>>
>>>
>>> On 01/30/2015 11:57 AM, Jun Nie wrote:
>>>> On 2015a1'01ae??30ae?JPY 10:36, Jun Nie wrote:
>>>>> Hi Marek & Arnd,
>>>>>
>>>>> Did you ever know issue that free CMA memory is high, but system is
>>>>> hungry for memory and page cache is very low? I am enabling CMA in
>>>>> Android on my board with 512MB memory and see FreeMem in /proc/meminfo
>>>>> increase a lot with CMA comparing the reservation solution on boot. But
>>>>> I find system is not borrowing memory from CMA pool when running 3dmark
>>>>> (high webkit workload at start). Because the FreeMem size is high, but
>>>>> cache size decreasing significantly to several MB during benchmark run,
>>>>> I suppose system is trying to reclaim memory from pagecache for new
>>>>> allocation. My question is that what API that page cache and webkit
>>>>> related functionality are using to allocate memory. Maybe page cache
>>>>> require memory that is not movable/reclaimable memory, where we may
>>>>> have
>>>>> optimization to go thru dma_alloc_xxx to borrow CMA memory? I suppose
>>>>> app level memory allocation shall be movable/reclaimable memory and can
>>>>> borrow from CMA pool, but not sure whether the flag match the
>>>>> movable/reclaimable memory and go thru the right path.
>>>>>
>>>>> Could you help share your experience/thoughts on this? Thanks!
>>>
>>> CC'ed linux-mm@kvack.org
>>>
>>> __zone_watermark_ok() assumes that free pages from CMA pageblock are not
>>> free when ALLOC_CMA is not set on alloc_flags. The main goal was to
>>> force core mm to keep some non-CMA always free and thus let kernel to
>>> allocate a few unmovable pages from any context (including atomic, irq,
>>> etc.). However, this behavior may cause excessive page reclamation as it
>>> is sometimes very hard to satisfy the high wmark + balance_gap with only
>>> non-CMA pages and reclaiming CMA pages does not help at all.
>> Seems it is tricky to tune it. Could you help share some experience on
>> this, how to change the parameters, what's pro/con? Thanks!
> 
> AFAIK, unfortunately there's no other way rather than reducing the 
> number of CMA pageblocks which are making anomalies. Selectively 
> ignoring CMA pages when we isolate pages from LRU could be an 
> alternative, but it has another side effect. I also want to know how to 
> handle this problem nicely.

Well maybe zone_balanced() could check watermarks with passing ALLOC_CMA in
alloc_flags instead of 0? This would mean that high watermark will be satisfied
for movable allocations, which pass ALLOC_CMA. That should fix your too-depleted
page cache problem, I think? But in that case it should probably also check low
watermark without ALLOC_CMA, to make sure unmovable/reclaimable allocations
won't stall.

There might however still be some side effects. IIRC unmovable allocations are
already treated badly due to CMA, and it could make it worse. And we should also
check if direct reclaim paths use watermark checking with proper alloc_flags and
classzone_idx. IIRC they don't always do, which can also result in mismatched
decisions on compaction.

But maybe this is all moot if the plan for moving CMA to a different zone works
out...

>>>
>>> It is observed that page cache pages are excessively reclaimed and
>>> entire system falls into thrashing even though the amount of free pages
>>> are much higer than the high wmark. In this case, majority of the free
>>> pages were from CMA page block (and about 30% pages in highmem zone were
>>> from CMA pageblock). Therefore, kswapd kept running and reclaiming too
>>> many pages. Although it is relatively rare and only observed on a
>>> specific workload, the device gets in an unresponsive state for a while
>>> (up to 10 secs), once it happens.
>>>
>> I am in this situation. kswapd is busy and most FreeMem is from CMA
>> because I have 192MB CMA memblock and most of them are free.
>>> regards,
>>> heesub
>>>
>>>>>
>>>>>
>>>>> B.R.
>>>>> Jun
>>>>
>>>> Add more people.
>>>>
>>>> _______________________________________________
>>>> linux-arm-kernel mailing list
>>>> linux-arm-kernel@lists.infradead.org
>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>
>> --
>> To unsubscribe, send a message with 'unsubscribe linux-mm' in
>> the body to majordomo@kvack.org.  For more info on Linux MM,
>> see: http://www.linux-mm.org/ .
>> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>>
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

CMA related memory questions

[Vlastimil Babka]

[CC some usual CMA suspects]

On 01/30/2015 06:14 AM, Heesub Shin wrote:
> 
> 
> On 01/30/2015 12:43 PM, Jun Nie wrote:
>> On 2015?01?30? 11:25, Heesub Shin wrote:
>>>
>>>
>>> On 01/30/2015 11:57 AM, Jun Nie wrote:
>>>> On 2015?01?30? 10:36, Jun Nie wrote:
>>>>> Hi Marek & Arnd,
>>>>>
>>>>> Did you ever know issue that free CMA memory is high, but system is
>>>>> hungry for memory and page cache is very low? I am enabling CMA in
>>>>> Android on my board with 512MB memory and see FreeMem in /proc/meminfo
>>>>> increase a lot with CMA comparing the reservation solution on boot. But
>>>>> I find system is not borrowing memory from CMA pool when running 3dmark
>>>>> (high webkit workload at start). Because the FreeMem size is high, but
>>>>> cache size decreasing significantly to several MB during benchmark run,
>>>>> I suppose system is trying to reclaim memory from pagecache for new
>>>>> allocation. My question is that what API that page cache and webkit
>>>>> related functionality are using to allocate memory. Maybe page cache
>>>>> require memory that is not movable/reclaimable memory, where we may
>>>>> have
>>>>> optimization to go thru dma_alloc_xxx to borrow CMA memory? I suppose
>>>>> app level memory allocation shall be movable/reclaimable memory and can
>>>>> borrow from CMA pool, but not sure whether the flag match the
>>>>> movable/reclaimable memory and go thru the right path.
>>>>>
>>>>> Could you help share your experience/thoughts on this? Thanks!
>>>
>>> CC'ed linux-mm at kvack.org
>>>
>>> __zone_watermark_ok() assumes that free pages from CMA pageblock are not
>>> free when ALLOC_CMA is not set on alloc_flags. The main goal was to
>>> force core mm to keep some non-CMA always free and thus let kernel to
>>> allocate a few unmovable pages from any context (including atomic, irq,
>>> etc.). However, this behavior may cause excessive page reclamation as it
>>> is sometimes very hard to satisfy the high wmark + balance_gap with only
>>> non-CMA pages and reclaiming CMA pages does not help at all.
>> Seems it is tricky to tune it. Could you help share some experience on
>> this, how to change the parameters, what's pro/con? Thanks!
> 
> AFAIK, unfortunately there's no other way rather than reducing the 
> number of CMA pageblocks which are making anomalies. Selectively 
> ignoring CMA pages when we isolate pages from LRU could be an 
> alternative, but it has another side effect. I also want to know how to 
> handle this problem nicely.

Well maybe zone_balanced() could check watermarks with passing ALLOC_CMA in
alloc_flags instead of 0? This would mean that high watermark will be satisfied
for movable allocations, which pass ALLOC_CMA. That should fix your too-depleted
page cache problem, I think? But in that case it should probably also check low
watermark without ALLOC_CMA, to make sure unmovable/reclaimable allocations
won't stall.

There might however still be some side effects. IIRC unmovable allocations are
already treated badly due to CMA, and it could make it worse. And we should also
check if direct reclaim paths use watermark checking with proper alloc_flags and
classzone_idx. IIRC they don't always do, which can also result in mismatched
decisions on compaction.

But maybe this is all moot if the plan for moving CMA to a different zone works
out...

>>>
>>> It is observed that page cache pages are excessively reclaimed and
>>> entire system falls into thrashing even though the amount of free pages
>>> are much higer than the high wmark. In this case, majority of the free
>>> pages were from CMA page block (and about 30% pages in highmem zone were
>>> from CMA pageblock). Therefore, kswapd kept running and reclaiming too
>>> many pages. Although it is relatively rare and only observed on a
>>> specific workload, the device gets in an unresponsive state for a while
>>> (up to 10 secs), once it happens.
>>>
>> I am in this situation. kswapd is busy and most FreeMem is from CMA
>> because I have 192MB CMA memblock and most of them are free.
>>> regards,
>>> heesub
>>>
>>>>>
>>>>>
>>>>> B.R.
>>>>> Jun
>>>>
>>>> Add more people.
>>>>
>>>> _______________________________________________
>>>> linux-arm-kernel mailing list
>>>> linux-arm-kernel at lists.infradead.org
>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>
>> --
>> To unsubscribe, send a message with 'unsubscribe linux-mm' in
>> the body to majordomo at kvack.org.  For more info on Linux MM,
>> see: http://www.linux-mm.org/ .
>> Don't email: <a href=mailto:"dont@kvack.org"> email at kvack.org </a>
>>
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo at kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email at kvack.org </a>
> 

Re: CMA related memory questions

[Hui Zhu]

On Fri, Jan 30, 2015 at 4:54 PM, Vlastimil Babka <vbabka@suse.cz> wrote:
> [CC some usual CMA suspects]
>
> On 01/30/2015 06:14 AM, Heesub Shin wrote:
>>
>>
>> On 01/30/2015 12:43 PM, Jun Nie wrote:
>>> On 2015年01月30日 11:25, Heesub Shin wrote:
>>>>
>>>>
>>>> On 01/30/2015 11:57 AM, Jun Nie wrote:
>>>>> On 2015年01月30日 10:36, Jun Nie wrote:
>>>>>> Hi Marek & Arnd,
>>>>>>
>>>>>> Did you ever know issue that free CMA memory is high, but system is
>>>>>> hungry for memory and page cache is very low? I am enabling CMA in
>>>>>> Android on my board with 512MB memory and see FreeMem in /proc/meminfo
>>>>>> increase a lot with CMA comparing the reservation solution on boot. But
>>>>>> I find system is not borrowing memory from CMA pool when running 3dmark
>>>>>> (high webkit workload at start). Because the FreeMem size is high, but
>>>>>> cache size decreasing significantly to several MB during benchmark run,
>>>>>> I suppose system is trying to reclaim memory from pagecache for new
>>>>>> allocation. My question is that what API that page cache and webkit
>>>>>> related functionality are using to allocate memory. Maybe page cache
>>>>>> require memory that is not movable/reclaimable memory, where we may
>>>>>> have
>>>>>> optimization to go thru dma_alloc_xxx to borrow CMA memory? I suppose
>>>>>> app level memory allocation shall be movable/reclaimable memory and can
>>>>>> borrow from CMA pool, but not sure whether the flag match the
>>>>>> movable/reclaimable memory and go thru the right path.
>>>>>>
>>>>>> Could you help share your experience/thoughts on this? Thanks!
>>>>
>>>> CC'ed linux-mm@kvack.org
>>>>
>>>> __zone_watermark_ok() assumes that free pages from CMA pageblock are not
>>>> free when ALLOC_CMA is not set on alloc_flags. The main goal was to
>>>> force core mm to keep some non-CMA always free and thus let kernel to
>>>> allocate a few unmovable pages from any context (including atomic, irq,
>>>> etc.). However, this behavior may cause excessive page reclamation as it
>>>> is sometimes very hard to satisfy the high wmark + balance_gap with only
>>>> non-CMA pages and reclaiming CMA pages does not help at all.
>>> Seems it is tricky to tune it. Could you help share some experience on
>>> this, how to change the parameters, what's pro/con? Thanks!
>>
>> AFAIK, unfortunately there's no other way rather than reducing the
>> number of CMA pageblocks which are making anomalies. Selectively
>> ignoring CMA pages when we isolate pages from LRU could be an
>> alternative, but it has another side effect. I also want to know how to
>> handle this problem nicely.
>
> Well maybe zone_balanced() could check watermarks with passing ALLOC_CMA in
> alloc_flags instead of 0? This would mean that high watermark will be satisfied
> for movable allocations, which pass ALLOC_CMA. That should fix your too-depleted
> page cache problem, I think? But in that case it should probably also check low
> watermark without ALLOC_CMA, to make sure unmovable/reclaimable allocations
> won't stall.
>
> There might however still be some side effects. IIRC unmovable allocations are
> already treated badly due to CMA, and it could make it worse. And we should also
> check if direct reclaim paths use watermark checking with proper alloc_flags and
> classzone_idx. IIRC they don't always do, which can also result in mismatched
> decisions on compaction.
>
> But maybe this is all moot if the plan for moving CMA to a different zone works
> out...

I did a lot of works around it to make current CMA code work OK with watermark.
It need too much work around it.  For example, my patch
https://lkml.org/lkml/2015/1/18/28 (It still has something wrong).
To make it work OK we need add more and more hook to page alloc code.

So I think special zone is the best way for that.

After we got CMA_ZONE, we can begin to handle the issue that how to
make it work OK with different board.

Thanks,
Hui


>
>>>>
>>>> It is observed that page cache pages are excessively reclaimed and
>>>> entire system falls into thrashing even though the amount of free pages
>>>> are much higer than the high wmark. In this case, majority of the free
>>>> pages were from CMA page block (and about 30% pages in highmem zone were
>>>> from CMA pageblock). Therefore, kswapd kept running and reclaiming too
>>>> many pages. Although it is relatively rare and only observed on a
>>>> specific workload, the device gets in an unresponsive state for a while
>>>> (up to 10 secs), once it happens.
>>>>
>>> I am in this situation. kswapd is busy and most FreeMem is from CMA
>>> because I have 192MB CMA memblock and most of them are free.
>>>> regards,
>>>> heesub
>>>>
>>>>>>
>>>>>>
>>>>>> B.R.
>>>>>> Jun
>>>>>
>>>>> Add more people.
>>>>>
>>>>> _______________________________________________
>>>>> linux-arm-kernel mailing list
>>>>> linux-arm-kernel@lists.infradead.org
>>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>>
>>> --
>>> To unsubscribe, send a message with 'unsubscribe linux-mm' in
>>> the body to majordomo@kvack.org.  For more info on Linux MM,
>>> see: http://www.linux-mm.org/ .
>>> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>>>
>>
>> --
>> To unsubscribe, send a message with 'unsubscribe linux-mm' in
>> the body to majordomo@kvack.org.  For more info on Linux MM,
>> see: http://www.linux-mm.org/ .
>> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>>
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

CMA related memory questions

[Hui Zhu]

On Fri, Jan 30, 2015 at 4:54 PM, Vlastimil Babka <vbabka@suse.cz> wrote:
> [CC some usual CMA suspects]
>
> On 01/30/2015 06:14 AM, Heesub Shin wrote:
>>
>>
>> On 01/30/2015 12:43 PM, Jun Nie wrote:
>>> On 2015?01?30? 11:25, Heesub Shin wrote:
>>>>
>>>>
>>>> On 01/30/2015 11:57 AM, Jun Nie wrote:
>>>>> On 2015?01?30? 10:36, Jun Nie wrote:
>>>>>> Hi Marek & Arnd,
>>>>>>
>>>>>> Did you ever know issue that free CMA memory is high, but system is
>>>>>> hungry for memory and page cache is very low? I am enabling CMA in
>>>>>> Android on my board with 512MB memory and see FreeMem in /proc/meminfo
>>>>>> increase a lot with CMA comparing the reservation solution on boot. But
>>>>>> I find system is not borrowing memory from CMA pool when running 3dmark
>>>>>> (high webkit workload at start). Because the FreeMem size is high, but
>>>>>> cache size decreasing significantly to several MB during benchmark run,
>>>>>> I suppose system is trying to reclaim memory from pagecache for new
>>>>>> allocation. My question is that what API that page cache and webkit
>>>>>> related functionality are using to allocate memory. Maybe page cache
>>>>>> require memory that is not movable/reclaimable memory, where we may
>>>>>> have
>>>>>> optimization to go thru dma_alloc_xxx to borrow CMA memory? I suppose
>>>>>> app level memory allocation shall be movable/reclaimable memory and can
>>>>>> borrow from CMA pool, but not sure whether the flag match the
>>>>>> movable/reclaimable memory and go thru the right path.
>>>>>>
>>>>>> Could you help share your experience/thoughts on this? Thanks!
>>>>
>>>> CC'ed linux-mm at kvack.org
>>>>
>>>> __zone_watermark_ok() assumes that free pages from CMA pageblock are not
>>>> free when ALLOC_CMA is not set on alloc_flags. The main goal was to
>>>> force core mm to keep some non-CMA always free and thus let kernel to
>>>> allocate a few unmovable pages from any context (including atomic, irq,
>>>> etc.). However, this behavior may cause excessive page reclamation as it
>>>> is sometimes very hard to satisfy the high wmark + balance_gap with only
>>>> non-CMA pages and reclaiming CMA pages does not help at all.
>>> Seems it is tricky to tune it. Could you help share some experience on
>>> this, how to change the parameters, what's pro/con? Thanks!
>>
>> AFAIK, unfortunately there's no other way rather than reducing the
>> number of CMA pageblocks which are making anomalies. Selectively
>> ignoring CMA pages when we isolate pages from LRU could be an
>> alternative, but it has another side effect. I also want to know how to
>> handle this problem nicely.
>
> Well maybe zone_balanced() could check watermarks with passing ALLOC_CMA in
> alloc_flags instead of 0? This would mean that high watermark will be satisfied
> for movable allocations, which pass ALLOC_CMA. That should fix your too-depleted
> page cache problem, I think? But in that case it should probably also check low
> watermark without ALLOC_CMA, to make sure unmovable/reclaimable allocations
> won't stall.
>
> There might however still be some side effects. IIRC unmovable allocations are
> already treated badly due to CMA, and it could make it worse. And we should also
> check if direct reclaim paths use watermark checking with proper alloc_flags and
> classzone_idx. IIRC they don't always do, which can also result in mismatched
> decisions on compaction.
>
> But maybe this is all moot if the plan for moving CMA to a different zone works
> out...

I did a lot of works around it to make current CMA code work OK with watermark.
It need too much work around it.  For example, my patch
https://lkml.org/lkml/2015/1/18/28 (It still has something wrong).
To make it work OK we need add more and more hook to page alloc code.

So I think special zone is the best way for that.

After we got CMA_ZONE, we can begin to handle the issue that how to
make it work OK with different board.

Thanks,
Hui


>
>>>>
>>>> It is observed that page cache pages are excessively reclaimed and
>>>> entire system falls into thrashing even though the amount of free pages
>>>> are much higer than the high wmark. In this case, majority of the free
>>>> pages were from CMA page block (and about 30% pages in highmem zone were
>>>> from CMA pageblock). Therefore, kswapd kept running and reclaiming too
>>>> many pages. Although it is relatively rare and only observed on a
>>>> specific workload, the device gets in an unresponsive state for a while
>>>> (up to 10 secs), once it happens.
>>>>
>>> I am in this situation. kswapd is busy and most FreeMem is from CMA
>>> because I have 192MB CMA memblock and most of them are free.
>>>> regards,
>>>> heesub
>>>>
>>>>>>
>>>>>>
>>>>>> B.R.
>>>>>> Jun
>>>>>
>>>>> Add more people.
>>>>>
>>>>> _______________________________________________
>>>>> linux-arm-kernel mailing list
>>>>> linux-arm-kernel at lists.infradead.org
>>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>>
>>> --
>>> To unsubscribe, send a message with 'unsubscribe linux-mm' in
>>> the body to majordomo at kvack.org.  For more info on Linux MM,
>>> see: http://www.linux-mm.org/ .
>>> Don't email: <a href=mailto:"dont@kvack.org"> email at kvack.org </a>
>>>
>>
>> --
>> To unsubscribe, send a message with 'unsubscribe linux-mm' in
>> the body to majordomo at kvack.org.  For more info on Linux MM,
>> see: http://www.linux-mm.org/ .
>> Don't email: <a href=mailto:"dont@kvack.org"> email at kvack.org </a>
>>
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo at kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email at kvack.org </a>

Re: CMA related memory questions

[Joonsoo Kim]

2015-01-30 12:25 GMT+09:00 Heesub Shin <heesub.shin@samsung.com>:
>
>
> On 01/30/2015 11:57 AM, Jun Nie wrote:
>>
>> On 2015年01月30日 10:36, Jun Nie wrote:
>>>
>>> Hi Marek & Arnd,
>>>
>>> Did you ever know issue that free CMA memory is high, but system is
>>> hungry for memory and page cache is very low? I am enabling CMA in
>>> Android on my board with 512MB memory and see FreeMem in /proc/meminfo
>>> increase a lot with CMA comparing the reservation solution on boot. But
>>> I find system is not borrowing memory from CMA pool when running 3dmark
>>> (high webkit workload at start). Because the FreeMem size is high, but
>>> cache size decreasing significantly to several MB during benchmark run,
>>> I suppose system is trying to reclaim memory from pagecache for new
>>> allocation. My question is that what API that page cache and webkit
>>> related functionality are using to allocate memory. Maybe page cache
>>> require memory that is not movable/reclaimable memory, where we may have
>>> optimization to go thru dma_alloc_xxx to borrow CMA memory? I suppose
>>> app level memory allocation shall be movable/reclaimable memory and can
>>> borrow from CMA pool, but not sure whether the flag match the
>>> movable/reclaimable memory and go thru the right path.

Hello,

Maybe, you experienced the problem what I tried to solve.
CMA freepage allocation logic in mainline doesn't work well now since
they are only allocated in fallback case. See below link for detailed
explanation.

https://lkml.org/lkml/2014/5/28/64

This problem can be solved by aggressive allocation approach in that link, but,
there are too many issues left in CMA. So, I'm trying to implement ZONE_CMA
now. Prototyping is nearly finished so I will send it soon.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

CMA related memory questions

[Joonsoo Kim]

2015-01-30 12:25 GMT+09:00 Heesub Shin <heesub.shin@samsung.com>:
>
>
> On 01/30/2015 11:57 AM, Jun Nie wrote:
>>
>> On 2015?01?30? 10:36, Jun Nie wrote:
>>>
>>> Hi Marek & Arnd,
>>>
>>> Did you ever know issue that free CMA memory is high, but system is
>>> hungry for memory and page cache is very low? I am enabling CMA in
>>> Android on my board with 512MB memory and see FreeMem in /proc/meminfo
>>> increase a lot with CMA comparing the reservation solution on boot. But
>>> I find system is not borrowing memory from CMA pool when running 3dmark
>>> (high webkit workload at start). Because the FreeMem size is high, but
>>> cache size decreasing significantly to several MB during benchmark run,
>>> I suppose system is trying to reclaim memory from pagecache for new
>>> allocation. My question is that what API that page cache and webkit
>>> related functionality are using to allocate memory. Maybe page cache
>>> require memory that is not movable/reclaimable memory, where we may have
>>> optimization to go thru dma_alloc_xxx to borrow CMA memory? I suppose
>>> app level memory allocation shall be movable/reclaimable memory and can
>>> borrow from CMA pool, but not sure whether the flag match the
>>> movable/reclaimable memory and go thru the right path.

Hello,

Maybe, you experienced the problem what I tried to solve.
CMA freepage allocation logic in mainline doesn't work well now since
they are only allocated in fallback case. See below link for detailed
explanation.

https://lkml.org/lkml/2014/5/28/64

This problem can be solved by aggressive allocation approach in that link, but,
there are too many issues left in CMA. So, I'm trying to implement ZONE_CMA
now. Prototyping is nearly finished so I will send it soon.

Thanks.