Re: [Question] Should direct reclaim time be bounded?

[Mike Kravetz <mike.kravetz@oracle.com>]

On 4/23/19 12:19 AM, Michal Hocko wrote:
> On Mon 22-04-19 21:07:28, Mike Kravetz wrote:
>> In our distro kernel, I am thinking about making allocations try "less hard"
>> on nodes where we start to see failures.  less hard == NORETRY/NORECLAIM.
>> I was going to try something like this on an upstream kernel when I noticed
>> that it seems like direct reclaim may never end/exit.  It 'may' exit, but I
>> instrumented __alloc_pages_slowpath() and saw it take well over an hour
>> before I 'tricked' it into exiting.
>>
>> [ 5916.248341] hpage_slow_alloc: jiffies 5295742  tries 2   node 0 success
>> [ 5916.249271]                   reclaim 5295741  compact 1
> 
> This is unexpected though. What does tries mean? Number of reclaim
> attempts? If yes could you enable tracing to see what takes so long in
> the reclaim path?

tries is the number of times we pass the 'retry:' label in
__alloc_pages_slowpath.  In this specific case, I am pretty sure all that
time is in one call to __alloc_pages_direct_reclaim.  My 'trick' to make this
succeed was to "echo 0 > nr_hugepages" in another shell.

>> This is where it stalled after "echo 4096 > nr_hugepages" on a little VM
>> with 8GB total memory.
>>
>> I have not started looking at the direct reclaim code to see exactly where
>> we may be stuck, or trying really hard.  My question is, "Is this expected
>> or should direct reclaim be somewhat bounded?"  With __alloc_pages_slowpath
>> getting 'stuck' in direct reclaim, the documented behavior for huge page
>> allocation is not going to happen.
> 
> Well, our "how hard to try for hugetlb pages" is quite arbitrary. We
> used to rety as long as at least order worth of pages have been
> reclaimed but that didn't make any sense since the lumpy reclaim was
> gone.

Yes, that is what I am seeing in our older distro kernel and I can at least
deal with that.

>       So the semantic has change to reclaim&compact as long as there is
> some progress. From what I understad above it seems that you are not
> thrashing and calling reclaim again and again but rather one reclaim
> round takes ages.

Correct

> That being said, I do not think __GFP_RETRY_MAYFAIL is wrong here. It
> looks like there is something wrong in the reclaim going on.

Ok, I will start digging into that.  Just wanted to make sure before I got
into it too deep.

BTW - This is very easy to reproduce.  Just try to allocate more huge pages
than will fit into memory.  I see this 'reclaim taking forever' behavior on
v5.1-rc5-mmotm-2019-04-19-14-53.  Looks like it was there in v5.0 as well.
-- 
Mike Kravetz