Re: OOM killer changes

[Ralf-Peter Rohbeck <Ralf-Peter.Rohbeck@quantum.com>]

On 02.08.2016 12:25, Ralf-Peter Rohbeck wrote:
> I can do that but it'll be later this week.
>
> Ralf-Peter
> On 08/02/2016 12:10 AM, Michal Hocko wrote:
>> On Mon 01-08-16 14:27:51, Ralf-Peter Rohbeck wrote:
>>> On 01.08.2016 14:14, Ralf-Peter Rohbeck wrote:
>>>> On 01.08.2016 13:26, Michal Hocko wrote:
>>>>>> sdc, sdd and sde each at max speed, with a little bit of garden
>>>>>> variety IO
>>>>>> on sda and sdb.
>>>>> So do I get it right that the majority of the IO is to those 
>>>>> slower USB
>>>>> disks?  If yes then does lowering the dirty_bytes to something 
>>>>> smaller
>>>>> help?
>>>> ADMIN
>>>> Yes, the vast majority.
>>>>
>>>> I set dirty_bytes to 128MiB and started a fairly IO and memory 
>>>> intensive
>>>> process and the OOM killer kicked in within a few seconds.
>>>>
>>>> Same with 16MiB dirty_bytes and 1MiB.
>>>>
>>>> Some additional IO load from my fast subsystem is enough:
>>>>
>>>> At 1MiB dirty_bytes,
>>>>
>>>> find /btrfs0/ -type f -exec md5sum {} \;
>>>>
>>>> was enough (where /btrfs0 is on a LVM2 LV and the PV is on sda.) It 
>>>> read
>>>> a few dozen files (random stuff with very mixed file sizes, none very
>>>> big) until the OOM killer kicked in.
>>>>
>>>> I'll try 4.6.
>>> With Debian 4.6.0.1 (4.6.4-1) it works: Writing to 3 USB drives and 
>>> running
>>> each of the 3 tests that triggered the OOM killer in parallel, with 
>>> default
>>> dirty settings.
>> Thanks for retesting! Now that it seems you are able to reproduce this,
>> could you do some experiments, please? First of all it would be great to
>> find out why we do not retry the compaction and whether it could make
>> some progress. The patch below will tell us the first part. Tracepoints
>> can tell us the other part. Vlastimil, could you recommend some which
>> would give us some hints without generating way too much output?
>> ---
>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>> index 8b3e1341b754..a10b29a918d4 100644
>> --- a/mm/page_alloc.c
>> +++ b/mm/page_alloc.c
>> @@ -3274,6 +3274,7 @@ should_compact_retry(struct alloc_context *ac, 
>> int order, int alloc_flags,
>>               *migrate_mode = MIGRATE_SYNC_LIGHT;
>>               return true;
>>           }
>> +        pr_info("XXX: compaction_failed\n");
>>           return false;
>>       }
>>   @@ -3283,8 +3284,12 @@ should_compact_retry(struct alloc_context 
>> *ac, int order, int alloc_flags,
>>        * But do not retry if the given zonelist is not suitable for
>>        * compaction.
>>        */
>> -    if (compaction_withdrawn(compact_result))
>> -        return compaction_zonelist_suitable(ac, order, alloc_flags);
>> +    if (compaction_withdrawn(compact_result)) {
>> +        int ret = compaction_zonelist_suitable(ac, order, alloc_flags);
>> +        if (!ret)
>> +            pr_info("XXX: no zone suitable for compaction\n");
>> +        return ret;
>> +    }
>>         /*
>>        * !costly requests are much more important than __GFP_REPEAT
>> @@ -3299,6 +3304,7 @@ should_compact_retry(struct alloc_context *ac, 
>> int order, int alloc_flags,
>>       if (compaction_retries <= max_retries)
>>           return true;
>>   +    pr_info("XXX: compaction retries fail after %d\n", 
>> compaction_retries);
>>       return false;
>>   }
>>   #else
>>
>
Took me a little longer than expected due to work. The failure wouldn't 
happen for a while and so I started a couple of scripts and let them 
run. When I checked today the server didn't respond on the network and 
sure enough it had killed everything. This is with 4.7.0 with the config 
based on Debian 4.7-rc7.

trace_pipe got a little big (5GB) so I uploaded the logs to 
https://filebin.net/box0wycfouvhl6sr/OOM_4.7.0.tar.bz2. before_btrfs is 
before the btrfs filesystems were mounted.
I did run a btrfs balance because it creates IO load and I needed to 
balance anyway. Maybe that's what caused it?

I'll make the changes requested by Michal and try again.

Thanks,
Ralf-Peter


----------------------------------------------------------------------
The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>