This time the OOM killer hit much quicker. No btrfs balance, just 
compiling the kernel with the new change did it.
Much smaller logs so I'm attaching them.

Ralf-Peter
On 15.08.2016 08:01, Michal Hocko wrote:
> On Mon 15-08-16 11:16:36, Vlastimil Babka wrote:
>> On 08/15/2016 06:48 AM, Ralf-Peter Rohbeck wrote:
>>> On 02.08.2016 12:25, Ralf-Peter Rohbeck wrote:
>>> Took me a little longer than expected due to work. The failure wouldn't
>>> happen for a while and so I started a couple of scripts and let them
>>> run. When I checked today the server didn't respond on the network and
>>> sure enough it had killed everything. This is with 4.7.0 with the config
>>> based on Debian 4.7-rc7.
>>>
>>> trace_pipe got a little big (5GB) so I uploaded the logs to
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__filebin.net_box0wycfouvhl6sr_OOM-5F4.7.0.tar.bz2&d=DQIBAg&c=8S5idjlO_n28Ko3lg6lskTMwneSC-WqZ5EBTEEvDlkg&r=yGQdEpZknbtYvR0TyhkCGu-ifLklIvXIf740poRFltQ&m=TBVC4CIIUzJlmpDNapp31jIbz3Gy1M-aQ9jhrv0U56I&s=ozhbhqcuwlWiU1Cd8PZGl5-CC69-m-sNUitSYI2ry1Y&e= . before_btrfs is
>>> before the btrfs filesystems were mounted.
>>> I did run a btrfs balance because it creates IO load and I needed to
>>> balance anyway. Maybe that's what caused it?
>> pgmigrate_success        46738962
>> pgmigrate_fail          135649772
>> compact_migrate_scanned 309726659
>> compact_free_scanned   9715615169
>> compact_isolated        229689596
>> compact_stall 4777
>> compact_fail 3068
>> compact_success 1709
>> compact_daemon_wake 207834
>>
>> The migration failures are quite enormous. Very quick analysis of the
>> trace seems to confirm that these are mostly "real", as opposed to result
>> of failure to isolate free pages for migration targets, although the free
>> scanner spent a lot of time:
>>
>>> grep "nr_failed=32" -B1 trace_pipe.log | grep isolate_freepages.*nr_taken=0 | wc -l
>> 3246
>>
>> So is it one of the cases where fs is unable to migrate dirty/writeback pages?
> It smells that way. Now we should find out why and what can we do about
> that. I suspect that try_to_release_page is not able to release the page
> for migration. Btrfs doesn't seem to have migratepage for page cache
> pages so it should go via fallback_migrate_page.
>
> The following diff should tell us whether this is really the case. Just
> open trace_pipe and see whether this path really triggered.
> ---
> diff --git a/mm/migrate.c b/mm/migrate.c
> index 72c09dea6526..120e2e5fcbea 100644
> --- a/mm/migrate.c
> +++ b/mm/migrate.c
> @@ -729,8 +729,10 @@ static int fallback_migrate_page(struct address_space *mapping,
>   	 * We must have no buffers or drop them.
>   	 */
>   	if (page_has_private(page) &&
> -	    !try_to_release_page(page, GFP_KERNEL))
> +	    !try_to_release_page(page, GFP_KERNEL)) {
> +		trace_printk("try_to_release_page failed for a_ops:%pS\n", page->a_ops);
>   		return -EAGAIN;
> +	}
>   
>   	return migrate_page(mapping, newpage, page, mode);
>   }


----------------------------------------------------------------------
The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt.