Re: Caching/buffers become useless after some time

[Vlastimil Babka <vbabka@suse.cz>]

On 11/2/18 2:50 PM, Marinko Catovic wrote:
> Am Fr., 2. Nov. 2018 um 14:13 Uhr schrieb Vlastimil Babka <vbabka@suse.cz>:
>>
>> On 11/2/18 1:41 PM, Marinko Catovic wrote:
>>>>>> any idea how to find out what that might be? I'd really have no idea,
>>>>>> I also wonder why this never was an issue with 3.x
>>>>>> find uses regex patterns, that's the only thing that may be unusual.
>>>>>
>>>>> The allocation tracepoint has the stack trace so that might help. This
>>>>
>>>> Well we already checked the mm_page_alloc traces and it seemed that only
>>>> THP allocations could be the culprit. But apparently defrag=defer made
>>>> no difference. I would still recommend it so we can see the effects on
>>>> the traces. And adding tracepoints
>>>> compaction/mm_compaction_try_to_compact_pages and
>>>> compaction/mm_compaction_suitable as I suggested should show which
>>>> high-order allocations actually invoke the compaction.
>>>
>>> Anything in particular I should do to figure this out?
>>
>> Setup the same monitoring as before, but with two additional tracepoints
>> (echo 1 > .../enable) and once the problem appears, provide the tracing
>> output.
> 
> I think I'll need more details about that setup  :)

It's like what you already did based on suggestion from Michal Hocko:

# mount -t tracefs none /debug/trace/
# echo stacktrace > /debug/trace/trace_options
# echo "order>0" > /debug/trace/events/kmem/mm_page_alloc/filter
# echo 1 > /debug/trace/events/kmem/mm_page_alloc/enable
# echo 1 > /debug/trace/events/compaction/mm_compaction_try_to_compact_pages
# echo 1 > /debug/trace/events/compaction/mm_compaction_suitable
# cat /debug/trace/trace_pipe | gzip > /path/to/trace_pipe.txt.gz

And later this to disable tracing.
# echo 0 > /debug/trace/events/kmem/mm_page_alloc/enable

> also, do you want the tracing output every 5sec or just once when it
> is around the worst case? what files exactly?

Collect vmstat periodically every 5 secs as you already did. Tracing is
continuous and results in the single trace_pipe.txt.gz file.
The trace should cover at least some time while you're experiencing the
too much free memory/too little pagecache phase. Might be enough to
enable the collection only after you detect the situation, and before
you e.g. drop caches to restore the system.

To remove THP allocations from the picture, it would be nice if the
system was configured with:
echo defer > /sys/kernel/mm/transparent_hugepage/defrag

Again you can do that only after detecting the problematic situation,
before starting to collect trace.