From: Robert Kudyba <rkudyba@fordham.edu>
To: Michal Hocko <mhocko@kernel.org>
Cc: linux-kernel@vger.kernel.org
Subject: Re: rsync: page allocation stalls in kernel 4.9.10 to a VessRAID NAS
Date: Tue, 28 Feb 2017 11:19:33 -0500 [thread overview]
Message-ID: <63A3D887-EEDA-46D2-AB59-D5955FC3D23D@fordham.edu> (raw)
In-Reply-To: <20170228151535.GE26792@dhcp22.suse.cz>
> On Feb 28, 2017, at 10:15 AM, Michal Hocko <mhocko@kernel.org> wrote:
>
> On Tue 28-02-17 09:59:35, Robert Kudyba wrote:
>>
>>> On Feb 28, 2017, at 9:40 AM, Michal Hocko <mhocko@kernel.org> wrote:
>>>
>>> On Tue 28-02-17 09:33:49, Robert Kudyba wrote:
>>>>
>>>>> On Feb 28, 2017, at 9:15 AM, Michal Hocko <mhocko@kernel.org> wrote:
>>>>> and this one is hitting the min watermark while there is not really
>>>>> much to reclaim. Only the page cache which might be pinned and not
>>>>> reclaimable from this context because this is GFP_NOFS request. It is
>>>>> not all that surprising the reclaim context fights to get some memory.
>>>>> There is a huge amount of the reclaimable slab which probably just makes
>>>>> a slow progress.
>>>>>
>>>>> That is not something completely surprsing on 32b system I am afraid.
>>>>>
>>>>> Btw. is the stall repeating with the increased time or it gets resolved
>>>>> eventually?
>>>>
>>>> Yes and if you mean by repeating it’s not only affecting rsync but
>>>> you can see just now automount and NetworkManager get these page
>>>> allocation stalls and kswapd0 is getting heavy CPU load, are there any
>>>> other settings I can adjust?
>>>
>>> None that I am aware of. You might want to talk to FS guys, maybe they
>>> can figure out who is pinning file pages so that they cannot be
>>> reclaimed. They do not seem to be dirty or under writeback. It would be
>>> also interesting to see whether that is a regression. The warning is
>>> relatively new so you might have had this problem before just haven't
>>> noticed it.
>>
>> We have been getting out of memory errors for a while but those seem
>> to have gone away.
>
> this sounds suspicious. Are you really sure that this is a new problem?
> Btw. is there any reason to use 32b kernel at all? It will always suffer
> from a really small lowmem…
No this has been a problem for a while. Not sure if this server can handle 64b it’s a bit old.
>
>> We did just replace the controller in the VessRAID
>> as there were some timeouts observed and multiple login/logout
>> attempts.
>>
>> By FS guys do you mean the linux-fsdevel or linux-fsf list?
>
> yeah linux-fsdevel. No idea what linux-fsf is. It would be great if you
> could collect some tracepoints before reporting the issue. At least
> those in events/vmscan/*.
Will do here’s a perf report:
Children Self Command Shared Object Symbol ◆
63.54% 0.06% swapper [kernel.kallsyms] [k] cpu_startup_entry
62.07% 0.01% swapper [kernel.kallsyms] [k] default_idle_call
62.06% 0.01% swapper [kernel.kallsyms] [k] arch_cpu_idle
54.73% 54.73% swapper [kernel.kallsyms] [k] mwait_idle
45.22% 0.00% swapper [kernel.kallsyms] [k] start_secondary
18.35% 0.00% swapper [kernel.kallsyms] [k] rest_init
18.35% 0.00% swapper [kernel.kallsyms] [k] start_kernel
18.35% 0.00% swapper [kernel.kallsyms] [k] i386_start_kernel
13.97% 0.01% rsync [vdso] [.] __kernel_vsyscall
13.93% 0.01% rsync [kernel.kallsyms] [k] sysenter_past_esp
13.92% 0.02% rsync [kernel.kallsyms] [k] do_fast_syscall_32
8.58% 0.01% rsync [kernel.kallsyms] [k] sys_write
8.56% 0.01% rsync [kernel.kallsyms] [k] vfs_write
8.55% 0.01% rsync [kernel.kallsyms] [k] __vfs_write
8.55% 0.00% rsync [kernel.kallsyms] [k] ext4_file_write_iter
8.54% 0.00% rsync [kernel.kallsyms] [k] __generic_file_write_iter
8.36% 0.04% rsync [kernel.kallsyms] [k] generic_perform_write
6.96% 0.00% swapper [kernel.kallsyms] [k] irq_exit
6.81% 0.01% swapper [kernel.kallsyms] [k] do_softirq_own_stack
6.80% 0.01% swapper [kernel.kallsyms] [k] __do_softirq
6.30% 0.00% swapper [kernel.kallsyms] [k] do_IRQ
6.30% 0.00% swapper [kernel.kallsyms] [k] common_interrupt
5.76% 0.01% swapper [kernel.kallsyms] [k] net_rx_action
5.75% 0.10% swapper [kernel.kallsyms] [k] e1000_clean
5.33% 0.01% rsync [kernel.kallsyms] [k] sys_read
5.30% 0.00% rsync [kernel.kallsyms] [k] vfs_read
5.28% 0.02% rsync [kernel.kallsyms] [k] __vfs_read
5.23% 0.10% rsync [kernel.kallsyms] [k] generic_file_read_iter
5.22% 0.00% swapper [kernel.kallsyms] [k] netif_receive_skb_internal
5.21% 0.01% swapper [kernel.kallsyms] [k] __netif_receive_skb
5.20% 0.01% swapper [kernel.kallsyms] [k] __netif_receive_skb_core
5.19% 0.02% swapper [kernel.kallsyms] [k] ip_rcv
4.93% 0.01% swapper [kernel.kallsyms] [k] ip_rcv_finish
4.90% 0.01% swapper [kernel.kallsyms] [k] ip_local_deliver
4.86% 0.05% swapper [kernel.kallsyms] [k] e1000_clean_rx_irq
4.80% 0.00% swapper [kernel.kallsyms] [k] ip_local_deliver_finish
4.80% 0.01% swapper [kernel.kallsyms] [k] tcp_v4_rcv
4.73% 0.01% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv
next prev parent reply other threads:[~2017-02-28 16:50 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-02-27 21:36 rsync: page allocation stalls in kernel 4.9.10 to a VessRAID NAS Robert Kudyba
2017-02-28 14:15 ` Michal Hocko
[not found] ` <40F07E96-7468-4355-B8EA-4B42F575ACAB@fordham.edu>
2017-02-28 14:40 ` Michal Hocko
[not found] ` <3E4C7821-A93D-4956-A0E0-730BEC67C9F0@fordham.edu>
2017-02-28 15:15 ` Michal Hocko
2017-02-28 16:19 ` Robert Kudyba [this message]
2017-02-28 16:56 ` Michal Hocko
2017-02-28 19:32 ` Robert Kudyba
2017-03-01 8:06 ` Michal Hocko
[not found] ` <F77DA4E6-EF9B-427D-8FE9-9FB940A9B009@fordham.edu>
2017-03-01 17:36 ` Michal Hocko
[not found] ` <D2C3C9EB-7E99-4420-887A-13526002E267@fordham.edu>
2017-03-01 19:19 ` Michal Hocko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=63A3D887-EEDA-46D2-AB59-D5955FC3D23D@fordham.edu \
--to=rkudyba@fordham.edu \
--cc=linux-kernel@vger.kernel.org \
--cc=mhocko@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).