From: Robert Kudyba <rkudyba@fordham.edu>
To: Michal Hocko <mhocko@kernel.org>
Cc: linux-kernel@vger.kernel.org
Subject: Re: rsync: page allocation stalls in kernel 4.9.10 to a VessRAID NAS
Date: Tue, 28 Feb 2017 14:32:18 -0500 [thread overview]
Message-ID: <FD013725-A937-45AD-93AD-D79AF7E381BC@fordham.edu> (raw)
In-Reply-To: <20170228165638.GA27726@dhcp22.suse.cz>
> On Feb 28, 2017, at 11:56 AM, Michal Hocko <mhocko@kernel.org> wrote:
>
> On Tue 28-02-17 11:19:33, Robert Kudyba wrote:
>>
>>> On Feb 28, 2017, at 10:15 AM, Michal Hocko <mhocko@kernel.org> wrote:
>>>
>>> On Tue 28-02-17 09:59:35, Robert Kudyba wrote:
>>>>
>>>>> On Feb 28, 2017, at 9:40 AM, Michal Hocko <mhocko@kernel.org> wrote:
>>>>>
>>>>> On Tue 28-02-17 09:33:49, Robert Kudyba wrote:
>>>>>>
>>>>>>> On Feb 28, 2017, at 9:15 AM, Michal Hocko <mhocko@kernel.org> wrote:
>>>>>>> and this one is hitting the min watermark while there is not really
>>>>>>> much to reclaim. Only the page cache which might be pinned and not
>>>>>>> reclaimable from this context because this is GFP_NOFS request. It is
>>>>>>> not all that surprising the reclaim context fights to get some memory.
>>>>>>> There is a huge amount of the reclaimable slab which probably just makes
>>>>>>> a slow progress.
>>>>>>>
>>>>>>> That is not something completely surprsing on 32b system I am afraid.
>>>>>>>
>>>>>>> Btw. is the stall repeating with the increased time or it gets resolved
>>>>>>> eventually?
>>>>>>
>>>>>> Yes and if you mean by repeating it’s not only affecting rsync but
>>>>>> you can see just now automount and NetworkManager get these page
>>>>>> allocation stalls and kswapd0 is getting heavy CPU load, are there any
>>>>>> other settings I can adjust?
>>>>>
>>>>> None that I am aware of. You might want to talk to FS guys, maybe they
>>>>> can figure out who is pinning file pages so that they cannot be
>>>>> reclaimed. They do not seem to be dirty or under writeback. It would be
>>>>> also interesting to see whether that is a regression. The warning is
>>>>> relatively new so you might have had this problem before just haven't
>>>>> noticed it.
>>>>
>>>> We have been getting out of memory errors for a while but those seem
>>>> to have gone away.
>>>
>>> this sounds suspicious. Are you really sure that this is a new problem?
>>> Btw. is there any reason to use 32b kernel at all? It will always suffer
>>> from a really small lowmem…
>>
>> No this has been a problem for a while. Not sure if this server can
>> handle 64b it’s a bit old.
>
> Ok, this is unfortunate. There is usually not much interest to fixing
> 32b issues which are inherent to the used memory model and which are not
> regressions which would be fixable, I am afraid.
>
>>>> We did just replace the controller in the VessRAID
>>>> as there were some timeouts observed and multiple login/logout
>>>> attempts.
>>>>
>>>> By FS guys do you mean the linux-fsdevel or linux-fsf list?
>>>
>>> yeah linux-fsdevel. No idea what linux-fsf is. It would be great if you
>>> could collect some tracepoints before reporting the issue. At least
>>> those in events/vmscan/*.
>>
>> Will do here’s a perf report:
>
> this will not tell us much. Tracepoints have much better chance to tell
> us how reclaim is progressing.
I have SystemTap configured are there any scripts in the SystemTap_Beginners_Guide.pdf that I can run to help? Sorry I’m brand new to tracepoints.
I do see these “vmscan” from this command:
stap -L 'kernel.trace("*")'|sort
kernel.trace("vmscan:mm_shrink_slab_end") $shr:struct shrinker* $nid:int $shrinker_retval:int $unused_scan_cnt:long int $new_scan_cnt:long int $total_scan:long int
kernel.trace("vmscan:mm_shrink_slab_start") $shr:struct shrinker* $sc:struct shrink_control* $nr_objects_to_shrink:long int $pgs_scanned:long unsigned int $lru_pgs:long unsigned int $cache_items:long unsigned int $delta:long long unsigned int $total_scan:long unsigned int
kernel.trace("vmscan:mm_vmscan_direct_reclaim_begin") $order:int $may_writepage:int $gfp_flags:gfp_t $classzone_idx:int
kernel.trace("vmscan:mm_vmscan_direct_reclaim_end") $nr_reclaimed:long unsigned int
kernel.trace("vmscan:mm_vmscan_kswapd_sleep") $nid:int
kernel.trace("vmscan:mm_vmscan_kswapd_wake") $nid:int $zid:int $order:int
kernel.trace("vmscan:mm_vmscan_lru_isolate") $classzone_idx:int $order:int $nr_requested:long unsigned int $nr_scanned:long unsigned int $nr_taken:long unsigned int $isolate_mode:isolate_mode_t $file:int
kernel.trace("vmscan:mm_vmscan_lru_shrink_inactive") $nid:int $nr_scanned:long unsigned int $nr_reclaimed:long unsigned int $priority:int $file:int
kernel.trace("vmscan:mm_vmscan_memcg_isolate") $classzone_idx:int $order:int $nr_requested:long unsigned int $nr_scanned:long unsigned int $nr_taken:long unsigned int $isolate_mode:isolate_mode_t $file:int
kernel.trace("vmscan:mm_vmscan_memcg_reclaim_begin") $order:int $may_writepage:int $gfp_flags:gfp_t $classzone_idx:int
kernel.trace("vmscan:mm_vmscan_memcg_reclaim_end") $nr_reclaimed:long unsigned int
kernel.trace("vmscan:mm_vmscan_memcg_softlimit_reclaim_begin") $order:int $may_writepage:int $gfp_flags:gfp_t $classzone_idx:int
kernel.trace("vmscan:mm_vmscan_memcg_softlimit_reclaim_end") $nr_reclaimed:long unsigned int
kernel.trace("vmscan:mm_vmscan_wakeup_kswapd") $nid:int $zid:int $order:int
kernel.trace("vmscan:mm_vmscan_writepage") $page:struct page*
next prev parent reply other threads:[~2017-02-28 19:34 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-02-27 21:36 rsync: page allocation stalls in kernel 4.9.10 to a VessRAID NAS Robert Kudyba
2017-02-28 14:15 ` Michal Hocko
[not found] ` <40F07E96-7468-4355-B8EA-4B42F575ACAB@fordham.edu>
2017-02-28 14:40 ` Michal Hocko
[not found] ` <3E4C7821-A93D-4956-A0E0-730BEC67C9F0@fordham.edu>
2017-02-28 15:15 ` Michal Hocko
2017-02-28 16:19 ` Robert Kudyba
2017-02-28 16:56 ` Michal Hocko
2017-02-28 19:32 ` Robert Kudyba [this message]
2017-03-01 8:06 ` Michal Hocko
[not found] ` <F77DA4E6-EF9B-427D-8FE9-9FB940A9B009@fordham.edu>
2017-03-01 17:36 ` Michal Hocko
[not found] ` <D2C3C9EB-7E99-4420-887A-13526002E267@fordham.edu>
2017-03-01 19:19 ` Michal Hocko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=FD013725-A937-45AD-93AD-D79AF7E381BC@fordham.edu \
--to=rkudyba@fordham.edu \
--cc=linux-kernel@vger.kernel.org \
--cc=mhocko@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).