From: Daniel Dao <dqminh@cloudflare.com>
To: Shakeel Butt <shakeelb@google.com>
Cc: "Ivan Babrou" <ivan@cloudflare.com>,
kernel-team <kernel-team@cloudflare.com>,
"Linux MM" <linux-mm@kvack.org>,
"Johannes Weiner" <hannes@cmpxchg.org>,
"Roman Gushchin" <guro@fb.com>, "Feng Tang" <feng.tang@intel.com>,
"Michal Hocko" <mhocko@kernel.org>,
"Hillf Danton" <hdanton@sina.com>,
"Michal Koutný" <mkoutny@suse.com>,
"Andrew Morton" <akpm@linux-foundation.org>,
"Linus Torvalds" <torvalds@linux-foundation.org>
Subject: Re: Regression in workingset_refault latency on 5.15
Date: Wed, 23 Feb 2022 17:07:08 +0000 [thread overview]
Message-ID: <CA+wXwBTXBTEBiYmaWGK=Oct_X5SctyaWTEJ0orCzWFoobexbHg@mail.gmail.com> (raw)
In-Reply-To: <CALvZod76u0qXBzrCZ16j9xTVkPPvqyw9+4s2K67bVwB1fQHnPw@mail.gmail.com>
On Wed, Feb 23, 2022 at 4:00 PM Shakeel Butt <shakeelb@google.com> wrote:
> > Can you share a bit more detail on your hardware configuration (num of
> > cpus) and if possible the flamegraph?
> >
We have a mix of 96 and 128 cpus. I'm not yet sure if it's possible to share
the flamegraphs. We may have to come back to that later if necessary.
>
> Also if you can reproduce the issue, can you try the patch at
> https://lore.kernel.org/all/20210929235936.2859271-1-shakeelb@google.com/
> ?
We can give it a try. I also wrote a bpftrace script to get the kernel
stack when we
encounter slow mem_cgroup_flush_stats ( with 10ms as threshold )
kprobe:mem_cgroup_flush_stats
{
@start[tid] = nsecs;
@stack[tid] = kstack;
}
kretprobe:mem_cgroup_flush_stats
/@start[tid]/
{
$usecs = (nsecs - @start[tid]) / 1000;
if ($usecs >= 10000) {
printf("mem_cgroup_flush_stats: %d us\n", $usecs);
printf("stack: %s\n", @stack[tid]);
}
delete(@start[tid]);
delete(@stack[tid]);
}
END
{
clear(@start);
clear(@stack);
}
Running it on a production node yields output like
mem_cgroup_flush_stats: 10697 us
stack:
mem_cgroup_flush_stats+1
workingset_refault+296
add_to_page_cache_lru+159
page_cache_ra_unbounded+340
force_page_cache_ra+226
filemap_get_pages+233
filemap_read+164
xfs_file_buffered_read+152
xfs_file_read_iter+106
new_sync_read+277
vfs_read+242
__x64_sys_pread64+137
do_syscall_64+56
entry_SYSCALL_64_after_hwframe+68
I think the addition of many milliseconds on workingset_refault is too high.
next prev parent reply other threads:[~2022-02-23 17:07 UTC|newest]
Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-02-23 13:51 Regression in workingset_refault latency on 5.15 Daniel Dao
2022-02-23 15:57 ` Shakeel Butt
2022-02-23 16:00 ` Shakeel Butt
2022-02-23 17:07 ` Daniel Dao [this message]
2022-02-23 17:36 ` Shakeel Butt
2022-02-23 19:28 ` Ivan Babrou
2022-02-23 20:28 ` Shakeel Butt
2022-02-23 21:16 ` Ivan Babrou
2022-02-24 14:46 ` Daniel Dao
2022-02-24 16:58 ` Shakeel Butt
2022-02-24 17:34 ` Daniel Dao
2022-02-24 18:00 ` Shakeel Butt
2022-02-24 18:52 ` Shakeel Butt
2022-02-25 10:23 ` Daniel Dao
2022-02-25 17:08 ` Ivan Babrou
2022-02-25 17:22 ` Shakeel Butt
2022-02-25 18:03 ` Michal Koutný
2022-02-25 18:08 ` Ivan Babrou
2022-02-28 23:09 ` Shakeel Butt
2022-02-28 23:34 ` Ivan Babrou
2022-02-28 23:43 ` Shakeel Butt
2022-03-02 0:48 ` Ivan Babrou
2022-03-02 2:50 ` Shakeel Butt
2022-03-02 3:40 ` Ivan Babrou
2022-03-02 22:33 ` Ivan Babrou
2022-03-03 2:32 ` Shakeel Butt
2022-03-03 2:35 ` Shakeel Butt
2022-03-04 0:21 ` Ivan Babrou
2022-03-04 1:05 ` Shakeel Butt
2022-03-04 1:12 ` Ivan Babrou
2022-03-02 11:49 ` Frank Hofmann
2022-03-02 15:52 ` Shakeel Butt
2022-03-02 10:08 ` Michal Koutný
2022-03-02 15:53 ` Shakeel Butt
2022-03-02 17:28 ` Ivan Babrou
2022-02-24 9:22 ` Thorsten Leemhuis
2022-04-11 10:17 ` Regression in workingset_refault latency on 5.15 #forregzbot Thorsten Leemhuis
2022-05-16 12:51 ` Thorsten Leemhuis
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CA+wXwBTXBTEBiYmaWGK=Oct_X5SctyaWTEJ0orCzWFoobexbHg@mail.gmail.com' \
--to=dqminh@cloudflare.com \
--cc=akpm@linux-foundation.org \
--cc=feng.tang@intel.com \
--cc=guro@fb.com \
--cc=hannes@cmpxchg.org \
--cc=hdanton@sina.com \
--cc=ivan@cloudflare.com \
--cc=kernel-team@cloudflare.com \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=mkoutny@suse.com \
--cc=shakeelb@google.com \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.