From: Justin Piszcz <jpiszcz@lucidpixels.com>
To: "Kirill A. Shutemov" <kirill@shutemov.name>
Cc: LKML <linux-kernel@vger.kernel.org>
Subject: Re: 5.1 kernel: khugepaged stuck at 100%
Date: Thu, 16 May 2019 10:17:51 -0400 [thread overview]
Message-ID: <CAO9zADyq44Sn0kYZBC55C0ykpHaASzp+27K3BofbkEniK6af-w@mail.gmail.com> (raw)
In-Reply-To: <CAO9zADww2v2ckHsNDwRgiyMr9b3JH1xOOSiRJ0Uh2XZT5c=MEQ@mail.gmail.com>
On Thu, May 16, 2019 at 10:14 AM Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>
> On Tue, May 14, 2019 at 9:16 AM Kirill A. Shutemov <kirill@shutemov.name> wrote:
>
> > Could you check what khugepaged doing?
> >
> > cat /proc/$(pidof khugepaged)/stack
>
> It is doing it again, 10:12am - 2019-05-16
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 77 root 39 19 0 0 0 R 100.0 0.0 92:06.94 khugepaged
>
> Kernel: 5.1.2
>
> $ sudo cat /proc/$(pidof khugepaged)/stack
> [<0>] 0xffffffffffffffff
>
> $ perf top
>
> PerfTop: 3716 irqs/sec kernel:92.9% exact: 99.1% lost: 68/68
> drop: 0/0 [4000Hz cycles], (all, 12 CPUs)
> -------------------------------------------------------------------------------
>
> 47.53% [kernel] [k] compaction_alloc
> 38.88% [kernel] [k] __pageblock_pfn_to_page
> 6.68% [kernel] [k] nmi
> 0.58% [kernel] [k] __list_del_entry_valid
> 0.48% [kernel] [k] format_decode
> 0.39% [kernel] [k] __rb_insert_augmented
> 0.25% libdbus-1.so.3.19.9 [.] _dbus_string_hex_decode
> 0.24% [kernel] [k] entry_SYSCALL_64_after_hwframe
> 0.20% perf [.] rb_next
> 0.19% perf [.] __symbols__insert
Incase this is asked for later:
# find . -maxdepth 1 -type f | while read f; do echo "$f
---------------------"; cat $f; echo " "; done
./environ ---------------------
./auxv ---------------------
./status ---------------------
Name: khugepaged
Umask: 0000
State: R (running)
Tgid: 77
Ngid: 0
Pid: 77
PPid: 2
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 64
Groups:
NStgid: 77
NSpid: 77
NSpgid: 0
NSsid: 0
Threads: 1
SigQ: 1/257125
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: ffffffffffffffff
SigCgt: 0000000000000000
CapInh: 0000000000000000
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
NoNewPrivs: 0
Seccomp: 0
Speculation_Store_Bypass: thread vulnerable
Cpus_allowed: fff
Cpus_allowed_list: 0-11
Mems_allowed: 1
Mems_allowed_list: 0
voluntary_ctxt_switches: 7728
nonvoluntary_ctxt_switches: 94662
./personality ---------------------
00000000
./limits ---------------------
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 257125 257125 processes
Max open files 1024 4096 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 257125 257125 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
./sched ---------------------
khugepaged (77, #threads: 1)
-------------------------------------------------------------------
se.exec_start : 84565632.684623
se.vruntime : 30287808.462012
se.sum_exec_runtime : 5808238.978150
se.nr_migrations : 946
nr_switches : 102390
nr_voluntary_switches : 7728
nr_involuntary_switches : 94662
se.load.weight : 15360
se.runnable_weight : 15360
se.avg.load_sum : 47454
se.avg.runnable_load_sum : 47454
se.avg.util_sum : 48835831
se.avg.load_avg : 14
se.avg.runnable_load_avg : 14
se.avg.util_avg : 1024
se.avg.last_update_time : 84565632684032
se.avg.util_est.ewma : 25
se.avg.util_est.enqueued : 16
policy : 0
prio : 139
clock-delta : 11
./comm ---------------------
khugepaged
./syscall ---------------------
running
./cmdline ---------------------
./stat ---------------------
77 (khugepaged) R 2 0 0 0 -1 27265088 0 0 0 0 0 580824 0 0 39 19 1 0
725 0 0 18446744073709551615 0 0 0 0 0 0 0 2147483647 0 0 0 0 17 4 0 0
0 0 0 0 0 0 0 0 0 0 0
./statm ---------------------
0 0 0 0 0 0 0
./maps ---------------------
./mem ---------------------
./clear_refs ---------------------
cat: ./clear_refs: Invalid argument
./smaps ---------------------
./smaps_rollup ---------------------
cat: ./smaps_rollup: No such process
./pagemap ---------------------
./wchan ---------------------
0
./stack ---------------------
[<0>] 0xffffffffffffffff
./schedstat ---------------------
5808249976150 707215957352 102392
./latency ---------------------
Latency Top version : v0.1
./cpuset ---------------------
/
./cgroup ---------------------
11:perf_event:/
10:cpuset:/
9:devices:/
8:rdma:/
7:cpu,cpuacct:/
6:blkio:/
5:freezer:/
4:net_cls:/
3:memory:/
2:pids:/
1:name=systemd:/
0::/
./oom_score ---------------------
0
./oom_adj ---------------------
0
./oom_score_adj ---------------------
0
./coredump_filter ---------------------
./io ---------------------
rchar: 0
wchar: 0
syscr: 0
syscw: 0
read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0
./uid_map ---------------------
0 0 4294967295
./gid_map ---------------------
0 0 4294967295
./projid_map ---------------------
0 0 4294967295
./setgroups ---------------------
allow
./timerslack_ns ---------------------
50000
./stack_depth ---------------------
previous stack depth: 16376
stack depth: 16376
next prev parent reply other threads:[~2019-05-16 14:18 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-09 9:54 5.1 kernel: khugepaged stuck at 100% Justin Piszcz
2019-05-09 11:13 ` Kirill A. Shutemov
2019-05-15 7:25 ` Justin Piszcz
2019-05-16 14:14 ` Justin Piszcz
2019-05-16 14:17 ` Justin Piszcz [this message]
2019-05-16 14:27 ` Justin Piszcz
2019-05-24 11:43 ` Oleksandr Natalenko
2019-06-06 17:24 ` Max Kellermann
2019-06-07 7:40 ` Max Kellermann
2019-06-07 7:50 ` Oleksandr Natalenko
2019-06-07 7:53 ` Max Kellermann
2019-05-09 11:17 ` Justin Piszcz
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAO9zADyq44Sn0kYZBC55C0ykpHaASzp+27K3BofbkEniK6af-w@mail.gmail.com \
--to=jpiszcz@lucidpixels.com \
--cc=kirill@shutemov.name \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).