* 5.1 kernel: khugepaged stuck at 100%
@ 2019-05-09 9:54 Justin Piszcz
2019-05-09 11:13 ` Kirill A. Shutemov
2019-05-09 11:17 ` Justin Piszcz
0 siblings, 2 replies; 12+ messages in thread
From: Justin Piszcz @ 2019-05-09 9:54 UTC (permalink / raw)
To: linux-kernel
[-- Attachment #1: Type: text/plain, Size: 1518 bytes --]
Hello,
Kernel: 5.1 (self-compiled, no modules)
Arch: x86_64
Distro: Debian Testing
Issue: I was performing a dump of ext3 and ext4 filesystems and then
restoring them to a separate volume (testing)-- afterwards I noticed that
khugepaged is stuck at 100% CPU. It is currently still stuck at 100% CPU, is
there anything I can do to debug what is happening prior to a reboot to work
around the issue? I have never seen this behavior prior to 5.1.
$ cat /proc/cmdline
auto BOOT_IMAGE=5.1.0-2 ro root=901 3w-sas.use_msi=1 nohugeiomap
page_poison=1 pcie_aspm=off pti=on rootfstype=ext4 slub_debug=P
zram.enabled=1 zram.num_devices=12 zswap.enabled=1 zswap.compressor=lz4
zswap.zpool=z3fold
$ 5.1 .config attached: config.txt.gz
$ attachment: graphic.JPG -> graph of the processes, dark green ->
khugepaged
$ top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
77 root 39 19 0 0 0 R 100.0 0.0 23:29.27
khugepaged
1 root 20 0 171800 7832 4948 S 0.0 0.0 1:25.59 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.02
kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
rcu_par_gp
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
kworker/0+
8 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
mm_percpu+
Thoughts on debugging / before reboot to clear this up?
Thanks,
Justin.
[-- Attachment #2: config.txt.gz --]
[-- Type: application/x-gzip, Size: 23869 bytes --]
[-- Attachment #3: graphic.JPG --]
[-- Type: image/jpeg, Size: 80182 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 5.1 kernel: khugepaged stuck at 100%
2019-05-09 9:54 5.1 kernel: khugepaged stuck at 100% Justin Piszcz
@ 2019-05-09 11:13 ` Kirill A. Shutemov
2019-05-15 7:25 ` Justin Piszcz
2019-05-16 14:14 ` Justin Piszcz
2019-05-09 11:17 ` Justin Piszcz
1 sibling, 2 replies; 12+ messages in thread
From: Kirill A. Shutemov @ 2019-05-09 11:13 UTC (permalink / raw)
To: Justin Piszcz; +Cc: linux-kernel
On Thu, May 09, 2019 at 05:54:56AM -0400, Justin Piszcz wrote:
> Hello,
>
> Kernel: 5.1 (self-compiled, no modules)
> Arch: x86_64
> Distro: Debian Testing
>
> Issue: I was performing a dump of ext3 and ext4 filesystems and then
> restoring them to a separate volume (testing)-- afterwards I noticed that
> khugepaged is stuck at 100% CPU. It is currently still stuck at 100% CPU, is
> there anything I can do to debug what is happening prior to a reboot to work
> around the issue? I have never seen this behavior prior to 5.1.
>
> $ cat /proc/cmdline
> auto BOOT_IMAGE=5.1.0-2 ro root=901 3w-sas.use_msi=1 nohugeiomap
> page_poison=1 pcie_aspm=off pti=on rootfstype=ext4 slub_debug=P
> zram.enabled=1 zram.num_devices=12 zswap.enabled=1 zswap.compressor=lz4
> zswap.zpool=z3fold
>
> $ 5.1 .config attached: config.txt.gz
>
> $ attachment: graphic.JPG -> graph of the processes, dark green ->
> khugepaged
>
> $ top
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 77 root 39 19 0 0 0 R 100.0 0.0 23:29.27
> khugepaged
> 1 root 20 0 171800 7832 4948 S 0.0 0.0 1:25.59 systemd
> 2 root 20 0 0 0 0 S 0.0 0.0 0:00.02
> kthreadd
> 3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
> 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
> rcu_par_gp
> 6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
> kworker/0+
> 8 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
> mm_percpu+
>
> Thoughts on debugging / before reboot to clear this up?
Could you check what khugepaged doing?
cat /proc/$(pidof khugepaged)/stack
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 5.1 kernel: khugepaged stuck at 100%
2019-05-09 9:54 5.1 kernel: khugepaged stuck at 100% Justin Piszcz
2019-05-09 11:13 ` Kirill A. Shutemov
@ 2019-05-09 11:17 ` Justin Piszcz
1 sibling, 0 replies; 12+ messages in thread
From: Justin Piszcz @ 2019-05-09 11:17 UTC (permalink / raw)
To: LKML
On Thu, May 9, 2019 at 5:54 AM Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>
> Hello,
>
> Kernel: 5.1 (self-compiled, no modules)
> Arch: x86_64
> Distro: Debian Testing
>
> Issue: I was performing a dump of ext3 and ext4 filesystems and then
> restoring them to a separate volume (testing)-- afterwards I noticed that
> khugepaged is stuck at 100% CPU. It is currently still stuck at 100% CPU, is
> there anything I can do to debug what is happening prior to a reboot to work
> around the issue? I have never seen this behavior prior to 5.1.
>
> $ cat /proc/cmdline
> auto BOOT_IMAGE=5.1.0-2 ro root=901 3w-sas.use_msi=1 nohugeiomap
> page_poison=1 pcie_aspm=off pti=on rootfstype=ext4 slub_debug=P
> zram.enabled=1 zram.num_devices=12 zswap.enabled=1 zswap.compressor=lz4
> zswap.zpool=z3fold
>
> $ 5.1 .config attached: config.txt.gz
>
> $ attachment: graphic.JPG -> graph of the processes, dark green ->
> khugepaged
>
> $ top
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 77 root 39 19 0 0 0 R 100.0 0.0 23:29.27
> khugepaged
> 1 root 20 0 171800 7832 4948 S 0.0 0.0 1:25.59 systemd
> 2 root 20 0 0 0 0 S 0.0 0.0 0:00.02
> kthreadd
> 3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
> 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
> rcu_par_gp
> 6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
> kworker/0+
> 8 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
> mm_percpu+
>
Will disable zcache/zswap for now, but FWIW
From perf top:
PerfTop: 1181 irqs/sec kernel:83.1% exact: 96.9% lost: 73/137
drop: 0/0 [4000Hz cycles], (all, 12 CPUs)
-------------------------------------------------------------------------------
54.56% [kernel] [k] compaction_alloc
29.46% [kernel] [k] __pageblock_pfn_to_page
10.66% [kernel] [k] nmi
0.61% [kernel] [k] _cond_resched
0.32% [kernel] [k] format_decode
0.25% [kernel] [k] __lock_text_start
0.20% [kernel] [k] __vma_adjust
0.16% perf [.] __symbols__insert
0.13% perf [.] rb_insert_color_cached
I tried dropping caches, etc, I will disable huge pages for now.
Some other posts:
https://bugzilla.redhat.com/show_bug.cgi?id=879801
https://lkml.org/lkml/2012/6/27/565
https://lkml.org/lkml/2011/11/9/252
^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: 5.1 kernel: khugepaged stuck at 100%
2019-05-09 11:13 ` Kirill A. Shutemov
@ 2019-05-15 7:25 ` Justin Piszcz
2019-05-16 14:14 ` Justin Piszcz
1 sibling, 0 replies; 12+ messages in thread
From: Justin Piszcz @ 2019-05-15 7:25 UTC (permalink / raw)
To: 'Kirill A. Shutemov'; +Cc: linux-kernel
> -----Original Message-----
> From: Kirill A. Shutemov [mailto:kirill@shutemov.name]
> Sent: Thursday, May 09, 2019 7:14 AM
> To: Justin Piszcz
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: 5.1 kernel: khugepaged stuck at 100%
>
> On Thu, May 09, 2019 at 05:54:56AM -0400, Justin Piszcz wrote:
[ ... ]
> > Thoughts on debugging / before reboot to clear this up?
>
> Could you check what khugepaged doing?
>
> cat /proc/$(pidof khugepaged)/stack
Hello, the next time I see this (khugepage at 100%) I will provide the
requested info.
However, this time I'm getting a kernel read fault (below) under 5.1.1,
under 5.0.x and earlier I've never seen any of these errors before:
[54101.281515] BUG: unable to handle kernel paging request at
ffffea0002060000
[54101.281519] #PF error: [normal kernel read fault]
[54101.281521] PGD 103ffee067 P4D 103ffee067 PUD 103ffed067 PMD 0
[54101.281524] Oops: 0000 [#1] SMP PTI
[54101.281527] CPU: 0 PID: 77 Comm: khugepaged Tainted: G T
5.1.1 #1
[54101.281528] Hardware name: Supermicro X9SRL-F/X9SRL-F, BIOS 3.2
01/16/2015
[54101.281534] RIP: 0010:isolate_freepages_block+0xb9/0x310
[54101.281535] Code: 24 28 48 c1 e0 06 40 f6 c5 1f 48 89 44 24 20 49 8d 45
79 48 89 44 24 18 44 89 f0 4d 89 ee 45 89 fd 41 89 c7 0f 84 ef 00 00 00 <48>
8b 03 41 83 c4 01 a9 00 00 01 00 75 0c 48 8b 43 08 a8 01 0f 84
[54101.281537] RSP: 0018:ffffc900003a7860 EFLAGS: 00010246
[54101.281538] RAX: 0000000000000000 RBX: ffffea0002060000 RCX:
ffffc900003a7b69
[54101.281539] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
ffffffff828c4790
[54101.281540] RBP: 0000000000081800 R08: 0000000000000001 R09:
0000000000000000
[54101.281542] R10: 0000000000000202 R11: ffffffff828c43d0 R12:
0000000000000000
[54101.281543] R13: 0000000000000000 R14: ffffc900003a7af0 R15:
0000000000000000
[54101.281544] FS: 0000000000000000(0000) GS:ffff88903f800000(0000)
knlGS:0000000000000000
[54101.281546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[54101.281547] CR2: ffffea0002060000 CR3: 000000000280e002 CR4:
00000000001606f0
[54101.281548] Call Trace:
[54101.281552] compaction_alloc+0x83d/0x8d0
[54101.281555] migrate_pages+0x30d/0x750
[54101.281557] ? isolate_migratepages_block+0xa10/0xa10
[54101.281559] ? __reset_isolation_suitable+0x110/0x110
[54101.281561] compact_zone+0x684/0xa70
[54101.281563] compact_zone_order+0x109/0x150
[54101.281566] ? schedule_timeout+0x1ba/0x290
[54101.281569] ? record_times+0x13/0xa0
[54101.281571] try_to_compact_pages+0x10d/0x220
[54101.281574] __alloc_pages_direct_compact+0x93/0x180
[54101.281576] __alloc_pages_nodemask+0x6c7/0xe20
[54101.281579] ? __wake_up_common_lock+0xb0/0xb0
[54101.281581] khugepaged+0x31f/0x19b0
[54101.281584] ? __wake_up_common_lock+0xb0/0xb0
[54101.281586] ? collapse_shmem.isra.4+0xc20/0xc20
[54101.281589] kthread+0x10a/0x120
[54101.281592] ? __kthread_create_on_node+0x1b0/0x1b0
[54101.281595] ret_from_fork+0x35/0x40
[54101.281597] CR2: ffffea0002060000
[54101.281599] ---[ end trace d9bfd1c296893a64 ]---
[54101.281601] RIP: 0010:isolate_freepages_block+0xb9/0x310
[54101.281602] Code: 24 28 48 c1 e0 06 40 f6 c5 1f 48 89 44 24 20 49 8d 45
79 48 89 44 24 18 44 89 f0 4d 89 ee 45 89 fd 41 89 c7 0f 84 ef 00 00 00 <48>
8b 03 41 83 c4 01 a9 00 00 01 00 75 0c 48 8b 43 08 a8 01 0f 84
[54101.281604] RSP: 0018:ffffc900003a7860 EFLAGS: 00010246
[54101.281605] RAX: 0000000000000000 RBX: ffffea0002060000 RCX:
ffffc900003a7b69
[54101.281606] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
ffffffff828c4790
[54101.281607] RBP: 0000000000081800 R08: 0000000000000001 R09:
0000000000000000
[54101.281608] R10: 0000000000000202 R11: ffffffff828c43d0 R12:
0000000000000000
[54101.281609] R13: 0000000000000000 R14: ffffc900003a7af0 R15:
0000000000000000
[54101.281610] FS: 0000000000000000(0000) GS:ffff88903f800000(0000)
knlGS:0000000000000000
[54101.281611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[54101.281613] CR2: ffffea0002060000 CR3: 000000000280e002 CR4:
00000000001606f0
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 5.1 kernel: khugepaged stuck at 100%
2019-05-09 11:13 ` Kirill A. Shutemov
2019-05-15 7:25 ` Justin Piszcz
@ 2019-05-16 14:14 ` Justin Piszcz
2019-05-16 14:17 ` Justin Piszcz
2019-06-06 17:24 ` Max Kellermann
1 sibling, 2 replies; 12+ messages in thread
From: Justin Piszcz @ 2019-05-16 14:14 UTC (permalink / raw)
To: Kirill A. Shutemov; +Cc: LKML
On Tue, May 14, 2019 at 9:16 AM Kirill A. Shutemov <kirill@shutemov.name> wrote:
> Could you check what khugepaged doing?
>
> cat /proc/$(pidof khugepaged)/stack
It is doing it again, 10:12am - 2019-05-16
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
77 root 39 19 0 0 0 R 100.0 0.0 92:06.94 khugepaged
Kernel: 5.1.2
$ sudo cat /proc/$(pidof khugepaged)/stack
[<0>] 0xffffffffffffffff
$ perf top
PerfTop: 3716 irqs/sec kernel:92.9% exact: 99.1% lost: 68/68
drop: 0/0 [4000Hz cycles], (all, 12 CPUs)
-------------------------------------------------------------------------------
47.53% [kernel] [k] compaction_alloc
38.88% [kernel] [k] __pageblock_pfn_to_page
6.68% [kernel] [k] nmi
0.58% [kernel] [k] __list_del_entry_valid
0.48% [kernel] [k] format_decode
0.39% [kernel] [k] __rb_insert_augmented
0.25% libdbus-1.so.3.19.9 [.] _dbus_string_hex_decode
0.24% [kernel] [k] entry_SYSCALL_64_after_hwframe
0.20% perf [.] rb_next
0.19% perf [.] __symbols__insert
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 5.1 kernel: khugepaged stuck at 100%
2019-05-16 14:14 ` Justin Piszcz
@ 2019-05-16 14:17 ` Justin Piszcz
2019-05-16 14:27 ` Justin Piszcz
2019-06-06 17:24 ` Max Kellermann
1 sibling, 1 reply; 12+ messages in thread
From: Justin Piszcz @ 2019-05-16 14:17 UTC (permalink / raw)
To: Kirill A. Shutemov; +Cc: LKML
On Thu, May 16, 2019 at 10:14 AM Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>
> On Tue, May 14, 2019 at 9:16 AM Kirill A. Shutemov <kirill@shutemov.name> wrote:
>
> > Could you check what khugepaged doing?
> >
> > cat /proc/$(pidof khugepaged)/stack
>
> It is doing it again, 10:12am - 2019-05-16
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 77 root 39 19 0 0 0 R 100.0 0.0 92:06.94 khugepaged
>
> Kernel: 5.1.2
>
> $ sudo cat /proc/$(pidof khugepaged)/stack
> [<0>] 0xffffffffffffffff
>
> $ perf top
>
> PerfTop: 3716 irqs/sec kernel:92.9% exact: 99.1% lost: 68/68
> drop: 0/0 [4000Hz cycles], (all, 12 CPUs)
> -------------------------------------------------------------------------------
>
> 47.53% [kernel] [k] compaction_alloc
> 38.88% [kernel] [k] __pageblock_pfn_to_page
> 6.68% [kernel] [k] nmi
> 0.58% [kernel] [k] __list_del_entry_valid
> 0.48% [kernel] [k] format_decode
> 0.39% [kernel] [k] __rb_insert_augmented
> 0.25% libdbus-1.so.3.19.9 [.] _dbus_string_hex_decode
> 0.24% [kernel] [k] entry_SYSCALL_64_after_hwframe
> 0.20% perf [.] rb_next
> 0.19% perf [.] __symbols__insert
Incase this is asked for later:
# find . -maxdepth 1 -type f | while read f; do echo "$f
---------------------"; cat $f; echo " "; done
./environ ---------------------
./auxv ---------------------
./status ---------------------
Name: khugepaged
Umask: 0000
State: R (running)
Tgid: 77
Ngid: 0
Pid: 77
PPid: 2
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 64
Groups:
NStgid: 77
NSpid: 77
NSpgid: 0
NSsid: 0
Threads: 1
SigQ: 1/257125
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: ffffffffffffffff
SigCgt: 0000000000000000
CapInh: 0000000000000000
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
NoNewPrivs: 0
Seccomp: 0
Speculation_Store_Bypass: thread vulnerable
Cpus_allowed: fff
Cpus_allowed_list: 0-11
Mems_allowed: 1
Mems_allowed_list: 0
voluntary_ctxt_switches: 7728
nonvoluntary_ctxt_switches: 94662
./personality ---------------------
00000000
./limits ---------------------
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 257125 257125 processes
Max open files 1024 4096 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 257125 257125 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
./sched ---------------------
khugepaged (77, #threads: 1)
-------------------------------------------------------------------
se.exec_start : 84565632.684623
se.vruntime : 30287808.462012
se.sum_exec_runtime : 5808238.978150
se.nr_migrations : 946
nr_switches : 102390
nr_voluntary_switches : 7728
nr_involuntary_switches : 94662
se.load.weight : 15360
se.runnable_weight : 15360
se.avg.load_sum : 47454
se.avg.runnable_load_sum : 47454
se.avg.util_sum : 48835831
se.avg.load_avg : 14
se.avg.runnable_load_avg : 14
se.avg.util_avg : 1024
se.avg.last_update_time : 84565632684032
se.avg.util_est.ewma : 25
se.avg.util_est.enqueued : 16
policy : 0
prio : 139
clock-delta : 11
./comm ---------------------
khugepaged
./syscall ---------------------
running
./cmdline ---------------------
./stat ---------------------
77 (khugepaged) R 2 0 0 0 -1 27265088 0 0 0 0 0 580824 0 0 39 19 1 0
725 0 0 18446744073709551615 0 0 0 0 0 0 0 2147483647 0 0 0 0 17 4 0 0
0 0 0 0 0 0 0 0 0 0 0
./statm ---------------------
0 0 0 0 0 0 0
./maps ---------------------
./mem ---------------------
./clear_refs ---------------------
cat: ./clear_refs: Invalid argument
./smaps ---------------------
./smaps_rollup ---------------------
cat: ./smaps_rollup: No such process
./pagemap ---------------------
./wchan ---------------------
0
./stack ---------------------
[<0>] 0xffffffffffffffff
./schedstat ---------------------
5808249976150 707215957352 102392
./latency ---------------------
Latency Top version : v0.1
./cpuset ---------------------
/
./cgroup ---------------------
11:perf_event:/
10:cpuset:/
9:devices:/
8:rdma:/
7:cpu,cpuacct:/
6:blkio:/
5:freezer:/
4:net_cls:/
3:memory:/
2:pids:/
1:name=systemd:/
0::/
./oom_score ---------------------
0
./oom_adj ---------------------
0
./oom_score_adj ---------------------
0
./coredump_filter ---------------------
./io ---------------------
rchar: 0
wchar: 0
syscr: 0
syscw: 0
read_bytes: 0
write_bytes: 0
cancelled_write_bytes: 0
./uid_map ---------------------
0 0 4294967295
./gid_map ---------------------
0 0 4294967295
./projid_map ---------------------
0 0 4294967295
./setgroups ---------------------
allow
./timerslack_ns ---------------------
50000
./stack_depth ---------------------
previous stack depth: 16376
stack depth: 16376
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 5.1 kernel: khugepaged stuck at 100%
2019-05-16 14:17 ` Justin Piszcz
@ 2019-05-16 14:27 ` Justin Piszcz
2019-05-24 11:43 ` Oleksandr Natalenko
0 siblings, 1 reply; 12+ messages in thread
From: Justin Piszcz @ 2019-05-16 14:27 UTC (permalink / raw)
To: Kirill A. Shutemov; +Cc: LKML
On Thu, May 16, 2019 at 10:17 AM Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>
> On Thu, May 16, 2019 at 10:14 AM Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
> >
> > On Tue, May 14, 2019 at 9:16 AM Kirill A. Shutemov <kirill@shutemov.name> wrote:
> >
> > > Could you check what khugepaged doing?
> > >
> > > cat /proc/$(pidof khugepaged)/stack
> >
> > It is doing it again, 10:12am - 2019-05-16
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > 77 root 39 19 0 0 0 R 100.0 0.0 92:06.94 khugepaged
> >
> > Kernel: 5.1.2
> >
> > $ sudo cat /proc/$(pidof khugepaged)/stack
> > [<0>] 0xffffffffffffffff
> >
As a workaround/in the meantime--I will add the following to lilo/grub:
transparent_hugepage=never
Justin
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 5.1 kernel: khugepaged stuck at 100%
2019-05-16 14:27 ` Justin Piszcz
@ 2019-05-24 11:43 ` Oleksandr Natalenko
0 siblings, 0 replies; 12+ messages in thread
From: Oleksandr Natalenko @ 2019-05-24 11:43 UTC (permalink / raw)
To: Justin Piszcz; +Cc: Kirill A. Shutemov, LKML
On Thu, May 16, 2019 at 10:27:50AM -0400, Justin Piszcz wrote:
> On Thu, May 16, 2019 at 10:17 AM Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
> >
> > On Thu, May 16, 2019 at 10:14 AM Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
> > >
> > > On Tue, May 14, 2019 at 9:16 AM Kirill A. Shutemov <kirill@shutemov.name> wrote:
> > >
> > > > Could you check what khugepaged doing?
> > > >
> > > > cat /proc/$(pidof khugepaged)/stack
> > >
> > > It is doing it again, 10:12am - 2019-05-16
> > >
> > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > > 77 root 39 19 0 0 0 R 100.0 0.0 92:06.94 khugepaged
> > >
> > > Kernel: 5.1.2
> > >
> > > $ sudo cat /proc/$(pidof khugepaged)/stack
> > > [<0>] 0xffffffffffffffff
> > >
>
> As a workaround/in the meantime--I will add the following to lilo/grub:
> transparent_hugepage=never
>
> Justin
Cc'ing myself since i observe such a behaviour sometimes right after KVM
VM is launched. No luck with reproducing it on demand so far, though.
--
Best regards,
Oleksandr Natalenko (post-factum)
Senior Software Maintenance Engineer
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 5.1 kernel: khugepaged stuck at 100%
2019-05-16 14:14 ` Justin Piszcz
2019-05-16 14:17 ` Justin Piszcz
@ 2019-06-06 17:24 ` Max Kellermann
2019-06-07 7:40 ` Max Kellermann
1 sibling, 1 reply; 12+ messages in thread
From: Max Kellermann @ 2019-06-06 17:24 UTC (permalink / raw)
To: Justin Piszcz; +Cc: Kirill A. Shutemov, LKML
On 2019/05/16 16:14, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
> Kernel: 5.1.2
>
> $ sudo cat /proc/$(pidof khugepaged)/stack
> [<0>] 0xffffffffffffffff
>
> $ perf top
>
> PerfTop: 3716 irqs/sec kernel:92.9% exact: 99.1% lost: 68/68
> drop: 0/0 [4000Hz cycles], (all, 12 CPUs)
> -------------------------------------------------------------------------------
>
> 47.53% [kernel] [k] compaction_alloc
> 38.88% [kernel] [k] __pageblock_pfn_to_page
> 6.68% [kernel] [k] nmi
> 0.58% [kernel] [k] __list_del_entry_valid
> 0.48% [kernel] [k] format_decode
> 0.39% [kernel] [k] __rb_insert_augmented
> 0.25% libdbus-1.so.3.19.9 [.] _dbus_string_hex_decode
> 0.24% [kernel] [k] entry_SYSCALL_64_after_hwframe
> 0.20% perf [.] rb_next
> 0.19% perf [.] __symbols__insert
I have the same problem (kernel 5.1.7), but over here, it's a PHP
process, not khugepaged, which is looping inside compaction_alloc.
This is from "perf report":
100.00% 0.00% php-cgi7.0 php-cgi7.0 [.] 0x000055d0e88bc5ee
|
---0x55d0e88bc5ee
page_fault
__do_page_fault
handle_mm_fault
__handle_mm_fault
do_huge_pmd_anonymous_page
__alloc_pages_nodemask
__alloc_pages_slowpath
__alloc_pages_direct_compact
try_to_compact_pages
compact_zone_order
compact_zone
migrate_pages
compaction_alloc
|
|--24.43%--__pageblock_pfn_to_page
|
--2.22%--_cond_resched
|
--0.89%--rcu_all_qs
ftrace:
<...>-263514 [012] .... 109004.793009: rcu_all_qs <-_cond_resched
<...>-263514 [012] .... 109004.793009: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793009: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793009: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793009: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793009: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793009: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793009: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: _cond_resched <-compaction_alloc
(Repeating this sequence over and over.)
Nothing useful in /proc/263514/{stack,wchan,syscall}.
What else can I do to collect more information to aid fix this?
Max
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 5.1 kernel: khugepaged stuck at 100%
2019-06-06 17:24 ` Max Kellermann
@ 2019-06-07 7:40 ` Max Kellermann
2019-06-07 7:50 ` Oleksandr Natalenko
2019-06-07 7:53 ` Max Kellermann
0 siblings, 2 replies; 12+ messages in thread
From: Max Kellermann @ 2019-06-07 7:40 UTC (permalink / raw)
To: Justin Piszcz, Kirill A. Shutemov, LKML
On 2019/06/06 19:24, Max Kellermann <max@blarg.de> wrote:
> I have the same problem (kernel 5.1.7), but over here, it's a PHP
> process, not khugepaged, which is looping inside compaction_alloc.
This is what happened an hour later:
kernel tried to execute NX-protected page - exploit attempt? (uid: 33333)
BUG: unable to handle kernel paging request at ffffffffc036f00f
#PF error: [PROT] [INSTR]
PGD 35fa10067 P4D 35fa10067 PUD 35fa12067 PMD 105ba71067 PTE 800000022d28e061
Oops: 0011 [#1] SMP PTI
CPU: 12 PID: 263514 Comm: php-cgi7.0 Not tainted 5.1.7-cmag1-th+ #5
Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 10/17/2018
RIP: 0010:0xffffffffc036f00f
Code: Bad RIP value.
RSP: 0018:ffffb63c4d547928 EFLAGS: 00010216
RAX: 0000000000000000 RBX: ffffb63c4d547b10 RCX: 0000ffc004d021bd
RDX: ffff9ac83fffc500 RSI: 7fe0026810dee7ff RDI: 7fe0026810dee400
RBP: 7fe0026810dee400 R08: 0000000000000002 R09: 0000000000020300
R10: 00010642641a0d3a R11: 0000000000000001 R12: 7fe0026810dee800
R13: 0000000000000001 R14: 0000000000000000 R15: ffff9ac83fffc500
FS: 00007fa5c1000740(0000) GS:ffff9ad01f600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffffc036efe5 CR3: 00000008eb8a0005 CR4: 00000000001606e0
Call Trace:
? move_freelist_tail+0xd0/0xd0
? migrate_pages+0xaa/0x780
? isolate_freepages_block+0x380/0x380
? compact_zone+0x6ec/0xca0
? compact_zone_order+0xd8/0x120
? try_to_compact_pages+0xb1/0x260
? __alloc_pages_direct_compact+0x87/0x160
? __alloc_pages_slowpath+0x427/0xd50
? __alloc_pages_nodemask+0x2d6/0x310
? do_huge_pmd_anonymous_page+0x131/0x680
? vma_merge+0x24f/0x3a0
? __handle_mm_fault+0xbca/0x1260
? handle_mm_fault+0x135/0x1b0
? __do_page_fault+0x242/0x4b0
? page_fault+0x8/0x30
? page_fault+0x1e/0x30
Modules linked in:
CR2: ffffffffc036f00f
---[ end trace 0f31edf3041f5d9e ]---
RIP: 0010:0xffffffffc036f00f
Code: Bad RIP value.
RSP: 0018:ffffb63c4d547928 EFLAGS: 00010216
RAX: 0000000000000000 RBX: ffffb63c4d547b10 RCX: 0000ffc004d021bd
RDX: ffff9ac83fffc500 RSI: 7fe0026810dee7ff RDI: 7fe0026810dee400
RBP: 7fe0026810dee400 R08: 0000000000000002 R09: 0000000000020300
R10: 00010642641a0d3a R11: 0000000000000001 R12: 7fe0026810dee800
R13: 0000000000000001 R14: 0000000000000000 R15: ffff9ac83fffc500
FS: 00007fa5c1000740(0000) GS:ffff9ad01f600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffffc036efe5 CR3: 00000008eb8a0005 CR4: 00000000001606e0
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 5.1 kernel: khugepaged stuck at 100%
2019-06-07 7:40 ` Max Kellermann
@ 2019-06-07 7:50 ` Oleksandr Natalenko
2019-06-07 7:53 ` Max Kellermann
1 sibling, 0 replies; 12+ messages in thread
From: Oleksandr Natalenko @ 2019-06-07 7:50 UTC (permalink / raw)
To: Max Kellermann, Justin Piszcz, Kirill A. Shutemov, LKML
Hi.
On Fri, Jun 07, 2019 at 09:40:52AM +0200, Max Kellermann wrote:
> On 2019/06/06 19:24, Max Kellermann <max@blarg.de> wrote:
> > I have the same problem (kernel 5.1.7), but over here, it's a PHP
> > process, not khugepaged, which is looping inside compaction_alloc.
>
> This is what happened an hour later:
>
> kernel tried to execute NX-protected page - exploit attempt? (uid: 33333)
> BUG: unable to handle kernel paging request at ffffffffc036f00f
> #PF error: [PROT] [INSTR]
> PGD 35fa10067 P4D 35fa10067 PUD 35fa12067 PMD 105ba71067 PTE 800000022d28e061
> Oops: 0011 [#1] SMP PTI
> CPU: 12 PID: 263514 Comm: php-cgi7.0 Not tainted 5.1.7-cmag1-th+ #5
> Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 10/17/2018
> RIP: 0010:0xffffffffc036f00f
> Code: Bad RIP value.
> RSP: 0018:ffffb63c4d547928 EFLAGS: 00010216
> RAX: 0000000000000000 RBX: ffffb63c4d547b10 RCX: 0000ffc004d021bd
> RDX: ffff9ac83fffc500 RSI: 7fe0026810dee7ff RDI: 7fe0026810dee400
> RBP: 7fe0026810dee400 R08: 0000000000000002 R09: 0000000000020300
> R10: 00010642641a0d3a R11: 0000000000000001 R12: 7fe0026810dee800
> R13: 0000000000000001 R14: 0000000000000000 R15: ffff9ac83fffc500
> FS: 00007fa5c1000740(0000) GS:ffff9ad01f600000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: ffffffffc036efe5 CR3: 00000008eb8a0005 CR4: 00000000001606e0
> Call Trace:
> ? move_freelist_tail+0xd0/0xd0
> ? migrate_pages+0xaa/0x780
> ? isolate_freepages_block+0x380/0x380
> ? compact_zone+0x6ec/0xca0
> ? compact_zone_order+0xd8/0x120
> ? try_to_compact_pages+0xb1/0x260
> ? __alloc_pages_direct_compact+0x87/0x160
> ? __alloc_pages_slowpath+0x427/0xd50
> ? __alloc_pages_nodemask+0x2d6/0x310
> ? do_huge_pmd_anonymous_page+0x131/0x680
> ? vma_merge+0x24f/0x3a0
> ? __handle_mm_fault+0xbca/0x1260
> ? handle_mm_fault+0x135/0x1b0
> ? __do_page_fault+0x242/0x4b0
> ? page_fault+0x8/0x30
> ? page_fault+0x1e/0x30
> Modules linked in:
> CR2: ffffffffc036f00f
> ---[ end trace 0f31edf3041f5d9e ]---
> RIP: 0010:0xffffffffc036f00f
> Code: Bad RIP value.
> RSP: 0018:ffffb63c4d547928 EFLAGS: 00010216
> RAX: 0000000000000000 RBX: ffffb63c4d547b10 RCX: 0000ffc004d021bd
> RDX: ffff9ac83fffc500 RSI: 7fe0026810dee7ff RDI: 7fe0026810dee400
> RBP: 7fe0026810dee400 R08: 0000000000000002 R09: 0000000000020300
> R10: 00010642641a0d3a R11: 0000000000000001 R12: 7fe0026810dee800
> R13: 0000000000000001 R14: 0000000000000000 R15: ffff9ac83fffc500
> FS: 00007fa5c1000740(0000) GS:ffff9ad01f600000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: ffffffffc036efe5 CR3: 00000008eb8a0005 CR4: 00000000001606e0
Make sure to check if e577c8b64d ("mm, compaction: make sure we isolate
a valid PFN") fixes your issue. It is staged for 5.1.8, BTW.
--
Best regards,
Oleksandr Natalenko (post-factum)
Senior Software Maintenance Engineer
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 5.1 kernel: khugepaged stuck at 100%
2019-06-07 7:40 ` Max Kellermann
2019-06-07 7:50 ` Oleksandr Natalenko
@ 2019-06-07 7:53 ` Max Kellermann
1 sibling, 0 replies; 12+ messages in thread
From: Max Kellermann @ 2019-06-07 7:53 UTC (permalink / raw)
To: Justin Piszcz, Kirill A. Shutemov, LKML
On 2019/06/07 09:40, Max Kellermann <max@blarg.de> wrote:
> On 2019/06/06 19:24, Max Kellermann <max@blarg.de> wrote:
> > I have the same problem (kernel 5.1.7), but over here, it's a PHP
> > process, not khugepaged, which is looping inside compaction_alloc.
>
> This is what happened an hour later:
>
> kernel tried to execute NX-protected page - exploit attempt? (uid: 33333)
> BUG: unable to handle kernel paging request at ffffffffc036f00f
Another machine in the cluster has just crashed (same kernel, and
hugepage/compaction again):
BUG: unable to handle kernel paging request at ffffd43382060000
#PF error: [normal kernel read fault]
PGD 85fff8067 P4D 85fff8067 PUD 85fff7067 PMD 0
Oops: 0000 [#2] SMP PTI
CPU: 27 PID: 117908 Comm: php-cgi7.0 Tainted: G D 5.1.7-cmag1-th+ #5
Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 10/17/2018
RIP: 0010:isolate_freepages_block+0xa3/0x380
Code: 10 4c 89 c0 c7 44 24 0c 00 00 00 00 4d 89 de 48 c1 e0 06 f6 c3 1f 48 89 44 24 18 44 89 c8 49 89 d1 41 89 c4 0f 84 03 01 00 00 <49> 8b 07 83 c5 01 a9 00 00 01 00 75 0c 49 8b 47 08 a8 01 0f 84 b8
RSP: 0018:ffffad0d258bf898 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000081800 RCX: ffffad0d258bfb89
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9c08ffffc8e0
RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000081bff
R10: 0000000000000000 R11: ffffad0d258bfb89 R12: 0000000000000000
R13: ffffad0d258bfb10 R14: ffffad0d258bfb89 R15: ffffd43382060000
FS: 00007f92c1c84740(0000) GS:ffff9c08df9c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffd43382060000 CR3: 0000000966150002 CR4: 00000000001606e0
Call Trace:
compaction_alloc+0x7e6/0x870
? move_freelist_tail+0xd0/0xd0
migrate_pages+0xaa/0x780
? isolate_freepages_block+0x380/0x380
compact_zone+0x6ec/0xca0
compact_zone_order+0xd8/0x120
try_to_compact_pages+0xb1/0x260
__alloc_pages_direct_compact+0x87/0x160
__alloc_pages_slowpath+0x427/0xd50
__alloc_pages_nodemask+0x2d6/0x310
do_huge_pmd_anonymous_page+0x131/0x680
? vma_merge+0x24f/0x3a0
__handle_mm_fault+0xbca/0x1260
handle_mm_fault+0x135/0x1b0
__do_page_fault+0x242/0x4b0
? page_fault+0x8/0x30
page_fault+0x1e/0x30
RIP: 0033:0x55ab2b7205ee
Code: 24 18 01 00 00 41 8b 8c 24 48 01 00 00 8d 51 01 41 3b 94 24 4c 01 00 00 41 89 94 24 48 01 00 00 7e 08 41 89 94 24 4c 01 00 00 <4c> 89 20 49 8b 94 24 38 01 00 00 45 31 db 41 bd 00 10 00 00 41 b9
RSP: 002b:00007ffd79f76750 EFLAGS: 00010202
RAX: 00007f929b800000 RBX: 0000000000000011 RCX: 0000000000000007
RDX: 0000000000000008 RSI: 0000000000200000 RDI: 00007f929b800000
RBP: 0000000000000006 R08: ffffffffffffffff R09: 0000000000000000
R10: 0000000000000022 R11: 0000000000000246 R12: 00007f92bd000040
R13: 00000000ffffffff R14: 0000000000000200 R15: 00007f929c7ac648
Modules linked in:
CR2: ffffd43382060000
---[ end trace 73412f77c0bd1f72 ]---
RIP: 0010:isolate_freepages_block+0xa3/0x380
Code: 10 4c 89 c0 c7 44 24 0c 00 00 00 00 4d 89 de 48 c1 e0 06 f6 c3 1f 48 89 44 24 18 44 89 c8 49 89 d1 41 89 c4 0f 84 03 01 00 00 <49> 8b 07 83 c5 01 a9 00 00 01 00 75 0c 49 8b 47 08 a8 01 0f 84 b8
RSP: 0018:ffffad0d0894f898 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000082000 RCX: ffffad0d0894fb89
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9c08ffffc8e0
RBP: 0000000000000000 R08: 0000000000000001 R09: 00000000000823ff
R10: 0000000000000000 R11: ffffad0d0894fb89 R12: 0000000000000000
R13: ffffad0d0894fb10 R14: ffffad0d0894fb89 R15: ffffd43382080000
FS: 00007f92c1c84740(0000) GS:ffff9c08df9c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffd43382060000 CR3: 0000000966150002 CR4: 00000000001606e0
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2019-06-07 7:53 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-05-09 9:54 5.1 kernel: khugepaged stuck at 100% Justin Piszcz
2019-05-09 11:13 ` Kirill A. Shutemov
2019-05-15 7:25 ` Justin Piszcz
2019-05-16 14:14 ` Justin Piszcz
2019-05-16 14:17 ` Justin Piszcz
2019-05-16 14:27 ` Justin Piszcz
2019-05-24 11:43 ` Oleksandr Natalenko
2019-06-06 17:24 ` Max Kellermann
2019-06-07 7:40 ` Max Kellermann
2019-06-07 7:50 ` Oleksandr Natalenko
2019-06-07 7:53 ` Max Kellermann
2019-05-09 11:17 ` Justin Piszcz
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).