* [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-27 16:09 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-27 16:09 UTC (permalink / raw)
To: linux-fsdevel, linux-mm; +Cc: linux-kernel
The bug manifests as a soft lockup in kswapd:
[ 155.759084] netconsole: network logging started
[ 598.920430] BUG: soft lockup - CPU#1 stuck for 67s! [kswapd0:46]
[ 598.920472] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole]
[ 598.920834] CPU 1
[ 598.920843] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole]
[ 598.926818]
[ 598.928043] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 598.929299] RIP: 0010:[<ffffffffa007ff9b>] [<ffffffffa007ff9b>] i915_gem_inactive_shrink+0x78/0x194 [i915]
[ 598.930603] RSP: 0018:ffff8800709ebd50 EFLAGS: 00000216
[ 598.931867] RAX: ffff88006ec5c6b0 RBX: 00000000000000c0 RCX: 0000000000000000
[ 598.933135] RDX: ffff880037e59638 RSI: 0000000000000000 RDI: ffff880037876020
[ 598.934408] RBP: ffff8800709ebd90 R08: 0000000000000000 R09: 000000000001bd90
[ 598.935680] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff8148300e
[ 598.936944] R13: ffff8800709ebcf0 R14: ffff8800709ebcf8 R15: ffffffff810dd64d
[ 598.938221] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000
[ 598.939525] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 598.940813] CR2: 00007fe16a34d380 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 598.942114] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 598.943441] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 598.944744] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006df8ae00)
[ 598.946044] Stack:
[ 598.947317] ffff8800709ebd90 ffff880037e59638 ffff8800709ebd60 ffff880037e595f0
[ 598.948654] 0000000000000000 0000000000000000 00000000000000d0 000000000004c24e
[ 598.949986] ffff8800709ebde0 ffffffff810e1f89 000000000000003d 0000000000000080
[ 598.951329] Call Trace:
[ 598.952673] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
[ 598.954003] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 598.955331] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 598.956683] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 598.958022] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 598.959379] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 598.960709] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 598.962032] Code: 31 ed 48 83 c3 48 48 2d b0 00 00 00 eb 0a 48 8d 82 50 ff ff ff 41 ff c5 48 8b 90 b0 00 00 00 48 05 b0 00 00 00 48 39 d8 0f 18 0a
[ 598.962192] e1 e9 da 00 00 00 4c 89 f7 e8 c6 fd ff ff 48 8b 43 48 4c 8b
[ 598.965009] Call Trace:
[ 598.966421] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
[ 598.966423] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 598.966426] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 598.966429] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 598.966432] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 598.966435] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 598.966438] [<ffffffff81483760>] ? gs_change+0x13/0x13
The traces are slightly different each reboot cycle, but it's always in
kswapd and usually in shrink_slab. Once it happens, anything that
touches the filesystem hangs in D wait, so the machine is basically
toast.
The box is a Lenovo T420s sandybridge core i5 based laptop with 2GB of
memory.
There is a corresponding Red Hat bugzilla report here:
https://bugzilla.redhat.com/show_bug.cgi?id=694818
And I've verified that the bug also shows in the 2.6.38.3 stable kernel.
If anyone wants me to try anything, please let me know.
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-27 16:09 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-27 16:09 UTC (permalink / raw)
To: linux-fsdevel, linux-mm; +Cc: linux-kernel
The bug manifests as a soft lockup in kswapd:
[ 155.759084] netconsole: network logging started
[ 598.920430] BUG: soft lockup - CPU#1 stuck for 67s! [kswapd0:46]
[ 598.920472] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole]
[ 598.920834] CPU 1
[ 598.920843] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole]
[ 598.926818]
[ 598.928043] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 598.929299] RIP: 0010:[<ffffffffa007ff9b>] [<ffffffffa007ff9b>] i915_gem_inactive_shrink+0x78/0x194 [i915]
[ 598.930603] RSP: 0018:ffff8800709ebd50 EFLAGS: 00000216
[ 598.931867] RAX: ffff88006ec5c6b0 RBX: 00000000000000c0 RCX: 0000000000000000
[ 598.933135] RDX: ffff880037e59638 RSI: 0000000000000000 RDI: ffff880037876020
[ 598.934408] RBP: ffff8800709ebd90 R08: 0000000000000000 R09: 000000000001bd90
[ 598.935680] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff8148300e
[ 598.936944] R13: ffff8800709ebcf0 R14: ffff8800709ebcf8 R15: ffffffff810dd64d
[ 598.938221] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000
[ 598.939525] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 598.940813] CR2: 00007fe16a34d380 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 598.942114] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 598.943441] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 598.944744] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006df8ae00)
[ 598.946044] Stack:
[ 598.947317] ffff8800709ebd90 ffff880037e59638 ffff8800709ebd60 ffff880037e595f0
[ 598.948654] 0000000000000000 0000000000000000 00000000000000d0 000000000004c24e
[ 598.949986] ffff8800709ebde0 ffffffff810e1f89 000000000000003d 0000000000000080
[ 598.951329] Call Trace:
[ 598.952673] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
[ 598.954003] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 598.955331] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 598.956683] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 598.958022] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 598.959379] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 598.960709] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 598.962032] Code: 31 ed 48 83 c3 48 48 2d b0 00 00 00 eb 0a 48 8d 82 50 ff ff ff 41 ff c5 48 8b 90 b0 00 00 00 48 05 b0 00 00 00 48 39 d8 0f 18 0a
[ 598.962192] e1 e9 da 00 00 00 4c 89 f7 e8 c6 fd ff ff 48 8b 43 48 4c 8b
[ 598.965009] Call Trace:
[ 598.966421] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
[ 598.966423] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 598.966426] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 598.966429] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 598.966432] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 598.966435] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 598.966438] [<ffffffff81483760>] ? gs_change+0x13/0x13
The traces are slightly different each reboot cycle, but it's always in
kswapd and usually in shrink_slab. Once it happens, anything that
touches the filesystem hangs in D wait, so the machine is basically
toast.
The box is a Lenovo T420s sandybridge core i5 based laptop with 2GB of
memory.
There is a corresponding Red Hat bugzilla report here:
https://bugzilla.redhat.com/show_bug.cgi?id=694818
And I've verified that the bug also shows in the 2.6.38.3 stable kernel.
If anyone wants me to try anything, please let me know.
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-27 16:09 ` James Bottomley
@ 2011-04-27 16:33 ` Chris Mason
-1 siblings, 0 replies; 138+ messages in thread
From: Chris Mason @ 2011-04-27 16:33 UTC (permalink / raw)
To: James Bottomley; +Cc: linux-fsdevel, linux-mm, linux-kernel
Excerpts from James Bottomley's message of 2011-04-27 12:09:13 -0400:
> The bug manifests as a soft lockup in kswapd:
>
> [ 155.759084] netconsole: network logging started
> [ 598.920430] BUG: soft lockup - CPU#1 stuck for 67s! [kswapd0:46]
> [ 598.920472] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole]
> [ 598.920834] CPU 1
> [ 598.920843] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole]
> [ 598.926818]
Probably easier to debug with a sysrq-l and sysrq-w. If you get stuck
on the filesystem, it is probably waiting on ram, which it probably
can't get because kswapd is spinning. Eventually everyone backs up
waiting for the transaction that never ends. If we're really lucky it
is just GFP_KERNEL where it should NOFS.
Since you're often stuck in different spots inside shrink_slab, we're
probably not stuck on a lock. But, trying with lock debugging, lockdep
enabled and preempt on is a good idea to rule out locking mistakes.
Does the fedora debug kernel enable preempt?
-chris
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-27 16:33 ` Chris Mason
0 siblings, 0 replies; 138+ messages in thread
From: Chris Mason @ 2011-04-27 16:33 UTC (permalink / raw)
To: James Bottomley; +Cc: linux-fsdevel, linux-mm, linux-kernel
Excerpts from James Bottomley's message of 2011-04-27 12:09:13 -0400:
> The bug manifests as a soft lockup in kswapd:
>
> [ 155.759084] netconsole: network logging started
> [ 598.920430] BUG: soft lockup - CPU#1 stuck for 67s! [kswapd0:46]
> [ 598.920472] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole]
> [ 598.920834] CPU 1
> [ 598.920843] Modules linked in: netconsole configfs fuse sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep arc4 snd_seq snd_seq_device snd_pcm iwlagn mac80211 snd_timer uvcvideo btusb bluetooth snd cfg80211 videodev soundcore v4l2_compat_ioctl32 iTCO_wdt xhci_hcd e1000e snd_page_alloc rfkill i2c_i801 wmi iTCO_vendor_support microcode pcspkr joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: netconsole]
> [ 598.926818]
Probably easier to debug with a sysrq-l and sysrq-w. If you get stuck
on the filesystem, it is probably waiting on ram, which it probably
can't get because kswapd is spinning. Eventually everyone backs up
waiting for the transaction that never ends. If we're really lucky it
is just GFP_KERNEL where it should NOFS.
Since you're often stuck in different spots inside shrink_slab, we're
probably not stuck on a lock. But, trying with lock debugging, lockdep
enabled and preempt on is a good idea to rule out locking mistakes.
Does the fedora debug kernel enable preempt?
-chris
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-27 16:33 ` Chris Mason
(?)
@ 2011-04-27 16:50 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-27 16:50 UTC (permalink / raw)
To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel
On Wed, 2011-04-27 at 12:33 -0400, Chris Mason wrote:
> Probably easier to debug with a sysrq-l and sysrq-w.
Yes, just enabling them for the repro ... I didn't echo 1
> /proc/sys/kernel/sysrq first time around
> If you get stuck
> on the filesystem, it is probably waiting on ram, which it probably
> can't get because kswapd is spinning. Eventually everyone backs up
> waiting for the transaction that never ends. If we're really lucky it
> is just GFP_KERNEL where it should NOFS.
>
> Since you're often stuck in different spots inside shrink_slab, we're
> probably not stuck on a lock. But, trying with lock debugging, lockdep
> enabled and preempt on is a good idea to rule out locking mistakes.
>
> Does the fedora debug kernel enable preempt?
No ... I've taken the Fedora 15 kernel config directly from their
install CD for this (with a bit of munging to get it to work on -rc4)
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-27 16:50 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-27 16:50 UTC (permalink / raw)
To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel
On Wed, 2011-04-27 at 12:33 -0400, Chris Mason wrote:
> Probably easier to debug with a sysrq-l and sysrq-w.
Yes, just enabling them for the repro ... I didn't echo 1
> /proc/sys/kernel/sysrq first time around
> If you get stuck
> on the filesystem, it is probably waiting on ram, which it probably
> can't get because kswapd is spinning. Eventually everyone backs up
> waiting for the transaction that never ends. If we're really lucky it
> is just GFP_KERNEL where it should NOFS.
>
> Since you're often stuck in different spots inside shrink_slab, we're
> probably not stuck on a lock. But, trying with lock debugging, lockdep
> enabled and preempt on is a good idea to rule out locking mistakes.
>
> Does the fedora debug kernel enable preempt?
No ... I've taken the Fedora 15 kernel config directly from their
install CD for this (with a bit of munging to get it to work on -rc4)
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-27 16:50 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-27 16:50 UTC (permalink / raw)
To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel
On Wed, 2011-04-27 at 12:33 -0400, Chris Mason wrote:
> Probably easier to debug with a sysrq-l and sysrq-w.
Yes, just enabling them for the repro ... I didn't echo 1
> /proc/sys/kernel/sysrq first time around
> If you get stuck
> on the filesystem, it is probably waiting on ram, which it probably
> can't get because kswapd is spinning. Eventually everyone backs up
> waiting for the transaction that never ends. If we're really lucky it
> is just GFP_KERNEL where it should NOFS.
>
> Since you're often stuck in different spots inside shrink_slab, we're
> probably not stuck on a lock. But, trying with lock debugging, lockdep
> enabled and preempt on is a good idea to rule out locking mistakes.
>
> Does the fedora debug kernel enable preempt?
No ... I've taken the Fedora 15 kernel config directly from their
install CD for this (with a bit of munging to get it to work on -rc4)
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-27 16:50 ` James Bottomley
@ 2011-04-27 16:54 ` Chris Mason
-1 siblings, 0 replies; 138+ messages in thread
From: Chris Mason @ 2011-04-27 16:54 UTC (permalink / raw)
To: James Bottomley; +Cc: linux-fsdevel, linux-mm, linux-kernel
Excerpts from James Bottomley's message of 2011-04-27 12:50:00 -0400:
> On Wed, 2011-04-27 at 12:33 -0400, Chris Mason wrote:
> > Probably easier to debug with a sysrq-l and sysrq-w.
>
> Yes, just enabling them for the repro ... I didn't echo 1
> > /proc/sys/kernel/sysrq first time around
>
> > If you get stuck
> > on the filesystem, it is probably waiting on ram, which it probably
> > can't get because kswapd is spinning. Eventually everyone backs up
> > waiting for the transaction that never ends. If we're really lucky it
> > is just GFP_KERNEL where it should NOFS.
> >
> > Since you're often stuck in different spots inside shrink_slab, we're
> > probably not stuck on a lock. But, trying with lock debugging, lockdep
> > enabled and preempt on is a good idea to rule out locking mistakes.
> >
> > Does the fedora debug kernel enable preempt?
>
> No ... I've taken the Fedora 15 kernel config directly from their
> install CD for this (with a bit of munging to get it to work on -rc4)
Ok, I'd try turning it on so we catch the sleeping with a spinlock held
case better.
-chris
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-27 16:54 ` Chris Mason
0 siblings, 0 replies; 138+ messages in thread
From: Chris Mason @ 2011-04-27 16:54 UTC (permalink / raw)
To: James Bottomley; +Cc: linux-fsdevel, linux-mm, linux-kernel
Excerpts from James Bottomley's message of 2011-04-27 12:50:00 -0400:
> On Wed, 2011-04-27 at 12:33 -0400, Chris Mason wrote:
> > Probably easier to debug with a sysrq-l and sysrq-w.
>
> Yes, just enabling them for the repro ... I didn't echo 1
> > /proc/sys/kernel/sysrq first time around
>
> > If you get stuck
> > on the filesystem, it is probably waiting on ram, which it probably
> > can't get because kswapd is spinning. Eventually everyone backs up
> > waiting for the transaction that never ends. If we're really lucky it
> > is just GFP_KERNEL where it should NOFS.
> >
> > Since you're often stuck in different spots inside shrink_slab, we're
> > probably not stuck on a lock. But, trying with lock debugging, lockdep
> > enabled and preempt on is a good idea to rule out locking mistakes.
> >
> > Does the fedora debug kernel enable preempt?
>
> No ... I've taken the Fedora 15 kernel config directly from their
> install CD for this (with a bit of munging to get it to work on -rc4)
Ok, I'd try turning it on so we catch the sleeping with a spinlock held
case better.
-chris
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-27 16:54 ` Chris Mason
(?)
@ 2011-04-27 17:21 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-27 17:21 UTC (permalink / raw)
To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel
On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
> Ok, I'd try turning it on so we catch the sleeping with a spinlock held
> case better.
Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
This is the trace with sysrq-l and sysrq-w
The repro this time doesn't have a soft lockup, just the tar is hung and
one of my CPUs is in 99% system.
James
---
[ 453.351255] SysRq : Show backtrace of all active CPUs
[ 453.352601] sending NMI to all CPUs:
[ 453.353849] NMI backtrace for cpu 3
[ 453.355545] CPU 3
[ 453.355560] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 453.363188]
[ 453.365162] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 453.367133] RIP: 0010:[<ffffffff8147af8c>] [<ffffffff8147af8c>] mutex_trylock+0x16/0x38
[ 453.369122] RSP: 0018:ffff88006dfc1d40 EFLAGS: 00000246
[ 453.371098] RAX: 0000000000000001 RBX: ffff880037de15f0 RCX: 0000000000000001
[ 453.373099] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880037875820
[ 453.375097] RBP: ffff88006dfc1d40 R08: 0000000000000000 R09: 00000000000074ad
[ 453.377079] R10: 0000000000000002 R11: ffffffff81a44e50 R12: 0000000000000000
[ 453.379052] R13: 0000000000000000 R14: ffff880037875800 R15: ffff880037875820
[ 453.381015] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000
[ 453.382985] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 453.384944] CR2: 00007fbf8ea8d090 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 453.386920] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 453.388887] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 453.390789] Process kswapd0 (pid: 46, threadinfo ffff88006dfc0000, task ffff88006dfb8000)
[ 453.392649] Stack:
[ 453.394473] ffff88006dfc1d90 ffffffffa007ff52 ffff88006dfc1d90 ffffffff811613b5
[ 453.396337] ffff88006dfc1d60 ffff880037de15f0 0000000000000000 0000000000000000
[ 453.398174] 00000000000000d0 000000000006366c ffff88006dfc1de0 ffffffff810e1f89
[ 453.400015] Call Trace:
[ 453.401845] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915]
[ 453.403702] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0
[ 453.405499] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
[ 453.407234] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 453.408952] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 453.410690] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 453.412432] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 453.414187] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 453.415945] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 453.417671] Code: 00 48 c7 47 18 00 00 00 00 f0 ff 07 7f 05 e8 ed 02 00 00 5d c3 55 48 89 e5 0f 1f 44 00 00 b9 01 00 00 00 31 d2 89 c8 f0 0f b1 17
[ 453.417905] c1 31 c0 ff c9 75 18 65 48 8b 04 25 c8 cc 00 00 48 2d d8 1f
[ 453.421534] Call Trace:
[ 453.423337] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915]
[ 453.425172] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0
[ 453.426997] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
[ 453.428818] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 453.430639] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 453.432485] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 453.434298] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 453.436112] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 453.437894] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 453.439654] NMI backtrace for cpu 2
[ 453.441508] CPU 2
[ 453.441525] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 453.449517]
[ 453.451516] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 453.453704] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
[ 453.455772] RSP: 0018:ffff8800715c9e68 EFLAGS: 00000046
[ 453.457827] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001
[ 453.459903] RDX: 0000000000000000 RSI: ffff8800715c9fd8 RDI: ffffffff81a0e640
[ 453.461999] RBP: ffff8800715c9eb8 R08: 000000000000006d R09: 00000000000003e5
[ 453.464039] R10: ffffffff00000002 R11: ffff880100293b40 R12: 0000000000000030
[ 453.466013] R13: 12187898d4512537 R14: 0000000000000004 R15: 0000000000000002
[ 453.467918] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 453.469797] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 453.471644] CR2: 0000000000452630 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 453.473512] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 453.475356] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 453.477185] Process kworker/0:1 (pid: 0, threadinfo ffff8800715c8000, task ffff8800715a1700)
[ 453.479024] Stack:
[ 453.480855] ffff8800715c9e88 ffffffff810731c0 ffff880100291290 0000000000011290
[ 453.482739] ffff8800715c9eb8 000000028139c97a ffffe8ffffc80170 ffffe8ffffc80170
[ 453.484631] ffffe8ffffc80300 0000000000000000 ffff8800715c9ef8 ffffffff8139b868
[ 453.486518] Call Trace:
[ 453.488365] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 453.490226] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.492096] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.493947] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 453.495791] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
[ 453.499958] Call Trace:
[ 453.501895] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 453.503837] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.505775] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.507687] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 453.509598] NMI backtrace for cpu 1
[ 453.511390] CPU 1
[ 453.511405] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 453.519288]
[ 453.521289] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 453.523353] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
[ 453.525377] RSP: 0018:ffff880071587e68 EFLAGS: 00000046
[ 453.527353] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
[ 453.529332] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640
[ 453.531303] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5
[ 453.533276] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010
[ 453.535249] R13: 12187898d4512ee3 R14: 0000000000000002 R15: 0000000000000001
[ 453.537229] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000
[ 453.539224] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 453.541210] CR2: 00000037d9071fa0 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 453.543220] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 453.545233] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 453.547239] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700)
[ 453.549274] Stack:
[ 453.551289] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290
[ 453.553301] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170
[ 453.555257] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868
[ 453.557156] Call Trace:
[ 453.558980] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 453.560817] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.562635] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.564438] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 453.566227] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
[ 453.570292] Call Trace:
[ 453.572196] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 453.574117] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.576031] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.577924] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 453.579811] NMI backtrace for cpu 0
[ 453.581279] CPU 0
[ 453.581289] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 453.587576]
[ 453.589160] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 453.590777] RIP: 0010:[<ffffffff8100f0fa>] [<ffffffff8100f0fa>] native_read_tsc+0x1/0x14
[ 453.592390] RSP: 0018:ffff880100203b98 EFLAGS: 00000883
[ 453.594001] RAX: 00000000f9ab6980 RBX: 0000000000002710 RCX: 0000000000000040
[ 453.595624] RDX: 000000000026066c RSI: 0000000000000100 RDI: 000000000026066d
[ 453.597250] RBP: ffff880100203ba8 R08: 000000008b000052 R09: 0000000000000000
[ 453.598872] R10: 0000000000000000 R11: 0000000000000003 R12: 000000000026066d
[ 453.600437] R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000001
[ 453.601924] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000
[ 453.603434] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 453.604942] CR2: 0000000000440360 CR3: 0000000001a03000 CR4: 00000000000406f0
[ 453.606460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 453.607981] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 453.609597] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020)
[ 453.611134] Stack:
[ 453.612655] ffff880100203ba8 ffffffff81232dfe ffff880100203bd8 ffffffff81232ecb
[ 453.614176] 0000000000002710 0000000000000008 000000000000006c 0000000000000002
[ 453.615656] ffff880100203be8 ffffffff81232e49 ffff880100203bf8 ffffffff81232e77
[ 453.617101] Call Trace:
[ 453.618492] <IRQ>
[ 453.619874] [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12
[ 453.621251] [<ffffffff81232ecb>] delay_tsc+0x27/0x74
[ 453.622613] [<ffffffff81232e49>] __delay+0xf/0x11
[ 453.623973] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e
[ 453.625322] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
[ 453.626673] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
[ 453.628027] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
[ 453.629372] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
[ 453.630706] [<ffffffff81365764>] input_pass_event+0x94/0xcc
[ 453.632028] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
[ 453.633342] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
[ 453.634655] [<ffffffff81366cf2>] input_event+0x69/0x87
[ 453.635978] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
[ 453.637283] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
[ 453.638563] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
[ 453.639830] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
[ 453.641092] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
[ 453.642354] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
[ 453.643598] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
[ 453.644832] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
[ 453.646069] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
[ 453.647299] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
[ 453.648527] [<ffffffff8147c253>] common_interrupt+0x13/0x13
[ 453.649695] <EOI>
[ 453.650794] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
[ 453.651911] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
[ 453.653025] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
[ 453.654129] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.655231] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.656329] [<ffffffff8145a91e>] rest_init+0x72/0x74
[ 453.657417] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
[ 453.658514] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
[ 453.659576] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
[ 453.660589] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
[ 453.661559] Code: 21 00 00 e8 74 3d 22 00 5d c3 90 90 90 55 40 88 f8 48 89 e5 e6 70 e4 71 5d c3 55 40 88 f0 48 89 e5 e6 70 40 88 f8 e6 71 5d c3 55
[ 453.661721] 89 e5 0f 31 89 c1 48 89 d0 48 c1 e0 20 48 09 c8 5d c3 55 b2
[ 453.663728] Call Trace:
[ 453.664713] <IRQ> [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12
[ 453.665710] [<ffffffff81232ecb>] delay_tsc+0x27/0x74
[ 453.666698] [<ffffffff81232e49>] __delay+0xf/0x11
[ 453.667686] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e
[ 453.668675] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
[ 453.669676] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
[ 453.670681] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
[ 453.671676] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
[ 453.672662] [<ffffffff81365764>] input_pass_event+0x94/0xcc
[ 453.673646] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
[ 453.674627] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
[ 453.675614] [<ffffffff81366cf2>] input_event+0x69/0x87
[ 453.676587] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
[ 453.677550] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
[ 453.678498] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
[ 453.679431] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
[ 453.680351] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
[ 453.681270] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
[ 453.682186] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
[ 453.683100] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
[ 453.684014] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
[ 453.684924] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
[ 453.685831] [<ffffffff8147c253>] common_interrupt+0x13/0x13
[ 453.686740] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
[ 453.687655] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
[ 453.688573] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
[ 453.689480] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.690387] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.691293] [<ffffffff8145a91e>] rest_init+0x72/0x74
[ 453.692195] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
[ 453.693095] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
[ 453.693996] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
[ 453.694905] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
[ 454.680802] SysRq : Show Blocked State
[ 454.683427] task PC stack pid father
[ 454.686058] systemd D 0000000000000000 0 1 0 0x00000000
[ 454.688752] ffff8801003bdcd8 0000000000000082 ffff8801003bdc88 ffffffff00000000
[ 454.691491] ffff880100370000 ffff8801003bdfd8 ffff8801003bdfd8 0000000000013b40
[ 454.694228] ffff8800715e1700 ffff880100370000 ffff8801003bdcb8 00000001811329db
[ 454.696969] Call Trace:
[ 454.699683] [<ffffffff8147ac4b>] schedule_timeout+0x34/0xde
[ 454.702447] [<ffffffff810ad618>] ? __call_rcu+0x123/0x12c
[ 454.705184] [<ffffffff810ad64d>] ? call_rcu_sched+0x15/0x17
[ 454.707906] [<ffffffff8147aa14>] wait_for_common+0xac/0x101
[ 454.710656] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226
[ 454.713445] [<ffffffff8147ab1d>] wait_for_completion+0x1d/0x1f
[ 454.716199] [<ffffffff810ada5a>] synchronize_sched+0x5a/0x5c
[ 454.718972] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43
[ 454.721766] [<ffffffff81090d05>] cgroup_diput+0x37/0xe3
[ 454.724562] [<ffffffff81090cce>] ? parse_cgroupfs_options+0x353/0x353
[ 454.727328] [<ffffffff8112fc79>] dentry_kill+0xfa/0x121
[ 454.730103] [<ffffffff81130189>] dput+0xdd/0xea
[ 454.732866] [<ffffffff8112aa68>] do_rmdir+0xc6/0xfe
[ 454.735481] [<ffffffff8111dc78>] ? filp_close+0x6e/0x7a
[ 454.737990] [<ffffffff8112b32f>] sys_rmdir+0x16/0x18
[ 454.740469] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000
[ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
[ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
[ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
[ 454.753048] Call Trace:
[ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
[ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
[ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
[ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
[ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
[ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
[ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
[ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
[ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
[ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
[ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
[ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
[ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
[ 454.791703] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
[ 454.794334] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
[ 454.796962] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
[ 454.799582] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
[ 454.802204] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
[ 454.804803] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[ 454.807427] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d
[ 454.810077] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde
[ 454.812776] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
[ 454.815464] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
[ 454.818129] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 454.820808] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 454.823452] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 454.826103] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000
[ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
[ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
[ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
[ 454.839453] Call Trace:
[ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
[ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
[ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12
[ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
[ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
[ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
[ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
[ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
[ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
[ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
[ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
[ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000
[ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282
[ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40
[ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3
[ 454.899530] Call Trace:
[ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
[ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
[ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
[ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
[ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
[ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
[ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
[ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
[ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
[ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e
[ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204
[ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a
[ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
[ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
[ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
[ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff
[ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
[ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
[ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
[ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
[ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3
[ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e
[ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 454.964732] dhclient-script D 0000000000000000 0 2856 2855 0x00000000
[ 454.967360] ffff88006e1f5b18 0000000000000082 ffff8800378da880 0000000000000000
[ 454.970056] ffff88006deb1700 ffff88006e1f5fd8 ffff88006e1f5fd8 0000000000013b40
[ 454.972706] ffff880071589700 ffff88006deb1700 ffff88006e1f5ad8 000000016e1f5ad8
[ 454.975323] Call Trace:
[ 454.977882] [<ffffffff810d5916>] ? sleep_on_page+0x12/0x12
[ 454.980477] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 454.983063] [<ffffffff810d5924>] sleep_on_page_killable+0xe/0x3b
[ 454.985622] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
[ 454.988182] [<ffffffff810d5819>] __lock_page_killable+0x66/0x68
[ 454.990785] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 454.993436] [<ffffffff810d5859>] lock_page_killable+0x3e/0x43
[ 454.996099] [<ffffffff810d71ea>] generic_file_aio_read+0x463/0x640
[ 454.998730] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
[ 455.001383] [<ffffffff811ebc34>] ? avc_has_perm+0x51/0x63
[ 455.004012] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
[ 455.006644] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
[ 455.009289] [<ffffffff8111f511>] vfs_read+0xa9/0xf0
[ 455.011917] [<ffffffff811233eb>] kernel_read+0x41/0x4f
[ 455.014511] [<ffffffff811234dd>] prepare_binprm+0xe4/0xe8
[ 455.017094] [<ffffffff81124d40>] do_execve+0x114/0x277
[ 455.019678] [<ffffffff8100ff91>] sys_execve+0x43/0x5a
[ 455.022281] [<ffffffff81482a9c>] stub_execve+0x6c/0xc0
[ 455.024888] Sched Debug Version: v0.10, 2.6.39-rc4+ #1
[ 455.027487] ktime : 455879.226285
[ 455.030142] sched_clk : 455024.886257
[ 455.032786] cpu_clk : 455024.886397
[ 455.035352] jiffies : 4295123185
[ 455.037904] sched_clock_stable : 1
[ 455.040413]
[ 455.042892] sysctl_sched
[ 455.045306] .sysctl_sched_latency : 18.000000
[ 455.047775] .sysctl_sched_min_granularity : 2.250000
[ 455.050206] .sysctl_sched_wakeup_granularity : 3.000000
[ 455.052643] .sysctl_sched_child_runs_first : 0
[ 455.055034] .sysctl_sched_features : 7279
[ 455.057423] .sysctl_sched_tunable_scaling : 1 (logaritmic)
[ 455.059829]
[ 455.059830] cpu#0, 2491.994 MHz
[ 455.064443] .nr_running : 0
[ 455.066757] .load : 0
[ 455.069054] .nr_switches : 146510
[ 455.071353] .nr_load_updates : 233084
[ 455.073642] .nr_uninterruptible : 2
[ 455.075894] .next_balance : 4295.122831
[ 455.078152] .curr->pid : 0
[ 455.080396] .clock : 454680.481348
[ 455.082634] .cpu_load[0] : 0
[ 455.084867] .cpu_load[1] : 0
[ 455.087065] .cpu_load[2] : 0
[ 455.089233] .cpu_load[3] : 0
[ 455.091390] .cpu_load[4] : 0
[ 455.093499] .yld_count : 0
[ 455.095605] .sched_switch : 0
[ 455.097667] .sched_count : 149062
[ 455.099765] .sched_goidle : 62756
[ 455.101781] .avg_idle : 1000000
[ 455.103807] .ttwu_count : 77219
[ 455.105958] .ttwu_local : 74144
[ 455.107957] .bkl_count : 0
[ 455.109914]
[ 455.109915] cfs_rq[0]:/
[ 455.113642] .exec_clock : 20017.048374
[ 455.115515] .MIN_vruntime : 0.000001
[ 455.117353] .min_vruntime : 28900.800090
[ 455.119185] .max_vruntime : 0.000001
[ 455.121028] .spread : 0.000000
[ 455.122820] .spread0 : 0.000000
[ 455.124581] .nr_spread_over : 54
[ 455.126318] .nr_running : 0
[ 455.128045] .load : 0
[ 455.129743] .load_avg : 0.000000
[ 455.131414] .load_period : 0.000000
[ 455.133052] .load_contrib : 0
[ 455.134692] .load_tg : 0
[ 455.136305]
[ 455.136306] runnable tasks:
[ 455.136307] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 455.136310] ----------------------------------------------------------------------------------------------------------
[ 455.142856]
[ 455.142857] cpu#1, 2491.994 MHz
[ 455.146215] .nr_running : 0
[ 455.147931] .load : 0
[ 455.149648] .nr_switches : 62175
[ 455.151348] .nr_load_updates : 217324
[ 455.153045] .nr_uninterruptible : 2
[ 455.154747] .next_balance : 4295.123306
[ 455.156457] .curr->pid : 0
[ 455.158149] .clock : 455157.364642
[ 455.159867] .cpu_load[0] : 0
[ 455.161594] .cpu_load[1] : 0
[ 455.163283] .cpu_load[2] : 0
[ 455.164954] .cpu_load[3] : 0
[ 455.166575] .cpu_load[4] : 0
[ 455.168185] .yld_count : 60
[ 455.169791] .sched_switch : 0
[ 455.171401] .sched_count : 62899
[ 455.172984] .sched_goidle : 27394
[ 455.174580] .avg_idle : 1000000
[ 455.176171] .ttwu_count : 30510
[ 455.177739] .ttwu_local : 25277
[ 455.179292] .bkl_count : 0
[ 455.180882]
[ 455.180883] cfs_rq[1]:/
[ 455.183954] .exec_clock : 10655.021809
[ 455.185550] .MIN_vruntime : 0.000001
[ 455.187141] .min_vruntime : 19718.135550
[ 455.188771] .max_vruntime : 0.000001
[ 455.190407] .spread : 0.000000
[ 455.192016] .spread0 : -9182.664540
[ 455.193634] .nr_spread_over : 80
[ 455.195242] .nr_running : 0
[ 455.196848] .load : 0
[ 455.198441] .load_avg : 0.000000
[ 455.200059] .load_period : 0.000000
[ 455.201654] .load_contrib : 0
[ 455.203240] .load_tg : 0
[ 455.204836]
[ 455.204837] runnable tasks:
[ 455.204838] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 455.204841] ----------------------------------------------------------------------------------------------------------
[ 455.211403]
[ 455.211404] cpu#2, 2491.994 MHz
[ 455.214958] .nr_running : 0
[ 455.216771] .load : 0
[ 455.218564] .nr_switches : 37444
[ 455.220394] .nr_load_updates : 219042
[ 455.222202] .nr_uninterruptible : 0
[ 455.224039] .next_balance : 4295.123375
[ 455.225897] .curr->pid : 0
[ 455.227769] .clock : 455227.204963
[ 455.229681] .cpu_load[0] : 0
[ 455.231548] .cpu_load[1] : 0
[ 455.233353] .cpu_load[2] : 0
[ 455.235059] .cpu_load[3] : 0
[ 455.236676] .cpu_load[4] : 0
[ 455.238280] .yld_count : 0
[ 455.239879] .sched_switch : 0
[ 455.241471] .sched_count : 37815
[ 455.243075] .sched_goidle : 16831
[ 455.244674] .avg_idle : 1000000
[ 455.246270] .ttwu_count : 18348
[ 455.247849] .ttwu_local : 16899
[ 455.249430] .bkl_count : 0
[ 455.250992]
[ 455.250993] cfs_rq[2]:/
[ 455.254040] .exec_clock : 6758.351942
[ 455.255630] .MIN_vruntime : 0.000001
[ 455.257236] .min_vruntime : 13719.942866
[ 455.258861] .max_vruntime : 0.000001
[ 455.260497] .spread : 0.000000
[ 455.262122] .spread0 : -15180.857224
[ 455.263753] .nr_spread_over : 21
[ 455.265389] .nr_running : 0
[ 455.267018] .load : 0
[ 455.268637] .load_avg : 0.000000
[ 455.270292] .load_period : 0.000000
[ 455.271911] .load_contrib : 0
[ 455.273530] .load_tg : 0
[ 455.275163]
[ 455.275165] runnable tasks:
[ 455.275166] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 455.275169] ----------------------------------------------------------------------------------------------------------
[ 455.281810]
[ 455.281811] cpu#3, 2491.994 MHz
[ 455.285349] .nr_running : 4
[ 455.287149] .load : 2048
[ 455.288952] .nr_switches : 30358
[ 455.290787] .nr_load_updates : 223302
[ 455.292599] .nr_uninterruptible : 1
[ 455.294419] .next_balance : 4295.123506
[ 455.296250] .curr->pid : 46
[ 455.298090] .clock : 431792.394314
[ 455.299953] .cpu_load[0] : 2048
[ 455.301810] .cpu_load[1] : 2048
[ 455.303610] .cpu_load[2] : 2048
[ 455.305311] .cpu_load[3] : 2048
[ 455.306926] .cpu_load[4] : 2048
[ 455.308531] .yld_count : 63
[ 455.310130] .sched_switch : 0
[ 455.311723] .sched_count : 30824
[ 455.313315] .sched_goidle : 13491
[ 455.314904] .avg_idle : 1000000
[ 455.316481] .ttwu_count : 13858
[ 455.318057] .ttwu_local : 12507
[ 455.319629] .bkl_count : 0
[ 455.321187]
[ 455.321188] cfs_rq[3]:/
[ 455.324213] .exec_clock : 5334.144946
[ 455.325795] .MIN_vruntime : 13295.262803
[ 455.327401] .min_vruntime : 13302.523317
[ 455.329046] .max_vruntime : 13295.262803
[ 455.330656] .spread : 0.000000
[ 455.332297] .spread0 : -15598.276773
[ 455.333925] .nr_spread_over : 117
[ 455.335577] .nr_running : 2
[ 455.337204] .load : 2048
[ 455.338823] .load_avg : 0.000000
[ 455.340476] .load_period : 0.000000
[ 455.342094] .load_contrib : 0
[ 455.343702] .load_tg : 0
[ 455.345302]
[ 455.345303] rt_rq[3]:/
[ 455.348416] .rt_nr_running : 1
[ 455.350006] .rt_throttled : 0
[ 455.351587] .rt_time : 0.000000
[ 455.353173] .rt_runtime : 950.000000
[ 455.354759]
[ 455.354760] runnable tasks:
[ 455.354761] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 455.354764] ----------------------------------------------------------------------------------------------------------
[ 455.361357] migration/3 17 0.000000 1084 0 0.000000 0.000933 0.000000 /
[ 455.363278] watchdog/3 20 0.000000 31 0 0.000000 0.780405 0.002351 /
[ 455.365212] R kswapd0 46 13302.523317 714 120 13302.523317 1148.767369 245855.504389 /
[ 455.367220] kworker/3:1 74 13295.262803 10488 120 13295.262803 324.924994 315669.659686 /
[ 455.369317]
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-27 17:21 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-27 17:21 UTC (permalink / raw)
To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel
On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
> Ok, I'd try turning it on so we catch the sleeping with a spinlock held
> case better.
Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
This is the trace with sysrq-l and sysrq-w
The repro this time doesn't have a soft lockup, just the tar is hung and
one of my CPUs is in 99% system.
James
---
[ 453.351255] SysRq : Show backtrace of all active CPUs
[ 453.352601] sending NMI to all CPUs:
[ 453.353849] NMI backtrace for cpu 3
[ 453.355545] CPU 3
[ 453.355560] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 453.363188]
[ 453.365162] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 453.367133] RIP: 0010:[<ffffffff8147af8c>] [<ffffffff8147af8c>] mutex_trylock+0x16/0x38
[ 453.369122] RSP: 0018:ffff88006dfc1d40 EFLAGS: 00000246
[ 453.371098] RAX: 0000000000000001 RBX: ffff880037de15f0 RCX: 0000000000000001
[ 453.373099] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880037875820
[ 453.375097] RBP: ffff88006dfc1d40 R08: 0000000000000000 R09: 00000000000074ad
[ 453.377079] R10: 0000000000000002 R11: ffffffff81a44e50 R12: 0000000000000000
[ 453.379052] R13: 0000000000000000 R14: ffff880037875800 R15: ffff880037875820
[ 453.381015] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000
[ 453.382985] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 453.384944] CR2: 00007fbf8ea8d090 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 453.386920] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 453.388887] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 453.390789] Process kswapd0 (pid: 46, threadinfo ffff88006dfc0000, task ffff88006dfb8000)
[ 453.392649] Stack:
[ 453.394473] ffff88006dfc1d90 ffffffffa007ff52 ffff88006dfc1d90 ffffffff811613b5
[ 453.396337] ffff88006dfc1d60 ffff880037de15f0 0000000000000000 0000000000000000
[ 453.398174] 00000000000000d0 000000000006366c ffff88006dfc1de0 ffffffff810e1f89
[ 453.400015] Call Trace:
[ 453.401845] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915]
[ 453.403702] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0
[ 453.405499] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
[ 453.407234] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 453.408952] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 453.410690] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 453.412432] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 453.414187] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 453.415945] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 453.417671] Code: 00 48 c7 47 18 00 00 00 00 f0 ff 07 7f 05 e8 ed 02 00 00 5d c3 55 48 89 e5 0f 1f 44 00 00 b9 01 00 00 00 31 d2 89 c8 f0 0f b1 17
[ 453.417905] c1 31 c0 ff c9 75 18 65 48 8b 04 25 c8 cc 00 00 48 2d d8 1f
[ 453.421534] Call Trace:
[ 453.423337] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915]
[ 453.425172] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0
[ 453.426997] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
[ 453.428818] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 453.430639] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 453.432485] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 453.434298] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 453.436112] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 453.437894] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 453.439654] NMI backtrace for cpu 2
[ 453.441508] CPU 2
[ 453.441525] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 453.449517]
[ 453.451516] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 453.453704] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
[ 453.455772] RSP: 0018:ffff8800715c9e68 EFLAGS: 00000046
[ 453.457827] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001
[ 453.459903] RDX: 0000000000000000 RSI: ffff8800715c9fd8 RDI: ffffffff81a0e640
[ 453.461999] RBP: ffff8800715c9eb8 R08: 000000000000006d R09: 00000000000003e5
[ 453.464039] R10: ffffffff00000002 R11: ffff880100293b40 R12: 0000000000000030
[ 453.466013] R13: 12187898d4512537 R14: 0000000000000004 R15: 0000000000000002
[ 453.467918] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 453.469797] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 453.471644] CR2: 0000000000452630 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 453.473512] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 453.475356] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 453.477185] Process kworker/0:1 (pid: 0, threadinfo ffff8800715c8000, task ffff8800715a1700)
[ 453.479024] Stack:
[ 453.480855] ffff8800715c9e88 ffffffff810731c0 ffff880100291290 0000000000011290
[ 453.482739] ffff8800715c9eb8 000000028139c97a ffffe8ffffc80170 ffffe8ffffc80170
[ 453.484631] ffffe8ffffc80300 0000000000000000 ffff8800715c9ef8 ffffffff8139b868
[ 453.486518] Call Trace:
[ 453.488365] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 453.490226] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.492096] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.493947] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 453.495791] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
[ 453.499958] Call Trace:
[ 453.501895] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 453.503837] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.505775] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.507687] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 453.509598] NMI backtrace for cpu 1
[ 453.511390] CPU 1
[ 453.511405] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 453.519288]
[ 453.521289] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 453.523353] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
[ 453.525377] RSP: 0018:ffff880071587e68 EFLAGS: 00000046
[ 453.527353] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
[ 453.529332] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640
[ 453.531303] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5
[ 453.533276] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010
[ 453.535249] R13: 12187898d4512ee3 R14: 0000000000000002 R15: 0000000000000001
[ 453.537229] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000
[ 453.539224] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 453.541210] CR2: 00000037d9071fa0 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 453.543220] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 453.545233] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 453.547239] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700)
[ 453.549274] Stack:
[ 453.551289] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290
[ 453.553301] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170
[ 453.555257] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868
[ 453.557156] Call Trace:
[ 453.558980] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 453.560817] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.562635] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.564438] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 453.566227] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
[ 453.570292] Call Trace:
[ 453.572196] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 453.574117] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.576031] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.577924] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 453.579811] NMI backtrace for cpu 0
[ 453.581279] CPU 0
[ 453.581289] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 453.587576]
[ 453.589160] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 453.590777] RIP: 0010:[<ffffffff8100f0fa>] [<ffffffff8100f0fa>] native_read_tsc+0x1/0x14
[ 453.592390] RSP: 0018:ffff880100203b98 EFLAGS: 00000883
[ 453.594001] RAX: 00000000f9ab6980 RBX: 0000000000002710 RCX: 0000000000000040
[ 453.595624] RDX: 000000000026066c RSI: 0000000000000100 RDI: 000000000026066d
[ 453.597250] RBP: ffff880100203ba8 R08: 000000008b000052 R09: 0000000000000000
[ 453.598872] R10: 0000000000000000 R11: 0000000000000003 R12: 000000000026066d
[ 453.600437] R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000001
[ 453.601924] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000
[ 453.603434] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 453.604942] CR2: 0000000000440360 CR3: 0000000001a03000 CR4: 00000000000406f0
[ 453.606460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 453.607981] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 453.609597] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020)
[ 453.611134] Stack:
[ 453.612655] ffff880100203ba8 ffffffff81232dfe ffff880100203bd8 ffffffff81232ecb
[ 453.614176] 0000000000002710 0000000000000008 000000000000006c 0000000000000002
[ 453.615656] ffff880100203be8 ffffffff81232e49 ffff880100203bf8 ffffffff81232e77
[ 453.617101] Call Trace:
[ 453.618492] <IRQ>
[ 453.619874] [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12
[ 453.621251] [<ffffffff81232ecb>] delay_tsc+0x27/0x74
[ 453.622613] [<ffffffff81232e49>] __delay+0xf/0x11
[ 453.623973] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e
[ 453.625322] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
[ 453.626673] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
[ 453.628027] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
[ 453.629372] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
[ 453.630706] [<ffffffff81365764>] input_pass_event+0x94/0xcc
[ 453.632028] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
[ 453.633342] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
[ 453.634655] [<ffffffff81366cf2>] input_event+0x69/0x87
[ 453.635978] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
[ 453.637283] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
[ 453.638563] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
[ 453.639830] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
[ 453.641092] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
[ 453.642354] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
[ 453.643598] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
[ 453.644832] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
[ 453.646069] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
[ 453.647299] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
[ 453.648527] [<ffffffff8147c253>] common_interrupt+0x13/0x13
[ 453.649695] <EOI>
[ 453.650794] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
[ 453.651911] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
[ 453.653025] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
[ 453.654129] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.655231] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.656329] [<ffffffff8145a91e>] rest_init+0x72/0x74
[ 453.657417] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
[ 453.658514] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
[ 453.659576] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
[ 453.660589] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
[ 453.661559] Code: 21 00 00 e8 74 3d 22 00 5d c3 90 90 90 55 40 88 f8 48 89 e5 e6 70 e4 71 5d c3 55 40 88 f0 48 89 e5 e6 70 40 88 f8 e6 71 5d c3 55
[ 453.661721] 89 e5 0f 31 89 c1 48 89 d0 48 c1 e0 20 48 09 c8 5d c3 55 b2
[ 453.663728] Call Trace:
[ 453.664713] <IRQ> [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12
[ 453.665710] [<ffffffff81232ecb>] delay_tsc+0x27/0x74
[ 453.666698] [<ffffffff81232e49>] __delay+0xf/0x11
[ 453.667686] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e
[ 453.668675] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
[ 453.669676] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
[ 453.670681] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
[ 453.671676] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
[ 453.672662] [<ffffffff81365764>] input_pass_event+0x94/0xcc
[ 453.673646] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
[ 453.674627] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
[ 453.675614] [<ffffffff81366cf2>] input_event+0x69/0x87
[ 453.676587] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
[ 453.677550] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
[ 453.678498] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
[ 453.679431] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
[ 453.680351] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
[ 453.681270] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
[ 453.682186] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
[ 453.683100] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
[ 453.684014] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
[ 453.684924] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
[ 453.685831] [<ffffffff8147c253>] common_interrupt+0x13/0x13
[ 453.686740] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
[ 453.687655] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
[ 453.688573] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
[ 453.689480] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.690387] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.691293] [<ffffffff8145a91e>] rest_init+0x72/0x74
[ 453.692195] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
[ 453.693095] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
[ 453.693996] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
[ 453.694905] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
[ 454.680802] SysRq : Show Blocked State
[ 454.683427] task PC stack pid father
[ 454.686058] systemd D 0000000000000000 0 1 0 0x00000000
[ 454.688752] ffff8801003bdcd8 0000000000000082 ffff8801003bdc88 ffffffff00000000
[ 454.691491] ffff880100370000 ffff8801003bdfd8 ffff8801003bdfd8 0000000000013b40
[ 454.694228] ffff8800715e1700 ffff880100370000 ffff8801003bdcb8 00000001811329db
[ 454.696969] Call Trace:
[ 454.699683] [<ffffffff8147ac4b>] schedule_timeout+0x34/0xde
[ 454.702447] [<ffffffff810ad618>] ? __call_rcu+0x123/0x12c
[ 454.705184] [<ffffffff810ad64d>] ? call_rcu_sched+0x15/0x17
[ 454.707906] [<ffffffff8147aa14>] wait_for_common+0xac/0x101
[ 454.710656] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226
[ 454.713445] [<ffffffff8147ab1d>] wait_for_completion+0x1d/0x1f
[ 454.716199] [<ffffffff810ada5a>] synchronize_sched+0x5a/0x5c
[ 454.718972] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43
[ 454.721766] [<ffffffff81090d05>] cgroup_diput+0x37/0xe3
[ 454.724562] [<ffffffff81090cce>] ? parse_cgroupfs_options+0x353/0x353
[ 454.727328] [<ffffffff8112fc79>] dentry_kill+0xfa/0x121
[ 454.730103] [<ffffffff81130189>] dput+0xdd/0xea
[ 454.732866] [<ffffffff8112aa68>] do_rmdir+0xc6/0xfe
[ 454.735481] [<ffffffff8111dc78>] ? filp_close+0x6e/0x7a
[ 454.737990] [<ffffffff8112b32f>] sys_rmdir+0x16/0x18
[ 454.740469] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000
[ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
[ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
[ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
[ 454.753048] Call Trace:
[ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
[ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
[ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
[ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
[ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
[ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
[ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
[ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
[ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
[ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
[ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
[ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
[ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
[ 454.791703] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
[ 454.794334] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
[ 454.796962] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
[ 454.799582] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
[ 454.802204] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
[ 454.804803] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[ 454.807427] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d
[ 454.810077] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde
[ 454.812776] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
[ 454.815464] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
[ 454.818129] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 454.820808] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 454.823452] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 454.826103] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000
[ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
[ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
[ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
[ 454.839453] Call Trace:
[ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
[ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
[ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12
[ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
[ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
[ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
[ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
[ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
[ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
[ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
[ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
[ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000
[ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282
[ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40
[ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3
[ 454.899530] Call Trace:
[ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
[ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
[ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
[ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
[ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
[ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
[ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
[ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
[ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
[ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e
[ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204
[ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a
[ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
[ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
[ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
[ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff
[ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
[ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
[ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
[ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
[ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3
[ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e
[ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 454.964732] dhclient-script D 0000000000000000 0 2856 2855 0x00000000
[ 454.967360] ffff88006e1f5b18 0000000000000082 ffff8800378da880 0000000000000000
[ 454.970056] ffff88006deb1700 ffff88006e1f5fd8 ffff88006e1f5fd8 0000000000013b40
[ 454.972706] ffff880071589700 ffff88006deb1700 ffff88006e1f5ad8 000000016e1f5ad8
[ 454.975323] Call Trace:
[ 454.977882] [<ffffffff810d5916>] ? sleep_on_page+0x12/0x12
[ 454.980477] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 454.983063] [<ffffffff810d5924>] sleep_on_page_killable+0xe/0x3b
[ 454.985622] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
[ 454.988182] [<ffffffff810d5819>] __lock_page_killable+0x66/0x68
[ 454.990785] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 454.993436] [<ffffffff810d5859>] lock_page_killable+0x3e/0x43
[ 454.996099] [<ffffffff810d71ea>] generic_file_aio_read+0x463/0x640
[ 454.998730] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
[ 455.001383] [<ffffffff811ebc34>] ? avc_has_perm+0x51/0x63
[ 455.004012] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
[ 455.006644] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
[ 455.009289] [<ffffffff8111f511>] vfs_read+0xa9/0xf0
[ 455.011917] [<ffffffff811233eb>] kernel_read+0x41/0x4f
[ 455.014511] [<ffffffff811234dd>] prepare_binprm+0xe4/0xe8
[ 455.017094] [<ffffffff81124d40>] do_execve+0x114/0x277
[ 455.019678] [<ffffffff8100ff91>] sys_execve+0x43/0x5a
[ 455.022281] [<ffffffff81482a9c>] stub_execve+0x6c/0xc0
[ 455.024888] Sched Debug Version: v0.10, 2.6.39-rc4+ #1
[ 455.027487] ktime : 455879.226285
[ 455.030142] sched_clk : 455024.886257
[ 455.032786] cpu_clk : 455024.886397
[ 455.035352] jiffies : 4295123185
[ 455.037904] sched_clock_stable : 1
[ 455.040413]
[ 455.042892] sysctl_sched
[ 455.045306] .sysctl_sched_latency : 18.000000
[ 455.047775] .sysctl_sched_min_granularity : 2.250000
[ 455.050206] .sysctl_sched_wakeup_granularity : 3.000000
[ 455.052643] .sysctl_sched_child_runs_first : 0
[ 455.055034] .sysctl_sched_features : 7279
[ 455.057423] .sysctl_sched_tunable_scaling : 1 (logaritmic)
[ 455.059829]
[ 455.059830] cpu#0, 2491.994 MHz
[ 455.064443] .nr_running : 0
[ 455.066757] .load : 0
[ 455.069054] .nr_switches : 146510
[ 455.071353] .nr_load_updates : 233084
[ 455.073642] .nr_uninterruptible : 2
[ 455.075894] .next_balance : 4295.122831
[ 455.078152] .curr->pid : 0
[ 455.080396] .clock : 454680.481348
[ 455.082634] .cpu_load[0] : 0
[ 455.084867] .cpu_load[1] : 0
[ 455.087065] .cpu_load[2] : 0
[ 455.089233] .cpu_load[3] : 0
[ 455.091390] .cpu_load[4] : 0
[ 455.093499] .yld_count : 0
[ 455.095605] .sched_switch : 0
[ 455.097667] .sched_count : 149062
[ 455.099765] .sched_goidle : 62756
[ 455.101781] .avg_idle : 1000000
[ 455.103807] .ttwu_count : 77219
[ 455.105958] .ttwu_local : 74144
[ 455.107957] .bkl_count : 0
[ 455.109914]
[ 455.109915] cfs_rq[0]:/
[ 455.113642] .exec_clock : 20017.048374
[ 455.115515] .MIN_vruntime : 0.000001
[ 455.117353] .min_vruntime : 28900.800090
[ 455.119185] .max_vruntime : 0.000001
[ 455.121028] .spread : 0.000000
[ 455.122820] .spread0 : 0.000000
[ 455.124581] .nr_spread_over : 54
[ 455.126318] .nr_running : 0
[ 455.128045] .load : 0
[ 455.129743] .load_avg : 0.000000
[ 455.131414] .load_period : 0.000000
[ 455.133052] .load_contrib : 0
[ 455.134692] .load_tg : 0
[ 455.136305]
[ 455.136306] runnable tasks:
[ 455.136307] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 455.136310] ----------------------------------------------------------------------------------------------------------
[ 455.142856]
[ 455.142857] cpu#1, 2491.994 MHz
[ 455.146215] .nr_running : 0
[ 455.147931] .load : 0
[ 455.149648] .nr_switches : 62175
[ 455.151348] .nr_load_updates : 217324
[ 455.153045] .nr_uninterruptible : 2
[ 455.154747] .next_balance : 4295.123306
[ 455.156457] .curr->pid : 0
[ 455.158149] .clock : 455157.364642
[ 455.159867] .cpu_load[0] : 0
[ 455.161594] .cpu_load[1] : 0
[ 455.163283] .cpu_load[2] : 0
[ 455.164954] .cpu_load[3] : 0
[ 455.166575] .cpu_load[4] : 0
[ 455.168185] .yld_count : 60
[ 455.169791] .sched_switch : 0
[ 455.171401] .sched_count : 62899
[ 455.172984] .sched_goidle : 27394
[ 455.174580] .avg_idle : 1000000
[ 455.176171] .ttwu_count : 30510
[ 455.177739] .ttwu_local : 25277
[ 455.179292] .bkl_count : 0
[ 455.180882]
[ 455.180883] cfs_rq[1]:/
[ 455.183954] .exec_clock : 10655.021809
[ 455.185550] .MIN_vruntime : 0.000001
[ 455.187141] .min_vruntime : 19718.135550
[ 455.188771] .max_vruntime : 0.000001
[ 455.190407] .spread : 0.000000
[ 455.192016] .spread0 : -9182.664540
[ 455.193634] .nr_spread_over : 80
[ 455.195242] .nr_running : 0
[ 455.196848] .load : 0
[ 455.198441] .load_avg : 0.000000
[ 455.200059] .load_period : 0.000000
[ 455.201654] .load_contrib : 0
[ 455.203240] .load_tg : 0
[ 455.204836]
[ 455.204837] runnable tasks:
[ 455.204838] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 455.204841] ----------------------------------------------------------------------------------------------------------
[ 455.211403]
[ 455.211404] cpu#2, 2491.994 MHz
[ 455.214958] .nr_running : 0
[ 455.216771] .load : 0
[ 455.218564] .nr_switches : 37444
[ 455.220394] .nr_load_updates : 219042
[ 455.222202] .nr_uninterruptible : 0
[ 455.224039] .next_balance : 4295.123375
[ 455.225897] .curr->pid : 0
[ 455.227769] .clock : 455227.204963
[ 455.229681] .cpu_load[0] : 0
[ 455.231548] .cpu_load[1] : 0
[ 455.233353] .cpu_load[2] : 0
[ 455.235059] .cpu_load[3] : 0
[ 455.236676] .cpu_load[4] : 0
[ 455.238280] .yld_count : 0
[ 455.239879] .sched_switch : 0
[ 455.241471] .sched_count : 37815
[ 455.243075] .sched_goidle : 16831
[ 455.244674] .avg_idle : 1000000
[ 455.246270] .ttwu_count : 18348
[ 455.247849] .ttwu_local : 16899
[ 455.249430] .bkl_count : 0
[ 455.250992]
[ 455.250993] cfs_rq[2]:/
[ 455.254040] .exec_clock : 6758.351942
[ 455.255630] .MIN_vruntime : 0.000001
[ 455.257236] .min_vruntime : 13719.942866
[ 455.258861] .max_vruntime : 0.000001
[ 455.260497] .spread : 0.000000
[ 455.262122] .spread0 : -15180.857224
[ 455.263753] .nr_spread_over : 21
[ 455.265389] .nr_running : 0
[ 455.267018] .load : 0
[ 455.268637] .load_avg : 0.000000
[ 455.270292] .load_period : 0.000000
[ 455.271911] .load_contrib : 0
[ 455.273530] .load_tg : 0
[ 455.275163]
[ 455.275165] runnable tasks:
[ 455.275166] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 455.275169] ----------------------------------------------------------------------------------------------------------
[ 455.281810]
[ 455.281811] cpu#3, 2491.994 MHz
[ 455.285349] .nr_running : 4
[ 455.287149] .load : 2048
[ 455.288952] .nr_switches : 30358
[ 455.290787] .nr_load_updates : 223302
[ 455.292599] .nr_uninterruptible : 1
[ 455.294419] .next_balance : 4295.123506
[ 455.296250] .curr->pid : 46
[ 455.298090] .clock : 431792.394314
[ 455.299953] .cpu_load[0] : 2048
[ 455.301810] .cpu_load[1] : 2048
[ 455.303610] .cpu_load[2] : 2048
[ 455.305311] .cpu_load[3] : 2048
[ 455.306926] .cpu_load[4] : 2048
[ 455.308531] .yld_count : 63
[ 455.310130] .sched_switch : 0
[ 455.311723] .sched_count : 30824
[ 455.313315] .sched_goidle : 13491
[ 455.314904] .avg_idle : 1000000
[ 455.316481] .ttwu_count : 13858
[ 455.318057] .ttwu_local : 12507
[ 455.319629] .bkl_count : 0
[ 455.321187]
[ 455.321188] cfs_rq[3]:/
[ 455.324213] .exec_clock : 5334.144946
[ 455.325795] .MIN_vruntime : 13295.262803
[ 455.327401] .min_vruntime : 13302.523317
[ 455.329046] .max_vruntime : 13295.262803
[ 455.330656] .spread : 0.000000
[ 455.332297] .spread0 : -15598.276773
[ 455.333925] .nr_spread_over : 117
[ 455.335577] .nr_running : 2
[ 455.337204] .load : 2048
[ 455.338823] .load_avg : 0.000000
[ 455.340476] .load_period : 0.000000
[ 455.342094] .load_contrib : 0
[ 455.343702] .load_tg : 0
[ 455.345302]
[ 455.345303] rt_rq[3]:/
[ 455.348416] .rt_nr_running : 1
[ 455.350006] .rt_throttled : 0
[ 455.351587] .rt_time : 0.000000
[ 455.353173] .rt_runtime : 950.000000
[ 455.354759]
[ 455.354760] runnable tasks:
[ 455.354761] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 455.354764] ----------------------------------------------------------------------------------------------------------
[ 455.361357] migration/3 17 0.000000 1084 0 0.000000 0.000933 0.000000 /
[ 455.363278] watchdog/3 20 0.000000 31 0 0.000000 0.780405 0.002351 /
[ 455.365212] R kswapd0 46 13302.523317 714 120 13302.523317 1148.767369 245855.504389 /
[ 455.367220] kworker/3:1 74 13295.262803 10488 120 13295.262803 324.924994 315669.659686 /
[ 455.369317]
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-27 17:21 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-27 17:21 UTC (permalink / raw)
To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel
On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
> Ok, I'd try turning it on so we catch the sleeping with a spinlock held
> case better.
Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
This is the trace with sysrq-l and sysrq-w
The repro this time doesn't have a soft lockup, just the tar is hung and
one of my CPUs is in 99% system.
James
---
[ 453.351255] SysRq : Show backtrace of all active CPUs
[ 453.352601] sending NMI to all CPUs:
[ 453.353849] NMI backtrace for cpu 3
[ 453.355545] CPU 3
[ 453.355560] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 453.363188]
[ 453.365162] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 453.367133] RIP: 0010:[<ffffffff8147af8c>] [<ffffffff8147af8c>] mutex_trylock+0x16/0x38
[ 453.369122] RSP: 0018:ffff88006dfc1d40 EFLAGS: 00000246
[ 453.371098] RAX: 0000000000000001 RBX: ffff880037de15f0 RCX: 0000000000000001
[ 453.373099] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880037875820
[ 453.375097] RBP: ffff88006dfc1d40 R08: 0000000000000000 R09: 00000000000074ad
[ 453.377079] R10: 0000000000000002 R11: ffffffff81a44e50 R12: 0000000000000000
[ 453.379052] R13: 0000000000000000 R14: ffff880037875800 R15: ffff880037875820
[ 453.381015] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000
[ 453.382985] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 453.384944] CR2: 00007fbf8ea8d090 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 453.386920] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 453.388887] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 453.390789] Process kswapd0 (pid: 46, threadinfo ffff88006dfc0000, task ffff88006dfb8000)
[ 453.392649] Stack:
[ 453.394473] ffff88006dfc1d90 ffffffffa007ff52 ffff88006dfc1d90 ffffffff811613b5
[ 453.396337] ffff88006dfc1d60 ffff880037de15f0 0000000000000000 0000000000000000
[ 453.398174] 00000000000000d0 000000000006366c ffff88006dfc1de0 ffffffff810e1f89
[ 453.400015] Call Trace:
[ 453.401845] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915]
[ 453.403702] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0
[ 453.405499] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
[ 453.407234] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 453.408952] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 453.410690] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 453.412432] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 453.414187] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 453.415945] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 453.417671] Code: 00 48 c7 47 18 00 00 00 00 f0 ff 07 7f 05 e8 ed 02 00 00 5d c3 55 48 89 e5 0f 1f 44 00 00 b9 01 00 00 00 31 d2 89 c8 f0 0f b1 17
[ 453.417905] c1 31 c0 ff c9 75 18 65 48 8b 04 25 c8 cc 00 00 48 2d d8 1f
[ 453.421534] Call Trace:
[ 453.423337] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915]
[ 453.425172] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0
[ 453.426997] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
[ 453.428818] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 453.430639] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 453.432485] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 453.434298] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 453.436112] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 453.437894] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 453.439654] NMI backtrace for cpu 2
[ 453.441508] CPU 2
[ 453.441525] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 453.449517]
[ 453.451516] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 453.453704] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
[ 453.455772] RSP: 0018:ffff8800715c9e68 EFLAGS: 00000046
[ 453.457827] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001
[ 453.459903] RDX: 0000000000000000 RSI: ffff8800715c9fd8 RDI: ffffffff81a0e640
[ 453.461999] RBP: ffff8800715c9eb8 R08: 000000000000006d R09: 00000000000003e5
[ 453.464039] R10: ffffffff00000002 R11: ffff880100293b40 R12: 0000000000000030
[ 453.466013] R13: 12187898d4512537 R14: 0000000000000004 R15: 0000000000000002
[ 453.467918] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 453.469797] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 453.471644] CR2: 0000000000452630 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 453.473512] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 453.475356] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 453.477185] Process kworker/0:1 (pid: 0, threadinfo ffff8800715c8000, task ffff8800715a1700)
[ 453.479024] Stack:
[ 453.480855] ffff8800715c9e88 ffffffff810731c0 ffff880100291290 0000000000011290
[ 453.482739] ffff8800715c9eb8 000000028139c97a ffffe8ffffc80170 ffffe8ffffc80170
[ 453.484631] ffffe8ffffc80300 0000000000000000 ffff8800715c9ef8 ffffffff8139b868
[ 453.486518] Call Trace:
[ 453.488365] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 453.490226] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.492096] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.493947] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 453.495791] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
[ 453.499958] Call Trace:
[ 453.501895] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 453.503837] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.505775] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.507687] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 453.509598] NMI backtrace for cpu 1
[ 453.511390] CPU 1
[ 453.511405] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 453.519288]
[ 453.521289] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 453.523353] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
[ 453.525377] RSP: 0018:ffff880071587e68 EFLAGS: 00000046
[ 453.527353] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
[ 453.529332] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640
[ 453.531303] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5
[ 453.533276] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010
[ 453.535249] R13: 12187898d4512ee3 R14: 0000000000000002 R15: 0000000000000001
[ 453.537229] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000
[ 453.539224] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 453.541210] CR2: 00000037d9071fa0 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 453.543220] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 453.545233] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 453.547239] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700)
[ 453.549274] Stack:
[ 453.551289] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290
[ 453.553301] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170
[ 453.555257] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868
[ 453.557156] Call Trace:
[ 453.558980] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 453.560817] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.562635] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.564438] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 453.566227] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
[ 453.570292] Call Trace:
[ 453.572196] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 453.574117] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.576031] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.577924] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 453.579811] NMI backtrace for cpu 0
[ 453.581279] CPU 0
[ 453.581289] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 453.587576]
[ 453.589160] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 453.590777] RIP: 0010:[<ffffffff8100f0fa>] [<ffffffff8100f0fa>] native_read_tsc+0x1/0x14
[ 453.592390] RSP: 0018:ffff880100203b98 EFLAGS: 00000883
[ 453.594001] RAX: 00000000f9ab6980 RBX: 0000000000002710 RCX: 0000000000000040
[ 453.595624] RDX: 000000000026066c RSI: 0000000000000100 RDI: 000000000026066d
[ 453.597250] RBP: ffff880100203ba8 R08: 000000008b000052 R09: 0000000000000000
[ 453.598872] R10: 0000000000000000 R11: 0000000000000003 R12: 000000000026066d
[ 453.600437] R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000001
[ 453.601924] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000
[ 453.603434] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 453.604942] CR2: 0000000000440360 CR3: 0000000001a03000 CR4: 00000000000406f0
[ 453.606460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 453.607981] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 453.609597] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020)
[ 453.611134] Stack:
[ 453.612655] ffff880100203ba8 ffffffff81232dfe ffff880100203bd8 ffffffff81232ecb
[ 453.614176] 0000000000002710 0000000000000008 000000000000006c 0000000000000002
[ 453.615656] ffff880100203be8 ffffffff81232e49 ffff880100203bf8 ffffffff81232e77
[ 453.617101] Call Trace:
[ 453.618492] <IRQ>
[ 453.619874] [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12
[ 453.621251] [<ffffffff81232ecb>] delay_tsc+0x27/0x74
[ 453.622613] [<ffffffff81232e49>] __delay+0xf/0x11
[ 453.623973] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e
[ 453.625322] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
[ 453.626673] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
[ 453.628027] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
[ 453.629372] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
[ 453.630706] [<ffffffff81365764>] input_pass_event+0x94/0xcc
[ 453.632028] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
[ 453.633342] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
[ 453.634655] [<ffffffff81366cf2>] input_event+0x69/0x87
[ 453.635978] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
[ 453.637283] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
[ 453.638563] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
[ 453.639830] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
[ 453.641092] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
[ 453.642354] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
[ 453.643598] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
[ 453.644832] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
[ 453.646069] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
[ 453.647299] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
[ 453.648527] [<ffffffff8147c253>] common_interrupt+0x13/0x13
[ 453.649695] <EOI>
[ 453.650794] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
[ 453.651911] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
[ 453.653025] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
[ 453.654129] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.655231] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.656329] [<ffffffff8145a91e>] rest_init+0x72/0x74
[ 453.657417] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
[ 453.658514] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
[ 453.659576] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
[ 453.660589] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
[ 453.661559] Code: 21 00 00 e8 74 3d 22 00 5d c3 90 90 90 55 40 88 f8 48 89 e5 e6 70 e4 71 5d c3 55 40 88 f0 48 89 e5 e6 70 40 88 f8 e6 71 5d c3 55
[ 453.661721] 89 e5 0f 31 89 c1 48 89 d0 48 c1 e0 20 48 09 c8 5d c3 55 b2
[ 453.663728] Call Trace:
[ 453.664713] <IRQ> [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12
[ 453.665710] [<ffffffff81232ecb>] delay_tsc+0x27/0x74
[ 453.666698] [<ffffffff81232e49>] __delay+0xf/0x11
[ 453.667686] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e
[ 453.668675] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
[ 453.669676] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
[ 453.670681] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
[ 453.671676] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
[ 453.672662] [<ffffffff81365764>] input_pass_event+0x94/0xcc
[ 453.673646] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
[ 453.674627] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
[ 453.675614] [<ffffffff81366cf2>] input_event+0x69/0x87
[ 453.676587] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
[ 453.677550] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
[ 453.678498] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
[ 453.679431] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
[ 453.680351] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
[ 453.681270] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
[ 453.682186] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
[ 453.683100] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
[ 453.684014] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
[ 453.684924] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
[ 453.685831] [<ffffffff8147c253>] common_interrupt+0x13/0x13
[ 453.686740] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
[ 453.687655] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
[ 453.688573] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
[ 453.689480] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 453.690387] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 453.691293] [<ffffffff8145a91e>] rest_init+0x72/0x74
[ 453.692195] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
[ 453.693095] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
[ 453.693996] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
[ 453.694905] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
[ 454.680802] SysRq : Show Blocked State
[ 454.683427] task PC stack pid father
[ 454.686058] systemd D 0000000000000000 0 1 0 0x00000000
[ 454.688752] ffff8801003bdcd8 0000000000000082 ffff8801003bdc88 ffffffff00000000
[ 454.691491] ffff880100370000 ffff8801003bdfd8 ffff8801003bdfd8 0000000000013b40
[ 454.694228] ffff8800715e1700 ffff880100370000 ffff8801003bdcb8 00000001811329db
[ 454.696969] Call Trace:
[ 454.699683] [<ffffffff8147ac4b>] schedule_timeout+0x34/0xde
[ 454.702447] [<ffffffff810ad618>] ? __call_rcu+0x123/0x12c
[ 454.705184] [<ffffffff810ad64d>] ? call_rcu_sched+0x15/0x17
[ 454.707906] [<ffffffff8147aa14>] wait_for_common+0xac/0x101
[ 454.710656] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226
[ 454.713445] [<ffffffff8147ab1d>] wait_for_completion+0x1d/0x1f
[ 454.716199] [<ffffffff810ada5a>] synchronize_sched+0x5a/0x5c
[ 454.718972] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43
[ 454.721766] [<ffffffff81090d05>] cgroup_diput+0x37/0xe3
[ 454.724562] [<ffffffff81090cce>] ? parse_cgroupfs_options+0x353/0x353
[ 454.727328] [<ffffffff8112fc79>] dentry_kill+0xfa/0x121
[ 454.730103] [<ffffffff81130189>] dput+0xdd/0xea
[ 454.732866] [<ffffffff8112aa68>] do_rmdir+0xc6/0xfe
[ 454.735481] [<ffffffff8111dc78>] ? filp_close+0x6e/0x7a
[ 454.737990] [<ffffffff8112b32f>] sys_rmdir+0x16/0x18
[ 454.740469] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000
[ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
[ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
[ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
[ 454.753048] Call Trace:
[ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
[ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
[ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
[ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
[ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
[ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
[ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
[ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
[ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
[ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
[ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
[ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
[ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
[ 454.791703] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
[ 454.794334] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
[ 454.796962] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
[ 454.799582] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
[ 454.802204] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
[ 454.804803] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[ 454.807427] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d
[ 454.810077] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde
[ 454.812776] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
[ 454.815464] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
[ 454.818129] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 454.820808] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 454.823452] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 454.826103] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000
[ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
[ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
[ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
[ 454.839453] Call Trace:
[ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
[ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
[ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12
[ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
[ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
[ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
[ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
[ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
[ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
[ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
[ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
[ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000
[ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282
[ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40
[ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3
[ 454.899530] Call Trace:
[ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
[ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
[ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
[ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
[ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
[ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
[ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
[ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
[ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
[ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e
[ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204
[ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a
[ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
[ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
[ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
[ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff
[ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
[ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
[ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
[ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
[ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3
[ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e
[ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 454.964732] dhclient-script D 0000000000000000 0 2856 2855 0x00000000
[ 454.967360] ffff88006e1f5b18 0000000000000082 ffff8800378da880 0000000000000000
[ 454.970056] ffff88006deb1700 ffff88006e1f5fd8 ffff88006e1f5fd8 0000000000013b40
[ 454.972706] ffff880071589700 ffff88006deb1700 ffff88006e1f5ad8 000000016e1f5ad8
[ 454.975323] Call Trace:
[ 454.977882] [<ffffffff810d5916>] ? sleep_on_page+0x12/0x12
[ 454.980477] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 454.983063] [<ffffffff810d5924>] sleep_on_page_killable+0xe/0x3b
[ 454.985622] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
[ 454.988182] [<ffffffff810d5819>] __lock_page_killable+0x66/0x68
[ 454.990785] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 454.993436] [<ffffffff810d5859>] lock_page_killable+0x3e/0x43
[ 454.996099] [<ffffffff810d71ea>] generic_file_aio_read+0x463/0x640
[ 454.998730] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
[ 455.001383] [<ffffffff811ebc34>] ? avc_has_perm+0x51/0x63
[ 455.004012] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
[ 455.006644] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
[ 455.009289] [<ffffffff8111f511>] vfs_read+0xa9/0xf0
[ 455.011917] [<ffffffff811233eb>] kernel_read+0x41/0x4f
[ 455.014511] [<ffffffff811234dd>] prepare_binprm+0xe4/0xe8
[ 455.017094] [<ffffffff81124d40>] do_execve+0x114/0x277
[ 455.019678] [<ffffffff8100ff91>] sys_execve+0x43/0x5a
[ 455.022281] [<ffffffff81482a9c>] stub_execve+0x6c/0xc0
[ 455.024888] Sched Debug Version: v0.10, 2.6.39-rc4+ #1
[ 455.027487] ktime : 455879.226285
[ 455.030142] sched_clk : 455024.886257
[ 455.032786] cpu_clk : 455024.886397
[ 455.035352] jiffies : 4295123185
[ 455.037904] sched_clock_stable : 1
[ 455.040413]
[ 455.042892] sysctl_sched
[ 455.045306] .sysctl_sched_latency : 18.000000
[ 455.047775] .sysctl_sched_min_granularity : 2.250000
[ 455.050206] .sysctl_sched_wakeup_granularity : 3.000000
[ 455.052643] .sysctl_sched_child_runs_first : 0
[ 455.055034] .sysctl_sched_features : 7279
[ 455.057423] .sysctl_sched_tunable_scaling : 1 (logaritmic)
[ 455.059829]
[ 455.059830] cpu#0, 2491.994 MHz
[ 455.064443] .nr_running : 0
[ 455.066757] .load : 0
[ 455.069054] .nr_switches : 146510
[ 455.071353] .nr_load_updates : 233084
[ 455.073642] .nr_uninterruptible : 2
[ 455.075894] .next_balance : 4295.122831
[ 455.078152] .curr->pid : 0
[ 455.080396] .clock : 454680.481348
[ 455.082634] .cpu_load[0] : 0
[ 455.084867] .cpu_load[1] : 0
[ 455.087065] .cpu_load[2] : 0
[ 455.089233] .cpu_load[3] : 0
[ 455.091390] .cpu_load[4] : 0
[ 455.093499] .yld_count : 0
[ 455.095605] .sched_switch : 0
[ 455.097667] .sched_count : 149062
[ 455.099765] .sched_goidle : 62756
[ 455.101781] .avg_idle : 1000000
[ 455.103807] .ttwu_count : 77219
[ 455.105958] .ttwu_local : 74144
[ 455.107957] .bkl_count : 0
[ 455.109914]
[ 455.109915] cfs_rq[0]:/
[ 455.113642] .exec_clock : 20017.048374
[ 455.115515] .MIN_vruntime : 0.000001
[ 455.117353] .min_vruntime : 28900.800090
[ 455.119185] .max_vruntime : 0.000001
[ 455.121028] .spread : 0.000000
[ 455.122820] .spread0 : 0.000000
[ 455.124581] .nr_spread_over : 54
[ 455.126318] .nr_running : 0
[ 455.128045] .load : 0
[ 455.129743] .load_avg : 0.000000
[ 455.131414] .load_period : 0.000000
[ 455.133052] .load_contrib : 0
[ 455.134692] .load_tg : 0
[ 455.136305]
[ 455.136306] runnable tasks:
[ 455.136307] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 455.136310] ----------------------------------------------------------------------------------------------------------
[ 455.142856]
[ 455.142857] cpu#1, 2491.994 MHz
[ 455.146215] .nr_running : 0
[ 455.147931] .load : 0
[ 455.149648] .nr_switches : 62175
[ 455.151348] .nr_load_updates : 217324
[ 455.153045] .nr_uninterruptible : 2
[ 455.154747] .next_balance : 4295.123306
[ 455.156457] .curr->pid : 0
[ 455.158149] .clock : 455157.364642
[ 455.159867] .cpu_load[0] : 0
[ 455.161594] .cpu_load[1] : 0
[ 455.163283] .cpu_load[2] : 0
[ 455.164954] .cpu_load[3] : 0
[ 455.166575] .cpu_load[4] : 0
[ 455.168185] .yld_count : 60
[ 455.169791] .sched_switch : 0
[ 455.171401] .sched_count : 62899
[ 455.172984] .sched_goidle : 27394
[ 455.174580] .avg_idle : 1000000
[ 455.176171] .ttwu_count : 30510
[ 455.177739] .ttwu_local : 25277
[ 455.179292] .bkl_count : 0
[ 455.180882]
[ 455.180883] cfs_rq[1]:/
[ 455.183954] .exec_clock : 10655.021809
[ 455.185550] .MIN_vruntime : 0.000001
[ 455.187141] .min_vruntime : 19718.135550
[ 455.188771] .max_vruntime : 0.000001
[ 455.190407] .spread : 0.000000
[ 455.192016] .spread0 : -9182.664540
[ 455.193634] .nr_spread_over : 80
[ 455.195242] .nr_running : 0
[ 455.196848] .load : 0
[ 455.198441] .load_avg : 0.000000
[ 455.200059] .load_period : 0.000000
[ 455.201654] .load_contrib : 0
[ 455.203240] .load_tg : 0
[ 455.204836]
[ 455.204837] runnable tasks:
[ 455.204838] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 455.204841] ----------------------------------------------------------------------------------------------------------
[ 455.211403]
[ 455.211404] cpu#2, 2491.994 MHz
[ 455.214958] .nr_running : 0
[ 455.216771] .load : 0
[ 455.218564] .nr_switches : 37444
[ 455.220394] .nr_load_updates : 219042
[ 455.222202] .nr_uninterruptible : 0
[ 455.224039] .next_balance : 4295.123375
[ 455.225897] .curr->pid : 0
[ 455.227769] .clock : 455227.204963
[ 455.229681] .cpu_load[0] : 0
[ 455.231548] .cpu_load[1] : 0
[ 455.233353] .cpu_load[2] : 0
[ 455.235059] .cpu_load[3] : 0
[ 455.236676] .cpu_load[4] : 0
[ 455.238280] .yld_count : 0
[ 455.239879] .sched_switch : 0
[ 455.241471] .sched_count : 37815
[ 455.243075] .sched_goidle : 16831
[ 455.244674] .avg_idle : 1000000
[ 455.246270] .ttwu_count : 18348
[ 455.247849] .ttwu_local : 16899
[ 455.249430] .bkl_count : 0
[ 455.250992]
[ 455.250993] cfs_rq[2]:/
[ 455.254040] .exec_clock : 6758.351942
[ 455.255630] .MIN_vruntime : 0.000001
[ 455.257236] .min_vruntime : 13719.942866
[ 455.258861] .max_vruntime : 0.000001
[ 455.260497] .spread : 0.000000
[ 455.262122] .spread0 : -15180.857224
[ 455.263753] .nr_spread_over : 21
[ 455.265389] .nr_running : 0
[ 455.267018] .load : 0
[ 455.268637] .load_avg : 0.000000
[ 455.270292] .load_period : 0.000000
[ 455.271911] .load_contrib : 0
[ 455.273530] .load_tg : 0
[ 455.275163]
[ 455.275165] runnable tasks:
[ 455.275166] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 455.275169] ----------------------------------------------------------------------------------------------------------
[ 455.281810]
[ 455.281811] cpu#3, 2491.994 MHz
[ 455.285349] .nr_running : 4
[ 455.287149] .load : 2048
[ 455.288952] .nr_switches : 30358
[ 455.290787] .nr_load_updates : 223302
[ 455.292599] .nr_uninterruptible : 1
[ 455.294419] .next_balance : 4295.123506
[ 455.296250] .curr->pid : 46
[ 455.298090] .clock : 431792.394314
[ 455.299953] .cpu_load[0] : 2048
[ 455.301810] .cpu_load[1] : 2048
[ 455.303610] .cpu_load[2] : 2048
[ 455.305311] .cpu_load[3] : 2048
[ 455.306926] .cpu_load[4] : 2048
[ 455.308531] .yld_count : 63
[ 455.310130] .sched_switch : 0
[ 455.311723] .sched_count : 30824
[ 455.313315] .sched_goidle : 13491
[ 455.314904] .avg_idle : 1000000
[ 455.316481] .ttwu_count : 13858
[ 455.318057] .ttwu_local : 12507
[ 455.319629] .bkl_count : 0
[ 455.321187]
[ 455.321188] cfs_rq[3]:/
[ 455.324213] .exec_clock : 5334.144946
[ 455.325795] .MIN_vruntime : 13295.262803
[ 455.327401] .min_vruntime : 13302.523317
[ 455.329046] .max_vruntime : 13295.262803
[ 455.330656] .spread : 0.000000
[ 455.332297] .spread0 : -15598.276773
[ 455.333925] .nr_spread_over : 117
[ 455.335577] .nr_running : 2
[ 455.337204] .load : 2048
[ 455.338823] .load_avg : 0.000000
[ 455.340476] .load_period : 0.000000
[ 455.342094] .load_contrib : 0
[ 455.343702] .load_tg : 0
[ 455.345302]
[ 455.345303] rt_rq[3]:/
[ 455.348416] .rt_nr_running : 1
[ 455.350006] .rt_throttled : 0
[ 455.351587] .rt_time : 0.000000
[ 455.353173] .rt_runtime : 950.000000
[ 455.354759]
[ 455.354760] runnable tasks:
[ 455.354761] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 455.354764] ----------------------------------------------------------------------------------------------------------
[ 455.361357] migration/3 17 0.000000 1084 0 0.000000 0.000933 0.000000 /
[ 455.363278] watchdog/3 20 0.000000 31 0 0.000000 0.780405 0.002351 /
[ 455.365212] R kswapd0 46 13302.523317 714 120 13302.523317 1148.767369 245855.504389 /
[ 455.367220] kworker/3:1 74 13295.262803 10488 120 13295.262803 324.924994 315669.659686 /
[ 455.369317]
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-27 17:21 ` James Bottomley
@ 2011-04-27 17:34 ` Chris Mason
-1 siblings, 0 replies; 138+ messages in thread
From: Chris Mason @ 2011-04-27 17:34 UTC (permalink / raw)
To: James Bottomley; +Cc: linux-fsdevel, linux-mm, linux-kernel, linux-ext4
[ cc'd linux-ext4 ]
James is hitting softlockups in kswapd while doing writes to a large
ext4 file.
Excerpts from James Bottomley's message of 2011-04-27 13:21:42 -0400:
> On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
> > Ok, I'd try turning it on so we catch the sleeping with a spinlock held
> > case better.
>
> Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
>
> This is the trace with sysrq-l and sysrq-w
>
> The repro this time doesn't have a soft lockup, just the tar is hung and
> one of my CPUs is in 99% system.
> [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000
> [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
> [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
> [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
> [ 454.753048] Call Trace:
> [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
> [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
> [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
> [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
> [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
> [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
> [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
> [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
> [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
> [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
So our flusher threads are stuck waiting in
jbd2_journal_get_write_access, which means they aren't cleaning dirty
pages.
In order to get write access, they probably need the transaction to
commit:
> [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000
> [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
> [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
> [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
> [ 454.839453] Call Trace:
> [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12
> [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
> [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
> [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
> [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
> [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
But that seems to be waiting for a page lock. Probably the same page
lock held by the flusher thread? Looks like tar is stuck in the same
boat down below.
> [ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
> [ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
> [ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
> [ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000
> [ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282
> [ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40
> [ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3
> [ 454.899530] Call Trace:
> [ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
> [ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> [ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
> [ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
> [ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
> [ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
> [ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
> [ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e
> [ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204
> [ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a
> [ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
> [ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
> [ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
> [ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff
> [ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
> [ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
> [ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3
> [ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e
> [ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
-chris
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-27 17:34 ` Chris Mason
0 siblings, 0 replies; 138+ messages in thread
From: Chris Mason @ 2011-04-27 17:34 UTC (permalink / raw)
To: James Bottomley; +Cc: linux-fsdevel, linux-mm, linux-kernel, linux-ext4
[ cc'd linux-ext4 ]
James is hitting softlockups in kswapd while doing writes to a large
ext4 file.
Excerpts from James Bottomley's message of 2011-04-27 13:21:42 -0400:
> On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
> > Ok, I'd try turning it on so we catch the sleeping with a spinlock held
> > case better.
>
> Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
>
> This is the trace with sysrq-l and sysrq-w
>
> The repro this time doesn't have a soft lockup, just the tar is hung and
> one of my CPUs is in 99% system.
> [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000
> [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
> [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
> [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
> [ 454.753048] Call Trace:
> [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
> [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
> [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
> [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
> [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
> [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
> [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
> [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
> [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
> [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
So our flusher threads are stuck waiting in
jbd2_journal_get_write_access, which means they aren't cleaning dirty
pages.
In order to get write access, they probably need the transaction to
commit:
> [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000
> [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
> [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
> [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
> [ 454.839453] Call Trace:
> [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12
> [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
> [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
> [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
> [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
> [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
But that seems to be waiting for a page lock. Probably the same page
lock held by the flusher thread? Looks like tar is stuck in the same
boat down below.
> [ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
> [ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
> [ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
> [ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000
> [ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282
> [ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40
> [ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3
> [ 454.899530] Call Trace:
> [ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
> [ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> [ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
> [ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
> [ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
> [ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
> [ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
> [ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e
> [ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204
> [ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a
> [ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
> [ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
> [ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
> [ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff
> [ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
> [ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
> [ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3
> [ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e
> [ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
-chris
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-27 17:34 ` Chris Mason
@ 2011-04-27 17:50 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-27 17:50 UTC (permalink / raw)
To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel, linux-ext4
On Wed, 2011-04-27 at 13:34 -0400, Chris Mason wrote:
> [ cc'd linux-ext4 ]
>
> James is hitting softlockups in kswapd while doing writes to a large
> ext4 file.
>
> Excerpts from James Bottomley's message of 2011-04-27 13:21:42 -0400:
> > On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
> > > Ok, I'd try turning it on so we catch the sleeping with a spinlock held
> > > case better.
> >
> > Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
> >
> > This is the trace with sysrq-l and sysrq-w
>
> >
> > The repro this time doesn't have a soft lockup, just the tar is hung and
> > one of my CPUs is in 99% system.
> > [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000
> > [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
> > [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
> > [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
> > [ 454.753048] Call Trace:
> > [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> > [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> > [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> > [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
> > [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> > [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
> > [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
> > [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
> > [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
> > [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
> > [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
> > [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
> > [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
> > [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
>
> So our flusher threads are stuck waiting in
> jbd2_journal_get_write_access, which means they aren't cleaning dirty
> pages.
>
> In order to get write access, they probably need the transaction to
> commit:
>
> > [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000
> > [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
> > [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
> > [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
> > [ 454.839453] Call Trace:
> > [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> > [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> > [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> > [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12
> > [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
> > [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
> > [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> > [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
> > [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
> > [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
>
> But that seems to be waiting for a page lock. Probably the same page
> lock held by the flusher thread? Looks like tar is stuck in the same
> boat down below.
To test the theory, Chris asked me to try with data=ordered.
Unfortunately, the deadlock still shows up. This is what I get.
James
---
[ 263.749738] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46]
[ 263.751207] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 263.756081] CPU 2
[ 263.756091] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 263.763033]
[ 263.764761] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 263.766509] RIP: 0010:[<ffffffff810e1fa2>] [<ffffffff810e1fa2>] shrink_slab+0x86/0x166
[ 263.768256] RSP: 0018:ffff8800709ebda0 EFLAGS: 00000206
[ 263.769976] RAX: 0000000000000000 RBX: ffff8800709ebde0 RCX: 0000000000000002
[ 263.771683] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff81a44e50
[ 263.773378] RBP: ffff8800709ebde0 R08: 0000000000000000 R09: 000000000000a2b2
[ 263.775053] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff8148300e
[ 263.776728] R13: ffff8800709ebe58 R14: 0000000000000004 R15: 00000000000178f9
[ 263.778408] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 263.780098] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 263.781777] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 263.783458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 263.785140] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 263.786811] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000)
[ 263.788491] Stack:
[ 263.790153] 000000000000003d 0000000000000080 ffff880000000000 ffff8801005e6700
[ 263.791810] ffff8801005e6000 0000000000000002 0000000000000000 000000000000000c
[ 263.793426] ffff8800709ebee0 ffffffff810e4bcc 0000000000000003 ffff88006dfa8000
[ 263.795038] Call Trace:
[ 263.796614] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 263.798183] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 263.799755] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 263.801328] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 263.802904] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 263.804434] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 263.805914] Code: 83 eb 10 e9 ce 00 00 00 44 89 f2 31 f6 48 89 df ff 13 48 63 4b 08 4c 63 e8 48 8b 45 c8 31 d2 48 f7 f1 31 d2 49 0f af c5 49 f7 f7
[ 263.806080] 03 43 20 48 85 c0 48 89 43 20 79 18 48 8b 33 48 89 c2 48 c7
[ 263.809185] Call Trace:
[ 263.810748] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 263.812327] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 263.813913] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 263.815486] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 263.817053] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 263.818621] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 272.421176] SysRq : Show backtrace of all active CPUs
[ 272.422748] sending NMI to all CPUs:
[ 272.424304] NMI backtrace for cpu 2
[ 272.426364] CPU 2
[ 272.426380] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 272.435128]
[ 272.437299] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 272.439506] RIP: 0010:[<ffffffff810d9b6e>] [<ffffffff810d9b6e>] zone_watermark_ok_safe+0xa8/0xae
[ 272.441728] RSP: 0018:ffff8800709ebd80 EFLAGS: 00000282
[ 272.443916] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 272.446101] RDX: 0000000000000813 RSI: 0000000000000000 RDI: ffff8801005e6000
[ 272.448258] RBP: ffff8800709ebd90 R08: 0000000000000000 R09: 00000000000007f0
[ 272.450362] R10: 0000000000000002 R11: 00000000000000cd R12: ffff8801005e6000
[ 272.452399] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8801005e6000
[ 272.454367] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 272.456306] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 272.458188] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 272.460021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 272.461789] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 272.463503] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000)
[ 272.465234] Stack:
[ 272.466929] 0000000000000000 0000000000000000 ffff8800709ebde0 ffffffff810e1e8a
[ 272.468675] ffff880000000002 ffffffff8106e701 ffff880000000002 ffff8801005e6000
[ 272.470406] ffff8801005f9e68 0000000000000002 0000000000000000 0000000000000000
[ 272.472144] Call Trace:
[ 272.473850] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2
[ 272.475582] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77
[ 272.477304] [<ffffffff810e47fe>] kswapd+0x165/0x798
[ 272.479015] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 272.480712] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 272.482414] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 272.484106] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 272.485809] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 272.487507] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 272.489175] Code: a6 00 48 8b 55 d8 8b 4d d0 44 8b 45 c8 7c c0 45 31 c9 4d 85 ed 4d 0f 49 cd 44 89 e6 48 89 df e8 5e f4 ff ff 48 83 c4 20 5b 41 5c
[ 272.489409] 5d 41 5e 5d c3 55 48 89 e5 0f 1f 44 00 00 bf da 00 02 00 e8
[ 272.492911] Call Trace:
[ 272.494651] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2
[ 272.496408] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77
[ 272.498173] [<ffffffff810e47fe>] kswapd+0x165/0x798
[ 272.499926] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 272.501678] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 272.503444] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 272.505198] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 272.506968] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 272.508691] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 272.510355] NMI backtrace for cpu 1
[ 272.511996] CPU 1
[ 272.512011] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 272.519303]
[ 272.521156] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 272.523190] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
[ 272.525110] RSP: 0018:ffff880071587e68 EFLAGS: 00000046
[ 272.527018] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
[ 272.528960] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640
[ 272.530906] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5
[ 272.532800] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010
[ 272.534634] R13: 12187a34a107f726 R14: 0000000000000002 R15: 0000000000000001
[ 272.536399] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000
[ 272.538145] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 272.539879] CR2: 00000037da2830e4 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 272.541614] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 272.543329] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 272.545026] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700)
[ 272.546756] Stack:
[ 272.548457] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290
[ 272.550225] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170
[ 272.552009] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868
[ 272.553780] Call Trace:
[ 272.555514] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 272.557259] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 272.559005] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 272.560755] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 272.562485] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
[ 272.566380] Call Trace:
[ 272.568193] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 272.570031] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 272.571855] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 272.573658] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 272.575463] NMI backtrace for cpu 0
[ 272.576865] CPU 0
[ 272.576875] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 272.582932]
[ 272.584474] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 272.586057] RIP: 0010:[<ffffffff81232e6e>] [<ffffffff81232e6e>] __const_udelay+0x23/0x2e
[ 272.587604] RSP: 0018:ffff880100203bf8 EFLAGS: 00000887
[ 272.589120] RAX: 00000000fa1c3100 RBX: 0000000000002710 RCX: 0000000000000040
[ 272.590635] RDX: 000000000026074e RSI: 0000000000000100 RDI: 0000000000418958
[ 272.592139] RBP: ffff880100203bf8 R08: 000000008b000052 R09: 0000000000000000
[ 272.593644] R10: 0000000000000000 R11: 0000000000000003 R12: 0000000000000008
[ 272.595143] R13: 000000000000006c R14: 0000000000000002 R15: 0000000000000001
[ 272.596635] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000
[ 272.598139] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 272.599637] CR2: 00000037d909e270 CR3: 000000006f2d7000 CR4: 00000000000406f0
[ 272.601151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 272.602669] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 272.604187] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020)
[ 272.605715] Stack:
[ 272.607238] ffff880100203c18 ffffffff8102166e 0000000000000000 ffffffff81a5e690
[ 272.608756] ffff880100203c28 ffffffff812be0ad ffff880100203c68 ffffffff812be310
[ 272.610226] ffff880037d01900 0000000000000026 ffff880037d01900 0000000000000001
[ 272.611653] Call Trace:
[ 272.613026] <IRQ>
[ 272.614388] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
[ 272.615761] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
[ 272.617118] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
[ 272.618465] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
[ 272.619814] [<ffffffff81365764>] input_pass_event+0x94/0xcc
[ 272.621158] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
[ 272.622497] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
[ 272.623844] [<ffffffff81366cf2>] input_event+0x69/0x87
[ 272.625172] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
[ 272.626489] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
[ 272.627791] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
[ 272.629085] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
[ 272.630383] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
[ 272.631675] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
[ 272.632946] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
[ 272.634205] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
[ 272.635455] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
[ 272.636699] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
[ 272.637927] [<ffffffff8147c253>] common_interrupt+0x13/0x13
[ 272.639155] <EOI>
[ 272.640374] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
[ 272.641612] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
[ 272.642840] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
[ 272.644012] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 272.645115] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 272.646215] [<ffffffff8145a91e>] rest_init+0x72/0x74
[ 272.647313] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
[ 272.648411] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
[ 272.649514] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
[ 272.650624] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
[ 272.651725] Code: ff 15 37 2b 82 00 5d c3 55 48 89 e5 0f 1f 44 00 00 65 48 8b 14 25 98 3a 01 00 48 8d 04 bd 00 00 00 00 48 69 d2 fa 00 00 00 f7 e2
[ 272.651891] 8d 7a 01 e8 c3 ff ff ff 5d c3 55 48 89 e5 0f 1f 44 00 00 48
[ 272.654210] Call Trace:
[ 272.655298] <IRQ> [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
[ 272.656372] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
[ 272.657421] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
[ 272.658456] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
[ 272.659480] [<ffffffff81365764>] input_pass_event+0x94/0xcc
[ 272.660486] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
[ 272.661487] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
[ 272.662497] [<ffffffff81366cf2>] input_event+0x69/0x87
[ 272.663502] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
[ 272.664519] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
[ 272.665525] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
[ 272.666522] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
[ 272.667509] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
[ 272.668492] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
[ 272.669467] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
[ 272.670445] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
[ 272.671408] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
[ 272.672357] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
[ 272.673284] [<ffffffff8147c253>] common_interrupt+0x13/0x13
[ 272.674225] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
[ 272.675166] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
[ 272.676098] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
[ 272.677032] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 272.677951] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 272.678865] [<ffffffff8145a91e>] rest_init+0x72/0x74
[ 272.679770] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
[ 272.680685] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
[ 272.681603] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
[ 272.682526] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
[ 272.683442] NMI backtrace for cpu 3
[ 272.684688] CPU 3
[ 272.684705] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 272.690650]
[ 272.692143] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 272.693726] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
[ 272.695312] RSP: 0018:ffff8800715dfe68 EFLAGS: 00000046
[ 272.696892] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001
[ 272.698508] RDX: 0000000000000000 RSI: ffff8800715dffd8 RDI: ffffffff81a0e640
[ 272.700117] RBP: ffff8800715dfeb8 R08: 000000000000006d R09: 00000000000003e4
[ 272.701740] R10: ffffffff00000003 R11: ffff8801002d3b40 R12: 0000000000000030
[ 272.703369] R13: 12187a34a107edc5 R14: 0000000000000004 R15: 0000000000000003
[ 272.704985] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000
[ 272.706626] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 272.708277] CR2: 00000037d904c480 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 272.709952] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 272.711633] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 272.713299] Process kworker/0:1 (pid: 0, threadinfo ffff8800715de000, task ffff8800715e1700)
[ 272.715001] Stack:
[ 272.716681] ffff8800715dfe88 ffffffff810731c0 ffff8801002d1290 0000000000011290
[ 272.718411] ffff8800715dfeb8 000000038139c97a ffffe8ffffcc0170 ffffe8ffffcc0170
[ 272.720153] ffffe8ffffcc0300 0000000000000000 ffff8800715dfef8 ffffffff8139b868
[ 272.721848] Call Trace:
[ 272.723465] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 272.725095] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 272.726712] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 272.728304] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 272.729891] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
[ 272.733594] Call Trace:
[ 272.735317] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 272.737068] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 272.738815] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 272.740554] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 274.865072] SysRq : Show Blocked State
[ 274.866350] task PC stack pid father
[ 274.867634] jbd2/dm-1-8 D 0000000000000000 0 363 2 0x00000000
[ 274.868930] ffff880037d05ba0 0000000000000046 ffff880037d05b30 ffffffff00000000
[ 274.870242] ffff880037acc500 ffff880037d05fd8 ffff880037d05fd8 0000000000013b40
[ 274.871555] ffffffff81a0b020 ffff880037acc500 ffff8801002143c0 00000001005bd1d0
[ 274.872869] Call Trace:
[ 274.874168] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.875485] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.876791] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 274.878089] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
[ 274.879382] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
[ 274.880684] [<ffffffff8147af4e>] out_of_line_wait_on_bit+0x72/0x7d
[ 274.881986] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.883290] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 274.884601] [<ffffffff811421c6>] __wait_on_buffer+0x26/0x28
[ 274.885912] [<ffffffff811c8a3c>] wait_on_buffer+0x35/0x39
[ 274.887221] [<ffffffff811c9633>] jbd2_journal_commit_transaction+0xab3/0x1194
[ 274.888547] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
[ 274.889864] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
[ 274.891178] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[ 274.892500] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
[ 274.893776] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
[ 274.895001] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 274.896179] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
[ 274.897328] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 274.898465] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 274.899591] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 274.900708] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 274.901813] flush-253:1 D 0000000000000000 0 419 2 0x00000000
[ 274.902938] ffff8800706bf760 0000000000000046 ffff8800706bf6f0 ffffffff00000000
[ 274.904067] ffff880037afc500 ffff8800706bffd8 ffff8800706bffd8 0000000000013b40
[ 274.905198] ffffffff81a0b020 ffff880037afc500 ffff8800706bf720 00000001706bf720
[ 274.906327] Call Trace:
[ 274.907430] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 274.908536] [<ffffffff81217b00>] get_request_wait+0x102/0x18b
[ 274.909633] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 274.910729] [<ffffffff81218841>] __make_request+0x18a/0x2b8
[ 274.911826] [<ffffffff81217538>] generic_make_request+0x2a9/0x323
[ 274.912908] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17
[ 274.913979] [<ffffffff81217690>] submit_bio+0xde/0xfd
[ 274.915040] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3
[ 274.916104] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29
[ 274.917161] [<ffffffff81141c42>] submit_bh+0xe6/0x105
[ 274.918208] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7
[ 274.919261] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
[ 274.920306] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
[ 274.921361] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
[ 274.922411] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
[ 274.923396] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97
[ 274.924324] [<ffffffff81144873>] block_write_full_page+0x15/0x17
[ 274.925248] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a
[ 274.926168] [<ffffffff810dd3c1>] __writepage+0x15/0x2e
[ 274.927083] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330
[ 274.928001] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33
[ 274.928923] [<ffffffff8104e4c1>] ? find_busiest_group+0x253/0x8b9
[ 274.929834] [<ffffffff81198ba9>] ? ext4_da_writepages+0x85/0x44d
[ 274.930744] [<ffffffff810dd363>] generic_writepages+0x40/0x56
[ 274.931646] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
[ 274.932536] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
[ 274.933430] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
[ 274.934317] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
[ 274.935194] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
[ 274.936061] [<ffffffff81045c65>] ? hrtick_update+0x32/0x34
[ 274.936941] [<ffffffff810dd79a>] ? global_dirty_limits+0x2b/0xd1
[ 274.936943] [<ffffffff8113d8cd>] wb_do_writeback+0x17f/0x19d
[ 274.936945] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
[ 274.936947] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
[ 274.936949] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 274.936951] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 274.936953] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 274.936956] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 274.936958] jbd2/dm-2-8 D ffff88006e17c700 0 787 2 0x00000000
[ 274.936960] ffff88006ffe5cd0 0000000000000046 0000000000000000 0000000000000000
[ 274.936962] ffff88006f0e0000 ffff88006ffe5fd8 ffff88006ffe5fd8 0000000000013b40
[ 274.936964] ffff8800715fc500 ffff88006f0e0000 ffff88006ffe5cd0 ffffffff8106e7c3
[ 274.936966] Call Trace:
[ 274.936968] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
[ 274.936970] [<ffffffff811c8d44>] jbd2_journal_commit_transaction+0x1c4/0x1194
[ 274.936972] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61
[ 274.936974] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd
[ 274.936977] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 274.936979] [<ffffffff8100804e>] ? load_TLS+0x10/0x14
[ 274.936981] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
[ 274.936983] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 274.936985] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
[ 274.936987] [<ffffffff8147be8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 274.936989] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81
[ 274.936991] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
[ 274.936993] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 274.936995] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
[ 274.936997] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 274.936998] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 274.937001] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 274.937003] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 274.937004] flush-253:2 D ffff88006d5a0000 0 851 2 0x00000000
[ 274.937006] ffff88006e45d760 0000000000000046 ffff880037a30000 0000000000000246
[ 274.937008] ffff88006d5a0000 ffff88006e45dfd8 ffff88006e45dfd8 0000000000013b40
[ 274.937010] ffff880071588000 ffff88006d5a0000 ffff88006e45d720 ffff88006e45d720
[ 274.937011] Call Trace:
[ 274.937013] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 274.937015] [<ffffffff81217b00>] get_request_wait+0x102/0x18b
[ 274.937017] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 274.937019] [<ffffffff81218841>] __make_request+0x18a/0x2b8
[ 274.937020] [<ffffffff81217538>] generic_make_request+0x2a9/0x323
[ 274.937022] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17
[ 274.937024] [<ffffffff81217690>] submit_bio+0xde/0xfd
[ 274.937026] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3
[ 274.937028] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29
[ 274.937030] [<ffffffff81141c42>] submit_bh+0xe6/0x105
[ 274.937031] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7
[ 274.937033] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
[ 274.937035] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
[ 274.937037] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
[ 274.937039] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
[ 274.937040] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97
[ 274.937042] [<ffffffff81144873>] block_write_full_page+0x15/0x17
[ 274.937044] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a
[ 274.937046] [<ffffffff810dd3c1>] __writepage+0x15/0x2e
[ 274.937048] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330
[ 274.937049] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33
[ 274.937052] [<ffffffff810dd363>] generic_writepages+0x40/0x56
[ 274.937054] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
[ 274.937055] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
[ 274.937057] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
[ 274.937059] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
[ 274.937061] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
[ 274.937063] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
[ 274.937065] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[ 274.937067] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d
[ 274.937069] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde
[ 274.937071] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
[ 274.937073] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
[ 274.937075] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 274.937077] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 274.937079] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 274.937081] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 274.937082] tar D 0000000000000000 0 980 826 0x00000000
[ 274.937084] ffff88006f19f9e8 0000000000000086 00000000ffff9fd7 0000000000000000
[ 274.937086] ffff88006d670000 ffff88006f19ffd8 ffff88006f19ffd8 0000000000013b40
[ 274.937088] ffffffff81a0b020 ffff88006d670000 ffff88006f19f9b8 0000000100000282
[ 274.937090] Call Trace:
[ 274.937091] [<ffffffff8147acbe>] schedule_timeout+0xa7/0xde
[ 274.937093] [<ffffffff8106015c>] ? del_timer+0x7a/0x7a
[ 274.937095] [<ffffffff8147abc2>] io_schedule_timeout+0x6f/0x98
[ 274.937097] [<ffffffff810ddc26>] balance_dirty_pages_ratelimited_nr+0x341/0x3b6
[ 274.937100] [<ffffffff810d5efb>] generic_file_buffered_write+0x1dc/0x23a
[ 274.937102] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
[ 274.937105] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
[ 274.937108] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
[ 274.937111] [<ffffffff8111edab>] do_sync_write+0xbf/0xff
[ 274.937112] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
[ 274.937115] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
[ 274.937118] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
[ 274.937120] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
[ 274.937122] [<ffffffff8111f421>] vfs_write+0xac/0xf3
[ 274.937123] [<ffffffff8111f610>] sys_write+0x4a/0x6e
[ 274.937126] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 274.937128] make D 0000000000000000 0 2252 2143 0x00000004
[ 274.937130] ffff88006d631948 0000000000000082 ffff88006d6318d8 ffffffff00000000
[ 274.937132] ffff880037d9dc00 ffff88006d631fd8 ffff88006d631fd8 0000000000013b40
[ 274.937133] ffffffff81a0b020 ffff880037d9dc00 ffff88006d631908 000000016d631908
[ 274.937135] Call Trace:
[ 274.937137] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.937139] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 274.937140] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
[ 274.937142] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
[ 274.937144] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d
[ 274.937146] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.937148] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 274.937149] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c
[ 274.937151] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d
[ 274.937153] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d
[ 274.937155] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d
[ 274.937156] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
[ 274.937158] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
[ 274.937161] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
[ 274.937163] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
[ 274.937165] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
[ 274.937167] [<ffffffff8147c515>] ? page_fault+0x25/0x30
[ 274.937169] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
[ 274.937171] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
[ 274.937173] [<ffffffff811322c7>] touch_atime+0x10e/0x131
[ 274.937176] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640
[ 274.937178] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
[ 274.937180] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
[ 274.937181] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
[ 274.937183] [<ffffffff8111f511>] vfs_read+0xa9/0xf0
[ 274.937185] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e
[ 274.937187] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 274.937188] make D 0000000000000000 0 2262 2254 0x00000004
[ 274.937190] ffff880070b2b948 0000000000000082 ffff880070b2b8d8 ffffffff00000000
[ 274.937192] ffff88006d428000 ffff880070b2bfd8 ffff880070b2bfd8 0000000000013b40
[ 274.937194] ffffffff81a0b020 ffff88006d428000 ffff880070b2b908 0000000170b2b908
[ 274.937195] Call Trace:
[ 274.937197] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.937199] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 274.937200] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
[ 274.937202] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
[ 274.937204] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d
[ 274.937205] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.937208] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 274.937209] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c
[ 274.937211] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d
[ 274.937212] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d
[ 274.937214] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d
[ 274.937216] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
[ 274.937217] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
[ 274.937219] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
[ 274.937221] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
[ 274.937223] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
[ 274.937225] [<ffffffff8147c515>] ? page_fault+0x25/0x30
[ 274.937227] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
[ 274.937228] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
[ 274.937230] [<ffffffff811322c7>] touch_atime+0x10e/0x131
[ 274.937232] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640
[ 274.937234] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
[ 274.937236] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
[ 274.937238] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
[ 274.937240] [<ffffffff8111f511>] vfs_read+0xa9/0xf0
[ 274.937241] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e
[ 274.937243] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 274.937244] sh D 0000000000000000 0 2270 2253 0x00000004
[ 274.937246] ffff8800709fd918 0000000000000086 ffff8800709fd8a8 ffffffff00000000
[ 274.937248] ffff880037f60000 ffff8800709fdfd8 ffff8800709fdfd8 0000000000013b40
[ 274.937250] ffff880071589700 ffff880037f60000 ffff8801002543c0 00000001005b89d0
[ 274.937252] Call Trace:
[ 274.937253] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.937255] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 274.937256] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
[ 274.937258] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
[ 274.937260] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d
[ 274.937261] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.937264] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 274.937265] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c
[ 274.937267] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d
[ 274.937268] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d
[ 274.937270] [<ffffffff81196fe4>] ? ext4_getblk+0x8e/0x153
[ 274.937272] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
[ 274.937274] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
[ 274.937276] [<ffffffff8119c0ce>] ext4_add_entry+0x11f/0x8cf
[ 274.937277] [<ffffffff811429f3>] ? __brelse+0x15/0x33
[ 274.937280] [<ffffffff81191bac>] ? ext4_new_inode+0xc63/0xd0a
[ 274.937282] [<ffffffff8119c89c>] ext4_add_nondir+0x1e/0x67
[ 274.937283] [<ffffffff8119cb1b>] ext4_create+0xf5/0x13e
[ 274.937286] [<ffffffff81129b12>] vfs_create+0x6c/0x8e
[ 274.937288] [<ffffffff81129dad>] do_last+0x279/0x5ab
[ 274.937289] [<ffffffff8112ac73>] path_openat+0xc8/0x31c
[ 274.937291] [<ffffffff810f2332>] ? handle_mm_fault+0x1ac/0x1bf
[ 274.937293] [<ffffffff8112aeff>] do_filp_open+0x38/0x86
[ 274.937295] [<ffffffff81233de1>] ? might_fault+0x21/0x23
[ 274.937298] [<ffffffff8113477b>] ? alloc_fd+0x72/0x11d
[ 274.937299] [<ffffffff8111e995>] do_sys_open+0x6e/0x100
[ 274.937301] [<ffffffff8111ea47>] sys_open+0x20/0x22
[ 274.937303] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 274.937304] gcc D 0000000000000000 0 2272 2271 0x00000004
[ 274.937306] ffff88006f197c58 0000000000000086 ffff88006f197d14 0000000000000000
[ 274.937307] ffff880037f64500 ffff88006f197fd8 ffff88006f197fd8 0000000000013b40
[ 274.937309] ffffffff81a0b020 ffff880037f64500 ffff88006f197c58 000000018106e7c3
[ 274.937311] Call Trace:
[ 274.937313] [<ffffffff811c6fd5>] start_this_handle+0x2e8/0x465
[ 274.937315] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 274.937317] [<ffffffff811c7418>] jbd2__journal_start+0x94/0xda
[ 274.937318] [<ffffffff811c7471>] jbd2_journal_start+0x13/0x15
[ 274.937320] [<ffffffff811ab92a>] ext4_journal_start_sb+0x108/0x120
[ 274.937323] [<ffffffff8119cc24>] ext4_symlink+0xc0/0x212
[ 274.937325] [<ffffffff81129928>] vfs_symlink+0x54/0x74
[ 274.937326] [<ffffffff8112b40a>] sys_symlinkat+0x96/0xef
[ 274.937329] [<ffffffff81122d89>] ? sys_newstat+0x2a/0x33
[ 274.937330] [<ffffffff8112b479>] sys_symlink+0x16/0x18
[ 274.937332] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 274.937335] Sched Debug Version: v0.10, 2.6.39-rc4+ #1
[ 274.937336] ktime : 275382.978028
[ 274.937337] sched_clk : 274937.334155
[ 274.937338] cpu_clk : 274937.334189
[ 274.937339] jiffies : 4294942678
[ 274.937340] sched_clock_stable : 1
[ 274.937341]
[ 274.937342] sysctl_sched
[ 274.937342] .sysctl_sched_latency : 18.000000
[ 274.937344] .sysctl_sched_min_granularity : 2.250000
[ 274.937345] .sysctl_sched_wakeup_granularity : 3.000000
[ 274.937346] .sysctl_sched_child_runs_first : 0
[ 274.937347] .sysctl_sched_features : 7279
[ 274.937348] .sysctl_sched_tunable_scaling : 1 (logaritmic)
[ 274.937349]
[ 274.937350] cpu#0, 2492.220 MHz
[ 274.937350] .nr_running : 0
[ 274.937351] .load : 0
[ 274.937352] .nr_switches : 105039
[ 274.937353] .nr_load_updates : 121182
[ 274.937354] .nr_uninterruptible : 8
[ 274.937355] .next_balance : 4294.942607
[ 274.937356] .curr->pid : 0
[ 274.937357] .clock : 274864.523786
[ 274.937359] .cpu_load[0] : 0
[ 274.937360] .cpu_load[1] : 0
[ 274.937361] .cpu_load[2] : 0
[ 274.937362] .cpu_load[3] : 0
[ 274.937363] .cpu_load[4] : 0
[ 274.937364] .yld_count : 0
[ 274.937364] .sched_switch : 0
[ 274.937365] .sched_count : 107487
[ 274.937366] .sched_goidle : 43368
[ 274.937367] .avg_idle : 1000000
[ 274.937368] .ttwu_count : 56534
[ 274.937369] .ttwu_local : 53925
[ 274.937370] .bkl_count : 0
[ 274.937371]
[ 274.937372] cfs_rq[0]:/system
[ 274.937372] .exec_clock : 5906.036519
[ 274.937374] .MIN_vruntime : 0.000001
[ 274.937375] .min_vruntime : 7138.254657
[ 274.937376] .max_vruntime : 0.000001
[ 274.937377] .spread : 0.000000
[ 274.937378] .spread0 : -14302.149798
[ 274.937379] .nr_spread_over : 0
[ 274.937379] .nr_running : 0
[ 274.937380] .load : 0
[ 274.937381] .load_avg : 159.999997
[ 274.937382] .load_period : 5.496191
[ 274.937383] .load_contrib : 29
[ 274.937384] .load_tg : 29
[ 274.937385] .se->exec_start : 272446.423795
[ 274.937386] .se->vruntime : 21431.398168
[ 274.937387] .se->sum_exec_runtime : 5906.111805
[ 274.937388] .se->statistics.wait_start : 0.000000
[ 274.937389] .se->statistics.sleep_start : 0.000000
[ 274.937390] .se->statistics.block_start : 0.000000
[ 274.937391] .se->statistics.sleep_max : 0.000000
[ 274.937392] .se->statistics.block_max : 0.000000
[ 274.937393] .se->statistics.exec_max : 15.266617
[ 274.937394] .se->statistics.slice_max : 5.258344
[ 274.937395] .se->statistics.wait_max : 7.305679
[ 274.937396] .se->statistics.wait_sum : 190.506686
[ 274.937397] .se->statistics.wait_count : 18946
[ 274.937398] .se->load.weight : 2
[ 274.937399]
[ 274.937399] cfs_rq[0]:/
[ 274.937400] .exec_clock : 17409.691339
[ 274.937401] .MIN_vruntime : 0.000001
[ 274.937402] .min_vruntime : 21440.404455
[ 274.937403] .max_vruntime : 0.000001
[ 274.937403] .spread : 0.000000
[ 274.937404] .spread0 : 0.000000
[ 274.937405] .nr_spread_over : 58
[ 274.937406] .nr_running : 0
[ 274.937407] .load : 0
[ 274.937408] .load_avg : 0.000000
[ 274.937409] .load_period : 0.000000
[ 274.937409] .load_contrib : 0
[ 274.937410] .load_tg : 0
[ 274.937411]
[ 274.937412] cfs_rq[0]:/system/rescue.service
[ 274.937413] .exec_clock : 4319.868509
[ 274.937414] .MIN_vruntime : 0.000001
[ 274.937414] .min_vruntime : 4255.887524
[ 274.937415] .max_vruntime : 0.000001
[ 274.937416] .spread : 0.000000
[ 274.937417] .spread0 : -17184.516931
[ 274.937418] .nr_spread_over : 24
[ 274.937419] .nr_running : 0
[ 274.937420] .load : 0
[ 274.937421] .load_avg : 160.004112
[ 274.937421] .load_period : 5.496149
[ 274.937422] .load_contrib : 29
[ 274.937423] .load_tg : 29
[ 274.937424] .se->exec_start : 272446.423795
[ 274.937425] .se->vruntime : 7138.254657
[ 274.937426] .se->sum_exec_runtime : 4319.796162
[ 274.937427] .se->statistics.wait_start : 0.000000
[ 274.937428] .se->statistics.sleep_start : 0.000000
[ 274.937429] .se->statistics.block_start : 0.000000
[ 274.937430] .se->statistics.sleep_max : 0.000000
[ 274.937431] .se->statistics.block_max : 0.000000
[ 274.937432] .se->statistics.exec_max : 6.213083
[ 274.937432] .se->statistics.slice_max : 7.894226
[ 274.937433] .se->statistics.wait_max : 4.870567
[ 274.937434] .se->statistics.wait_sum : 91.737803
[ 274.937435] .se->statistics.wait_count : 15504
[ 274.937436] .se->load.weight : 2
[ 274.937437]
[ 274.937438] runnable tasks:
[ 274.937438] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 274.937439] ----------------------------------------------------------------------------------------------------------
[ 274.937444]
[ 274.937445] cpu#1, 2492.220 MHz
[ 274.937445] .nr_running : 0
[ 274.937446] .load : 0
[ 274.937447] .nr_switches : 35274
[ 274.937448] .nr_load_updates : 105259
[ 274.937449] .nr_uninterruptible : 1
[ 274.937449] .next_balance : 4294.942681
[ 274.937450] .curr->pid : 0
[ 274.937451] .clock : 274937.357922
[ 274.937452] .cpu_load[0] : 0
[ 274.937453] .cpu_load[1] : 0
[ 274.937454] .cpu_load[2] : 0
[ 274.937454] .cpu_load[3] : 0
[ 274.937455] .cpu_load[4] : 0
[ 274.937456] .yld_count : 0
[ 274.937457] .sched_switch : 0
[ 274.937458] .sched_count : 35560
[ 274.937458] .sched_goidle : 14561
[ 274.937459] .avg_idle : 1000000
[ 274.937460] .ttwu_count : 16938
[ 274.937461] .ttwu_local : 14491
[ 274.937462] .bkl_count : 0
[ 274.937463]
[ 274.937463] cfs_rq[1]:/
[ 274.937464] .exec_clock : 5221.725565
[ 274.937465] .MIN_vruntime : 0.000001
[ 274.937466] .min_vruntime : 9077.562937
[ 274.937466] .max_vruntime : 0.000001
[ 274.937467] .spread : 0.000000
[ 274.937468] .spread0 : -12362.841518
[ 274.937469] .nr_spread_over : 69
[ 274.937470] .nr_running : 0
[ 274.937471] .load : 0
[ 274.937472] .load_avg : 0.000000
[ 274.937473] .load_period : 0.000000
[ 274.937473] .load_contrib : 0
[ 274.937474] .load_tg : 0
[ 274.937475]
[ 274.937476] runnable tasks:
[ 274.937476] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 274.937477] ----------------------------------------------------------------------------------------------------------
[ 274.937481]
[ 274.937482] cpu#2, 2492.220 MHz
[ 274.937482] .nr_running : 4
[ 274.937483] .load : 2048
[ 274.937484] .nr_switches : 7092
[ 274.937485] .nr_load_updates : 100502
[ 274.937486] .nr_uninterruptible : 0
[ 274.937487] .next_balance : 4294.942806
[ 274.937487] .curr->pid : 46
[ 274.937488] .clock : 183206.600407
[ 274.937489] .cpu_load[0] : 2048
[ 274.937490] .cpu_load[1] : 2048
[ 274.937491] .cpu_load[2] : 2048
[ 274.937492] .cpu_load[3] : 2048
[ 274.937492] .cpu_load[4] : 2048
[ 274.937493] .yld_count : 0
[ 274.937494] .sched_switch : 0
[ 274.937495] .sched_count : 7323
[ 274.937496] .sched_goidle : 2831
[ 274.937496] .avg_idle : 1000000
[ 274.937497] .ttwu_count : 3275
[ 274.937498] .ttwu_local : 2028
[ 274.937499] .bkl_count : 0
[ 274.937500]
[ 274.937500] cfs_rq[2]:/
[ 274.937501] .exec_clock : 2187.192139
[ 274.937502] .MIN_vruntime : 5109.505143
[ 274.937503] .min_vruntime : 5118.505143
[ 274.937504] .max_vruntime : 5109.505143
[ 274.937505] .spread : 0.000000
[ 274.937506] .spread0 : -16321.899312
[ 274.937506] .nr_spread_over : 31
[ 274.937507] .nr_running : 2
[ 274.937508] .load : 2048
[ 274.937509] .load_avg : 0.000000
[ 274.937510] .load_period : 0.000000
[ 274.937511] .load_contrib : 0
[ 274.937511] .load_tg : 0
[ 274.937512]
[ 274.937513] rt_rq[2]:/
[ 274.937513] .rt_nr_running : 1
[ 274.937514] .rt_throttled : 0
[ 274.937515] .rt_time : 0.000000
[ 274.937516] .rt_runtime : 950.000000
[ 274.937517]
[ 274.937517] runnable tasks:
[ 274.937518] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 274.937519] ----------------------------------------------------------------------------------------------------------
[ 274.937520] migration/2 13 0.000000 113 0 0.000000 0.000815 0.000000 /
[ 274.937524] watchdog/2 16 0.000000 17 0 0.000000 1.785026 0.002124 /
[ 274.937527] R kswapd0 46 5118.505143 370 120 5118.505143 415.766870 181943.841718 /
[ 274.937531] kworker/2:1 76 5109.505143 1083 120 5109.505143 25.615305 181183.059001 /
[ 274.937535]
[ 274.937536] cpu#3, 2492.220 MHz
[ 274.937536] .nr_running : 1
[ 274.937537] .load : 1024
[ 274.937538] .nr_switches : 22324
[ 274.937539] .nr_load_updates : 59855
[ 274.937540] .nr_uninterruptible : 0
[ 274.937541] .next_balance : 4294.942679
[ 274.937542] .curr->pid : 73
[ 274.937543] .clock : 274936.938702
[ 274.937543] .cpu_load[0] : 0
[ 274.937544] .cpu_load[1] : 0
[ 274.937545] .cpu_load[2] : 0
[ 274.937546] .cpu_load[3] : 0
[ 274.937546] .cpu_load[4] : 0
[ 274.937547] .yld_count : 123
[ 274.937548] .sched_switch : 0
[ 274.937549] .sched_count : 22774
[ 274.937550] .sched_goidle : 9952
[ 274.937551] .avg_idle : 946660
[ 274.937551] .ttwu_count : 10075
[ 274.937552] .ttwu_local : 8964
[ 274.937553] .bkl_count : 0
[ 274.937554]
[ 274.937554] cfs_rq[3]:/
[ 274.937555] .exec_clock : 3244.912177
[ 274.937556] .MIN_vruntime : 0.000001
[ 274.937557] .min_vruntime : 7517.482154
[ 274.937558] .max_vruntime : 0.000001
[ 274.937559] .spread : 0.000000
[ 274.937559] .spread0 : -13922.922301
[ 274.937560] .nr_spread_over : 128
[ 274.937561] .nr_running : 1
[ 274.937562] .load : 1024
[ 274.937563] .load_avg : 0.000000
[ 274.937564] .load_period : 0.000000
[ 274.937565] .load_contrib : 0
[ 274.937565] .load_tg : 0
[ 274.937567]
[ 274.937567] runnable tasks:
[ 274.937567] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 274.937568] ----------------------------------------------------------------------------------------------------------
[ 274.937571] R kworker/3:1 73 7517.482154 7294 120 7517.482154 264.876541 272702.297840 /
[ 274.937575]
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-27 17:50 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-27 17:50 UTC (permalink / raw)
To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel, linux-ext4
On Wed, 2011-04-27 at 13:34 -0400, Chris Mason wrote:
> [ cc'd linux-ext4 ]
>
> James is hitting softlockups in kswapd while doing writes to a large
> ext4 file.
>
> Excerpts from James Bottomley's message of 2011-04-27 13:21:42 -0400:
> > On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
> > > Ok, I'd try turning it on so we catch the sleeping with a spinlock held
> > > case better.
> >
> > Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
> >
> > This is the trace with sysrq-l and sysrq-w
>
> >
> > The repro this time doesn't have a soft lockup, just the tar is hung and
> > one of my CPUs is in 99% system.
> > [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000
> > [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
> > [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
> > [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
> > [ 454.753048] Call Trace:
> > [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> > [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> > [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> > [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
> > [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> > [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
> > [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
> > [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
> > [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
> > [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
> > [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
> > [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
> > [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
> > [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
>
> So our flusher threads are stuck waiting in
> jbd2_journal_get_write_access, which means they aren't cleaning dirty
> pages.
>
> In order to get write access, they probably need the transaction to
> commit:
>
> > [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000
> > [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
> > [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
> > [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
> > [ 454.839453] Call Trace:
> > [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> > [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> > [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> > [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12
> > [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
> > [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
> > [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> > [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
> > [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
> > [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
>
> But that seems to be waiting for a page lock. Probably the same page
> lock held by the flusher thread? Looks like tar is stuck in the same
> boat down below.
To test the theory, Chris asked me to try with data=ordered.
Unfortunately, the deadlock still shows up. This is what I get.
James
---
[ 263.749738] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46]
[ 263.751207] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 263.756081] CPU 2
[ 263.756091] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 263.763033]
[ 263.764761] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 263.766509] RIP: 0010:[<ffffffff810e1fa2>] [<ffffffff810e1fa2>] shrink_slab+0x86/0x166
[ 263.768256] RSP: 0018:ffff8800709ebda0 EFLAGS: 00000206
[ 263.769976] RAX: 0000000000000000 RBX: ffff8800709ebde0 RCX: 0000000000000002
[ 263.771683] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff81a44e50
[ 263.773378] RBP: ffff8800709ebde0 R08: 0000000000000000 R09: 000000000000a2b2
[ 263.775053] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff8148300e
[ 263.776728] R13: ffff8800709ebe58 R14: 0000000000000004 R15: 00000000000178f9
[ 263.778408] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 263.780098] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 263.781777] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 263.783458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 263.785140] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 263.786811] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000)
[ 263.788491] Stack:
[ 263.790153] 000000000000003d 0000000000000080 ffff880000000000 ffff8801005e6700
[ 263.791810] ffff8801005e6000 0000000000000002 0000000000000000 000000000000000c
[ 263.793426] ffff8800709ebee0 ffffffff810e4bcc 0000000000000003 ffff88006dfa8000
[ 263.795038] Call Trace:
[ 263.796614] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 263.798183] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 263.799755] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 263.801328] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 263.802904] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 263.804434] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 263.805914] Code: 83 eb 10 e9 ce 00 00 00 44 89 f2 31 f6 48 89 df ff 13 48 63 4b 08 4c 63 e8 48 8b 45 c8 31 d2 48 f7 f1 31 d2 49 0f af c5 49 f7 f7
[ 263.806080] 03 43 20 48 85 c0 48 89 43 20 79 18 48 8b 33 48 89 c2 48 c7
[ 263.809185] Call Trace:
[ 263.810748] [<ffffffff810e4bcc>] kswapd+0x533/0x798
[ 263.812327] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 263.813913] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 263.815486] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 263.817053] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 263.818621] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 272.421176] SysRq : Show backtrace of all active CPUs
[ 272.422748] sending NMI to all CPUs:
[ 272.424304] NMI backtrace for cpu 2
[ 272.426364] CPU 2
[ 272.426380] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 272.435128]
[ 272.437299] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 272.439506] RIP: 0010:[<ffffffff810d9b6e>] [<ffffffff810d9b6e>] zone_watermark_ok_safe+0xa8/0xae
[ 272.441728] RSP: 0018:ffff8800709ebd80 EFLAGS: 00000282
[ 272.443916] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 272.446101] RDX: 0000000000000813 RSI: 0000000000000000 RDI: ffff8801005e6000
[ 272.448258] RBP: ffff8800709ebd90 R08: 0000000000000000 R09: 00000000000007f0
[ 272.450362] R10: 0000000000000002 R11: 00000000000000cd R12: ffff8801005e6000
[ 272.452399] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8801005e6000
[ 272.454367] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 272.456306] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 272.458188] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 272.460021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 272.461789] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 272.463503] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000)
[ 272.465234] Stack:
[ 272.466929] 0000000000000000 0000000000000000 ffff8800709ebde0 ffffffff810e1e8a
[ 272.468675] ffff880000000002 ffffffff8106e701 ffff880000000002 ffff8801005e6000
[ 272.470406] ffff8801005f9e68 0000000000000002 0000000000000000 0000000000000000
[ 272.472144] Call Trace:
[ 272.473850] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2
[ 272.475582] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77
[ 272.477304] [<ffffffff810e47fe>] kswapd+0x165/0x798
[ 272.479015] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 272.480712] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 272.482414] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 272.484106] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 272.485809] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 272.487507] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 272.489175] Code: a6 00 48 8b 55 d8 8b 4d d0 44 8b 45 c8 7c c0 45 31 c9 4d 85 ed 4d 0f 49 cd 44 89 e6 48 89 df e8 5e f4 ff ff 48 83 c4 20 5b 41 5c
[ 272.489409] 5d 41 5e 5d c3 55 48 89 e5 0f 1f 44 00 00 bf da 00 02 00 e8
[ 272.492911] Call Trace:
[ 272.494651] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2
[ 272.496408] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77
[ 272.498173] [<ffffffff810e47fe>] kswapd+0x165/0x798
[ 272.499926] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 272.501678] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 272.503444] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 272.505198] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 272.506968] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 272.508691] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 272.510355] NMI backtrace for cpu 1
[ 272.511996] CPU 1
[ 272.512011] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 272.519303]
[ 272.521156] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 272.523190] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
[ 272.525110] RSP: 0018:ffff880071587e68 EFLAGS: 00000046
[ 272.527018] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
[ 272.528960] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640
[ 272.530906] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5
[ 272.532800] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010
[ 272.534634] R13: 12187a34a107f726 R14: 0000000000000002 R15: 0000000000000001
[ 272.536399] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000
[ 272.538145] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 272.539879] CR2: 00000037da2830e4 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 272.541614] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 272.543329] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 272.545026] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700)
[ 272.546756] Stack:
[ 272.548457] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290
[ 272.550225] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170
[ 272.552009] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868
[ 272.553780] Call Trace:
[ 272.555514] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 272.557259] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 272.559005] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 272.560755] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 272.562485] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
[ 272.566380] Call Trace:
[ 272.568193] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 272.570031] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 272.571855] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 272.573658] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 272.575463] NMI backtrace for cpu 0
[ 272.576865] CPU 0
[ 272.576875] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 272.582932]
[ 272.584474] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 272.586057] RIP: 0010:[<ffffffff81232e6e>] [<ffffffff81232e6e>] __const_udelay+0x23/0x2e
[ 272.587604] RSP: 0018:ffff880100203bf8 EFLAGS: 00000887
[ 272.589120] RAX: 00000000fa1c3100 RBX: 0000000000002710 RCX: 0000000000000040
[ 272.590635] RDX: 000000000026074e RSI: 0000000000000100 RDI: 0000000000418958
[ 272.592139] RBP: ffff880100203bf8 R08: 000000008b000052 R09: 0000000000000000
[ 272.593644] R10: 0000000000000000 R11: 0000000000000003 R12: 0000000000000008
[ 272.595143] R13: 000000000000006c R14: 0000000000000002 R15: 0000000000000001
[ 272.596635] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000
[ 272.598139] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 272.599637] CR2: 00000037d909e270 CR3: 000000006f2d7000 CR4: 00000000000406f0
[ 272.601151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 272.602669] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 272.604187] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020)
[ 272.605715] Stack:
[ 272.607238] ffff880100203c18 ffffffff8102166e 0000000000000000 ffffffff81a5e690
[ 272.608756] ffff880100203c28 ffffffff812be0ad ffff880100203c68 ffffffff812be310
[ 272.610226] ffff880037d01900 0000000000000026 ffff880037d01900 0000000000000001
[ 272.611653] Call Trace:
[ 272.613026] <IRQ>
[ 272.614388] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
[ 272.615761] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
[ 272.617118] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
[ 272.618465] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
[ 272.619814] [<ffffffff81365764>] input_pass_event+0x94/0xcc
[ 272.621158] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
[ 272.622497] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
[ 272.623844] [<ffffffff81366cf2>] input_event+0x69/0x87
[ 272.625172] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
[ 272.626489] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
[ 272.627791] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
[ 272.629085] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
[ 272.630383] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
[ 272.631675] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
[ 272.632946] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
[ 272.634205] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
[ 272.635455] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
[ 272.636699] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
[ 272.637927] [<ffffffff8147c253>] common_interrupt+0x13/0x13
[ 272.639155] <EOI>
[ 272.640374] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
[ 272.641612] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
[ 272.642840] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
[ 272.644012] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 272.645115] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 272.646215] [<ffffffff8145a91e>] rest_init+0x72/0x74
[ 272.647313] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
[ 272.648411] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
[ 272.649514] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
[ 272.650624] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
[ 272.651725] Code: ff 15 37 2b 82 00 5d c3 55 48 89 e5 0f 1f 44 00 00 65 48 8b 14 25 98 3a 01 00 48 8d 04 bd 00 00 00 00 48 69 d2 fa 00 00 00 f7 e2
[ 272.651891] 8d 7a 01 e8 c3 ff ff ff 5d c3 55 48 89 e5 0f 1f 44 00 00 48
[ 272.654210] Call Trace:
[ 272.655298] <IRQ> [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
[ 272.656372] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
[ 272.657421] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
[ 272.658456] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
[ 272.659480] [<ffffffff81365764>] input_pass_event+0x94/0xcc
[ 272.660486] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
[ 272.661487] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
[ 272.662497] [<ffffffff81366cf2>] input_event+0x69/0x87
[ 272.663502] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
[ 272.664519] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
[ 272.665525] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
[ 272.666522] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
[ 272.667509] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
[ 272.668492] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
[ 272.669467] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
[ 272.670445] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
[ 272.671408] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
[ 272.672357] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
[ 272.673284] [<ffffffff8147c253>] common_interrupt+0x13/0x13
[ 272.674225] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
[ 272.675166] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
[ 272.676098] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
[ 272.677032] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 272.677951] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 272.678865] [<ffffffff8145a91e>] rest_init+0x72/0x74
[ 272.679770] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
[ 272.680685] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
[ 272.681603] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
[ 272.682526] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
[ 272.683442] NMI backtrace for cpu 3
[ 272.684688] CPU 3
[ 272.684705] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 272.690650]
[ 272.692143] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
[ 272.693726] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
[ 272.695312] RSP: 0018:ffff8800715dfe68 EFLAGS: 00000046
[ 272.696892] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001
[ 272.698508] RDX: 0000000000000000 RSI: ffff8800715dffd8 RDI: ffffffff81a0e640
[ 272.700117] RBP: ffff8800715dfeb8 R08: 000000000000006d R09: 00000000000003e4
[ 272.701740] R10: ffffffff00000003 R11: ffff8801002d3b40 R12: 0000000000000030
[ 272.703369] R13: 12187a34a107edc5 R14: 0000000000000004 R15: 0000000000000003
[ 272.704985] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000
[ 272.706626] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 272.708277] CR2: 00000037d904c480 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 272.709952] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 272.711633] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 272.713299] Process kworker/0:1 (pid: 0, threadinfo ffff8800715de000, task ffff8800715e1700)
[ 272.715001] Stack:
[ 272.716681] ffff8800715dfe88 ffffffff810731c0 ffff8801002d1290 0000000000011290
[ 272.718411] ffff8800715dfeb8 000000038139c97a ffffe8ffffcc0170 ffffe8ffffcc0170
[ 272.720153] ffffe8ffffcc0300 0000000000000000 ffff8800715dfef8 ffffffff8139b868
[ 272.721848] Call Trace:
[ 272.723465] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 272.725095] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 272.726712] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 272.728304] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 272.729891] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
[ 272.733594] Call Trace:
[ 272.735317] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 272.737068] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
[ 272.738815] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 272.740554] [<ffffffff8146ae57>] start_secondary+0x223/0x225
[ 274.865072] SysRq : Show Blocked State
[ 274.866350] task PC stack pid father
[ 274.867634] jbd2/dm-1-8 D 0000000000000000 0 363 2 0x00000000
[ 274.868930] ffff880037d05ba0 0000000000000046 ffff880037d05b30 ffffffff00000000
[ 274.870242] ffff880037acc500 ffff880037d05fd8 ffff880037d05fd8 0000000000013b40
[ 274.871555] ffffffff81a0b020 ffff880037acc500 ffff8801002143c0 00000001005bd1d0
[ 274.872869] Call Trace:
[ 274.874168] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.875485] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.876791] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 274.878089] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
[ 274.879382] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
[ 274.880684] [<ffffffff8147af4e>] out_of_line_wait_on_bit+0x72/0x7d
[ 274.881986] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.883290] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 274.884601] [<ffffffff811421c6>] __wait_on_buffer+0x26/0x28
[ 274.885912] [<ffffffff811c8a3c>] wait_on_buffer+0x35/0x39
[ 274.887221] [<ffffffff811c9633>] jbd2_journal_commit_transaction+0xab3/0x1194
[ 274.888547] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
[ 274.889864] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
[ 274.891178] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[ 274.892500] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
[ 274.893776] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
[ 274.895001] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 274.896179] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
[ 274.897328] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 274.898465] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 274.899591] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 274.900708] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 274.901813] flush-253:1 D 0000000000000000 0 419 2 0x00000000
[ 274.902938] ffff8800706bf760 0000000000000046 ffff8800706bf6f0 ffffffff00000000
[ 274.904067] ffff880037afc500 ffff8800706bffd8 ffff8800706bffd8 0000000000013b40
[ 274.905198] ffffffff81a0b020 ffff880037afc500 ffff8800706bf720 00000001706bf720
[ 274.906327] Call Trace:
[ 274.907430] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 274.908536] [<ffffffff81217b00>] get_request_wait+0x102/0x18b
[ 274.909633] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 274.910729] [<ffffffff81218841>] __make_request+0x18a/0x2b8
[ 274.911826] [<ffffffff81217538>] generic_make_request+0x2a9/0x323
[ 274.912908] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17
[ 274.913979] [<ffffffff81217690>] submit_bio+0xde/0xfd
[ 274.915040] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3
[ 274.916104] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29
[ 274.917161] [<ffffffff81141c42>] submit_bh+0xe6/0x105
[ 274.918208] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7
[ 274.919261] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
[ 274.920306] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
[ 274.921361] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
[ 274.922411] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
[ 274.923396] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97
[ 274.924324] [<ffffffff81144873>] block_write_full_page+0x15/0x17
[ 274.925248] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a
[ 274.926168] [<ffffffff810dd3c1>] __writepage+0x15/0x2e
[ 274.927083] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330
[ 274.928001] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33
[ 274.928923] [<ffffffff8104e4c1>] ? find_busiest_group+0x253/0x8b9
[ 274.929834] [<ffffffff81198ba9>] ? ext4_da_writepages+0x85/0x44d
[ 274.930744] [<ffffffff810dd363>] generic_writepages+0x40/0x56
[ 274.931646] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
[ 274.932536] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
[ 274.933430] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
[ 274.934317] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
[ 274.935194] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
[ 274.936061] [<ffffffff81045c65>] ? hrtick_update+0x32/0x34
[ 274.936941] [<ffffffff810dd79a>] ? global_dirty_limits+0x2b/0xd1
[ 274.936943] [<ffffffff8113d8cd>] wb_do_writeback+0x17f/0x19d
[ 274.936945] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
[ 274.936947] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
[ 274.936949] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 274.936951] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 274.936953] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 274.936956] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 274.936958] jbd2/dm-2-8 D ffff88006e17c700 0 787 2 0x00000000
[ 274.936960] ffff88006ffe5cd0 0000000000000046 0000000000000000 0000000000000000
[ 274.936962] ffff88006f0e0000 ffff88006ffe5fd8 ffff88006ffe5fd8 0000000000013b40
[ 274.936964] ffff8800715fc500 ffff88006f0e0000 ffff88006ffe5cd0 ffffffff8106e7c3
[ 274.936966] Call Trace:
[ 274.936968] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
[ 274.936970] [<ffffffff811c8d44>] jbd2_journal_commit_transaction+0x1c4/0x1194
[ 274.936972] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61
[ 274.936974] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd
[ 274.936977] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 274.936979] [<ffffffff8100804e>] ? load_TLS+0x10/0x14
[ 274.936981] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
[ 274.936983] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 274.936985] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
[ 274.936987] [<ffffffff8147be8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 274.936989] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81
[ 274.936991] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
[ 274.936993] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 274.936995] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
[ 274.936997] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 274.936998] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 274.937001] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 274.937003] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 274.937004] flush-253:2 D ffff88006d5a0000 0 851 2 0x00000000
[ 274.937006] ffff88006e45d760 0000000000000046 ffff880037a30000 0000000000000246
[ 274.937008] ffff88006d5a0000 ffff88006e45dfd8 ffff88006e45dfd8 0000000000013b40
[ 274.937010] ffff880071588000 ffff88006d5a0000 ffff88006e45d720 ffff88006e45d720
[ 274.937011] Call Trace:
[ 274.937013] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 274.937015] [<ffffffff81217b00>] get_request_wait+0x102/0x18b
[ 274.937017] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 274.937019] [<ffffffff81218841>] __make_request+0x18a/0x2b8
[ 274.937020] [<ffffffff81217538>] generic_make_request+0x2a9/0x323
[ 274.937022] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17
[ 274.937024] [<ffffffff81217690>] submit_bio+0xde/0xfd
[ 274.937026] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3
[ 274.937028] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29
[ 274.937030] [<ffffffff81141c42>] submit_bh+0xe6/0x105
[ 274.937031] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7
[ 274.937033] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
[ 274.937035] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
[ 274.937037] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
[ 274.937039] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
[ 274.937040] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97
[ 274.937042] [<ffffffff81144873>] block_write_full_page+0x15/0x17
[ 274.937044] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a
[ 274.937046] [<ffffffff810dd3c1>] __writepage+0x15/0x2e
[ 274.937048] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330
[ 274.937049] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33
[ 274.937052] [<ffffffff810dd363>] generic_writepages+0x40/0x56
[ 274.937054] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
[ 274.937055] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
[ 274.937057] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
[ 274.937059] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
[ 274.937061] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
[ 274.937063] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
[ 274.937065] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[ 274.937067] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d
[ 274.937069] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde
[ 274.937071] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
[ 274.937073] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
[ 274.937075] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 274.937077] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
[ 274.937079] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 274.937081] [<ffffffff81483760>] ? gs_change+0x13/0x13
[ 274.937082] tar D 0000000000000000 0 980 826 0x00000000
[ 274.937084] ffff88006f19f9e8 0000000000000086 00000000ffff9fd7 0000000000000000
[ 274.937086] ffff88006d670000 ffff88006f19ffd8 ffff88006f19ffd8 0000000000013b40
[ 274.937088] ffffffff81a0b020 ffff88006d670000 ffff88006f19f9b8 0000000100000282
[ 274.937090] Call Trace:
[ 274.937091] [<ffffffff8147acbe>] schedule_timeout+0xa7/0xde
[ 274.937093] [<ffffffff8106015c>] ? del_timer+0x7a/0x7a
[ 274.937095] [<ffffffff8147abc2>] io_schedule_timeout+0x6f/0x98
[ 274.937097] [<ffffffff810ddc26>] balance_dirty_pages_ratelimited_nr+0x341/0x3b6
[ 274.937100] [<ffffffff810d5efb>] generic_file_buffered_write+0x1dc/0x23a
[ 274.937102] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
[ 274.937105] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
[ 274.937108] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
[ 274.937111] [<ffffffff8111edab>] do_sync_write+0xbf/0xff
[ 274.937112] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
[ 274.937115] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
[ 274.937118] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
[ 274.937120] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
[ 274.937122] [<ffffffff8111f421>] vfs_write+0xac/0xf3
[ 274.937123] [<ffffffff8111f610>] sys_write+0x4a/0x6e
[ 274.937126] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 274.937128] make D 0000000000000000 0 2252 2143 0x00000004
[ 274.937130] ffff88006d631948 0000000000000082 ffff88006d6318d8 ffffffff00000000
[ 274.937132] ffff880037d9dc00 ffff88006d631fd8 ffff88006d631fd8 0000000000013b40
[ 274.937133] ffffffff81a0b020 ffff880037d9dc00 ffff88006d631908 000000016d631908
[ 274.937135] Call Trace:
[ 274.937137] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.937139] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 274.937140] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
[ 274.937142] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
[ 274.937144] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d
[ 274.937146] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.937148] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 274.937149] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c
[ 274.937151] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d
[ 274.937153] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d
[ 274.937155] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d
[ 274.937156] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
[ 274.937158] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
[ 274.937161] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
[ 274.937163] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
[ 274.937165] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
[ 274.937167] [<ffffffff8147c515>] ? page_fault+0x25/0x30
[ 274.937169] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
[ 274.937171] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
[ 274.937173] [<ffffffff811322c7>] touch_atime+0x10e/0x131
[ 274.937176] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640
[ 274.937178] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
[ 274.937180] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
[ 274.937181] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
[ 274.937183] [<ffffffff8111f511>] vfs_read+0xa9/0xf0
[ 274.937185] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e
[ 274.937187] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 274.937188] make D 0000000000000000 0 2262 2254 0x00000004
[ 274.937190] ffff880070b2b948 0000000000000082 ffff880070b2b8d8 ffffffff00000000
[ 274.937192] ffff88006d428000 ffff880070b2bfd8 ffff880070b2bfd8 0000000000013b40
[ 274.937194] ffffffff81a0b020 ffff88006d428000 ffff880070b2b908 0000000170b2b908
[ 274.937195] Call Trace:
[ 274.937197] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.937199] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 274.937200] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
[ 274.937202] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
[ 274.937204] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d
[ 274.937205] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.937208] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 274.937209] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c
[ 274.937211] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d
[ 274.937212] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d
[ 274.937214] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d
[ 274.937216] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
[ 274.937217] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
[ 274.937219] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
[ 274.937221] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
[ 274.937223] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
[ 274.937225] [<ffffffff8147c515>] ? page_fault+0x25/0x30
[ 274.937227] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
[ 274.937228] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
[ 274.937230] [<ffffffff811322c7>] touch_atime+0x10e/0x131
[ 274.937232] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640
[ 274.937234] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
[ 274.937236] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
[ 274.937238] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
[ 274.937240] [<ffffffff8111f511>] vfs_read+0xa9/0xf0
[ 274.937241] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e
[ 274.937243] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 274.937244] sh D 0000000000000000 0 2270 2253 0x00000004
[ 274.937246] ffff8800709fd918 0000000000000086 ffff8800709fd8a8 ffffffff00000000
[ 274.937248] ffff880037f60000 ffff8800709fdfd8 ffff8800709fdfd8 0000000000013b40
[ 274.937250] ffff880071589700 ffff880037f60000 ffff8801002543c0 00000001005b89d0
[ 274.937252] Call Trace:
[ 274.937253] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.937255] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
[ 274.937256] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
[ 274.937258] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
[ 274.937260] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d
[ 274.937261] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
[ 274.937264] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
[ 274.937265] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c
[ 274.937267] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d
[ 274.937268] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d
[ 274.937270] [<ffffffff81196fe4>] ? ext4_getblk+0x8e/0x153
[ 274.937272] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
[ 274.937274] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
[ 274.937276] [<ffffffff8119c0ce>] ext4_add_entry+0x11f/0x8cf
[ 274.937277] [<ffffffff811429f3>] ? __brelse+0x15/0x33
[ 274.937280] [<ffffffff81191bac>] ? ext4_new_inode+0xc63/0xd0a
[ 274.937282] [<ffffffff8119c89c>] ext4_add_nondir+0x1e/0x67
[ 274.937283] [<ffffffff8119cb1b>] ext4_create+0xf5/0x13e
[ 274.937286] [<ffffffff81129b12>] vfs_create+0x6c/0x8e
[ 274.937288] [<ffffffff81129dad>] do_last+0x279/0x5ab
[ 274.937289] [<ffffffff8112ac73>] path_openat+0xc8/0x31c
[ 274.937291] [<ffffffff810f2332>] ? handle_mm_fault+0x1ac/0x1bf
[ 274.937293] [<ffffffff8112aeff>] do_filp_open+0x38/0x86
[ 274.937295] [<ffffffff81233de1>] ? might_fault+0x21/0x23
[ 274.937298] [<ffffffff8113477b>] ? alloc_fd+0x72/0x11d
[ 274.937299] [<ffffffff8111e995>] do_sys_open+0x6e/0x100
[ 274.937301] [<ffffffff8111ea47>] sys_open+0x20/0x22
[ 274.937303] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 274.937304] gcc D 0000000000000000 0 2272 2271 0x00000004
[ 274.937306] ffff88006f197c58 0000000000000086 ffff88006f197d14 0000000000000000
[ 274.937307] ffff880037f64500 ffff88006f197fd8 ffff88006f197fd8 0000000000013b40
[ 274.937309] ffffffff81a0b020 ffff880037f64500 ffff88006f197c58 000000018106e7c3
[ 274.937311] Call Trace:
[ 274.937313] [<ffffffff811c6fd5>] start_this_handle+0x2e8/0x465
[ 274.937315] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 274.937317] [<ffffffff811c7418>] jbd2__journal_start+0x94/0xda
[ 274.937318] [<ffffffff811c7471>] jbd2_journal_start+0x13/0x15
[ 274.937320] [<ffffffff811ab92a>] ext4_journal_start_sb+0x108/0x120
[ 274.937323] [<ffffffff8119cc24>] ext4_symlink+0xc0/0x212
[ 274.937325] [<ffffffff81129928>] vfs_symlink+0x54/0x74
[ 274.937326] [<ffffffff8112b40a>] sys_symlinkat+0x96/0xef
[ 274.937329] [<ffffffff81122d89>] ? sys_newstat+0x2a/0x33
[ 274.937330] [<ffffffff8112b479>] sys_symlink+0x16/0x18
[ 274.937332] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
[ 274.937335] Sched Debug Version: v0.10, 2.6.39-rc4+ #1
[ 274.937336] ktime : 275382.978028
[ 274.937337] sched_clk : 274937.334155
[ 274.937338] cpu_clk : 274937.334189
[ 274.937339] jiffies : 4294942678
[ 274.937340] sched_clock_stable : 1
[ 274.937341]
[ 274.937342] sysctl_sched
[ 274.937342] .sysctl_sched_latency : 18.000000
[ 274.937344] .sysctl_sched_min_granularity : 2.250000
[ 274.937345] .sysctl_sched_wakeup_granularity : 3.000000
[ 274.937346] .sysctl_sched_child_runs_first : 0
[ 274.937347] .sysctl_sched_features : 7279
[ 274.937348] .sysctl_sched_tunable_scaling : 1 (logaritmic)
[ 274.937349]
[ 274.937350] cpu#0, 2492.220 MHz
[ 274.937350] .nr_running : 0
[ 274.937351] .load : 0
[ 274.937352] .nr_switches : 105039
[ 274.937353] .nr_load_updates : 121182
[ 274.937354] .nr_uninterruptible : 8
[ 274.937355] .next_balance : 4294.942607
[ 274.937356] .curr->pid : 0
[ 274.937357] .clock : 274864.523786
[ 274.937359] .cpu_load[0] : 0
[ 274.937360] .cpu_load[1] : 0
[ 274.937361] .cpu_load[2] : 0
[ 274.937362] .cpu_load[3] : 0
[ 274.937363] .cpu_load[4] : 0
[ 274.937364] .yld_count : 0
[ 274.937364] .sched_switch : 0
[ 274.937365] .sched_count : 107487
[ 274.937366] .sched_goidle : 43368
[ 274.937367] .avg_idle : 1000000
[ 274.937368] .ttwu_count : 56534
[ 274.937369] .ttwu_local : 53925
[ 274.937370] .bkl_count : 0
[ 274.937371]
[ 274.937372] cfs_rq[0]:/system
[ 274.937372] .exec_clock : 5906.036519
[ 274.937374] .MIN_vruntime : 0.000001
[ 274.937375] .min_vruntime : 7138.254657
[ 274.937376] .max_vruntime : 0.000001
[ 274.937377] .spread : 0.000000
[ 274.937378] .spread0 : -14302.149798
[ 274.937379] .nr_spread_over : 0
[ 274.937379] .nr_running : 0
[ 274.937380] .load : 0
[ 274.937381] .load_avg : 159.999997
[ 274.937382] .load_period : 5.496191
[ 274.937383] .load_contrib : 29
[ 274.937384] .load_tg : 29
[ 274.937385] .se->exec_start : 272446.423795
[ 274.937386] .se->vruntime : 21431.398168
[ 274.937387] .se->sum_exec_runtime : 5906.111805
[ 274.937388] .se->statistics.wait_start : 0.000000
[ 274.937389] .se->statistics.sleep_start : 0.000000
[ 274.937390] .se->statistics.block_start : 0.000000
[ 274.937391] .se->statistics.sleep_max : 0.000000
[ 274.937392] .se->statistics.block_max : 0.000000
[ 274.937393] .se->statistics.exec_max : 15.266617
[ 274.937394] .se->statistics.slice_max : 5.258344
[ 274.937395] .se->statistics.wait_max : 7.305679
[ 274.937396] .se->statistics.wait_sum : 190.506686
[ 274.937397] .se->statistics.wait_count : 18946
[ 274.937398] .se->load.weight : 2
[ 274.937399]
[ 274.937399] cfs_rq[0]:/
[ 274.937400] .exec_clock : 17409.691339
[ 274.937401] .MIN_vruntime : 0.000001
[ 274.937402] .min_vruntime : 21440.404455
[ 274.937403] .max_vruntime : 0.000001
[ 274.937403] .spread : 0.000000
[ 274.937404] .spread0 : 0.000000
[ 274.937405] .nr_spread_over : 58
[ 274.937406] .nr_running : 0
[ 274.937407] .load : 0
[ 274.937408] .load_avg : 0.000000
[ 274.937409] .load_period : 0.000000
[ 274.937409] .load_contrib : 0
[ 274.937410] .load_tg : 0
[ 274.937411]
[ 274.937412] cfs_rq[0]:/system/rescue.service
[ 274.937413] .exec_clock : 4319.868509
[ 274.937414] .MIN_vruntime : 0.000001
[ 274.937414] .min_vruntime : 4255.887524
[ 274.937415] .max_vruntime : 0.000001
[ 274.937416] .spread : 0.000000
[ 274.937417] .spread0 : -17184.516931
[ 274.937418] .nr_spread_over : 24
[ 274.937419] .nr_running : 0
[ 274.937420] .load : 0
[ 274.937421] .load_avg : 160.004112
[ 274.937421] .load_period : 5.496149
[ 274.937422] .load_contrib : 29
[ 274.937423] .load_tg : 29
[ 274.937424] .se->exec_start : 272446.423795
[ 274.937425] .se->vruntime : 7138.254657
[ 274.937426] .se->sum_exec_runtime : 4319.796162
[ 274.937427] .se->statistics.wait_start : 0.000000
[ 274.937428] .se->statistics.sleep_start : 0.000000
[ 274.937429] .se->statistics.block_start : 0.000000
[ 274.937430] .se->statistics.sleep_max : 0.000000
[ 274.937431] .se->statistics.block_max : 0.000000
[ 274.937432] .se->statistics.exec_max : 6.213083
[ 274.937432] .se->statistics.slice_max : 7.894226
[ 274.937433] .se->statistics.wait_max : 4.870567
[ 274.937434] .se->statistics.wait_sum : 91.737803
[ 274.937435] .se->statistics.wait_count : 15504
[ 274.937436] .se->load.weight : 2
[ 274.937437]
[ 274.937438] runnable tasks:
[ 274.937438] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 274.937439] ----------------------------------------------------------------------------------------------------------
[ 274.937444]
[ 274.937445] cpu#1, 2492.220 MHz
[ 274.937445] .nr_running : 0
[ 274.937446] .load : 0
[ 274.937447] .nr_switches : 35274
[ 274.937448] .nr_load_updates : 105259
[ 274.937449] .nr_uninterruptible : 1
[ 274.937449] .next_balance : 4294.942681
[ 274.937450] .curr->pid : 0
[ 274.937451] .clock : 274937.357922
[ 274.937452] .cpu_load[0] : 0
[ 274.937453] .cpu_load[1] : 0
[ 274.937454] .cpu_load[2] : 0
[ 274.937454] .cpu_load[3] : 0
[ 274.937455] .cpu_load[4] : 0
[ 274.937456] .yld_count : 0
[ 274.937457] .sched_switch : 0
[ 274.937458] .sched_count : 35560
[ 274.937458] .sched_goidle : 14561
[ 274.937459] .avg_idle : 1000000
[ 274.937460] .ttwu_count : 16938
[ 274.937461] .ttwu_local : 14491
[ 274.937462] .bkl_count : 0
[ 274.937463]
[ 274.937463] cfs_rq[1]:/
[ 274.937464] .exec_clock : 5221.725565
[ 274.937465] .MIN_vruntime : 0.000001
[ 274.937466] .min_vruntime : 9077.562937
[ 274.937466] .max_vruntime : 0.000001
[ 274.937467] .spread : 0.000000
[ 274.937468] .spread0 : -12362.841518
[ 274.937469] .nr_spread_over : 69
[ 274.937470] .nr_running : 0
[ 274.937471] .load : 0
[ 274.937472] .load_avg : 0.000000
[ 274.937473] .load_period : 0.000000
[ 274.937473] .load_contrib : 0
[ 274.937474] .load_tg : 0
[ 274.937475]
[ 274.937476] runnable tasks:
[ 274.937476] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 274.937477] ----------------------------------------------------------------------------------------------------------
[ 274.937481]
[ 274.937482] cpu#2, 2492.220 MHz
[ 274.937482] .nr_running : 4
[ 274.937483] .load : 2048
[ 274.937484] .nr_switches : 7092
[ 274.937485] .nr_load_updates : 100502
[ 274.937486] .nr_uninterruptible : 0
[ 274.937487] .next_balance : 4294.942806
[ 274.937487] .curr->pid : 46
[ 274.937488] .clock : 183206.600407
[ 274.937489] .cpu_load[0] : 2048
[ 274.937490] .cpu_load[1] : 2048
[ 274.937491] .cpu_load[2] : 2048
[ 274.937492] .cpu_load[3] : 2048
[ 274.937492] .cpu_load[4] : 2048
[ 274.937493] .yld_count : 0
[ 274.937494] .sched_switch : 0
[ 274.937495] .sched_count : 7323
[ 274.937496] .sched_goidle : 2831
[ 274.937496] .avg_idle : 1000000
[ 274.937497] .ttwu_count : 3275
[ 274.937498] .ttwu_local : 2028
[ 274.937499] .bkl_count : 0
[ 274.937500]
[ 274.937500] cfs_rq[2]:/
[ 274.937501] .exec_clock : 2187.192139
[ 274.937502] .MIN_vruntime : 5109.505143
[ 274.937503] .min_vruntime : 5118.505143
[ 274.937504] .max_vruntime : 5109.505143
[ 274.937505] .spread : 0.000000
[ 274.937506] .spread0 : -16321.899312
[ 274.937506] .nr_spread_over : 31
[ 274.937507] .nr_running : 2
[ 274.937508] .load : 2048
[ 274.937509] .load_avg : 0.000000
[ 274.937510] .load_period : 0.000000
[ 274.937511] .load_contrib : 0
[ 274.937511] .load_tg : 0
[ 274.937512]
[ 274.937513] rt_rq[2]:/
[ 274.937513] .rt_nr_running : 1
[ 274.937514] .rt_throttled : 0
[ 274.937515] .rt_time : 0.000000
[ 274.937516] .rt_runtime : 950.000000
[ 274.937517]
[ 274.937517] runnable tasks:
[ 274.937518] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 274.937519] ----------------------------------------------------------------------------------------------------------
[ 274.937520] migration/2 13 0.000000 113 0 0.000000 0.000815 0.000000 /
[ 274.937524] watchdog/2 16 0.000000 17 0 0.000000 1.785026 0.002124 /
[ 274.937527] R kswapd0 46 5118.505143 370 120 5118.505143 415.766870 181943.841718 /
[ 274.937531] kworker/2:1 76 5109.505143 1083 120 5109.505143 25.615305 181183.059001 /
[ 274.937535]
[ 274.937536] cpu#3, 2492.220 MHz
[ 274.937536] .nr_running : 1
[ 274.937537] .load : 1024
[ 274.937538] .nr_switches : 22324
[ 274.937539] .nr_load_updates : 59855
[ 274.937540] .nr_uninterruptible : 0
[ 274.937541] .next_balance : 4294.942679
[ 274.937542] .curr->pid : 73
[ 274.937543] .clock : 274936.938702
[ 274.937543] .cpu_load[0] : 0
[ 274.937544] .cpu_load[1] : 0
[ 274.937545] .cpu_load[2] : 0
[ 274.937546] .cpu_load[3] : 0
[ 274.937546] .cpu_load[4] : 0
[ 274.937547] .yld_count : 123
[ 274.937548] .sched_switch : 0
[ 274.937549] .sched_count : 22774
[ 274.937550] .sched_goidle : 9952
[ 274.937551] .avg_idle : 946660
[ 274.937551] .ttwu_count : 10075
[ 274.937552] .ttwu_local : 8964
[ 274.937553] .bkl_count : 0
[ 274.937554]
[ 274.937554] cfs_rq[3]:/
[ 274.937555] .exec_clock : 3244.912177
[ 274.937556] .MIN_vruntime : 0.000001
[ 274.937557] .min_vruntime : 7517.482154
[ 274.937558] .max_vruntime : 0.000001
[ 274.937559] .spread : 0.000000
[ 274.937559] .spread0 : -13922.922301
[ 274.937560] .nr_spread_over : 128
[ 274.937561] .nr_running : 1
[ 274.937562] .load : 1024
[ 274.937563] .load_avg : 0.000000
[ 274.937564] .load_period : 0.000000
[ 274.937565] .load_contrib : 0
[ 274.937565] .load_tg : 0
[ 274.937567]
[ 274.937567] runnable tasks:
[ 274.937567] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 274.937568] ----------------------------------------------------------------------------------------------------------
[ 274.937571] R kworker/3:1 73 7517.482154 7294 120 7517.482154 264.876541 272702.297840 /
[ 274.937575]
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-27 17:50 ` James Bottomley
@ 2011-04-27 18:25 ` Colin Ian King
-1 siblings, 0 replies; 138+ messages in thread
From: Colin Ian King @ 2011-04-27 18:25 UTC (permalink / raw)
To: James Bottomley
Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4
Hi,
Just like to add that I've seen almost identical issues with 2.6.38
copying large amounts of data to an ext4 filesystem with systems with
small amounts of memory.
I found that increasing /sys/fs/ext4/sdaX/max_writeback_mb_bump worked
around the issue.
Colin
On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote:
> On Wed, 2011-04-27 at 13:34 -0400, Chris Mason wrote:
> > [ cc'd linux-ext4 ]
> >
> > James is hitting softlockups in kswapd while doing writes to a large
> > ext4 file.
> >
> > Excerpts from James Bottomley's message of 2011-04-27 13:21:42 -0400:
> > > On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
> > > > Ok, I'd try turning it on so we catch the sleeping with a spinlock held
> > > > case better.
> > >
> > > Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
> > >
> > > This is the trace with sysrq-l and sysrq-w
> >
> > >
> > > The repro this time doesn't have a soft lockup, just the tar is hung and
> > > one of my CPUs is in 99% system.
> > > [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000
> > > [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
> > > [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
> > > [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
> > > [ 454.753048] Call Trace:
> > > [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> > > [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> > > [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> > > [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
> > > [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> > > [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
> > > [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
> > > [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
> > > [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
> > > [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
> > > [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
> > > [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
> > > [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
> > > [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
> >
> > So our flusher threads are stuck waiting in
> > jbd2_journal_get_write_access, which means they aren't cleaning dirty
> > pages.
> >
> > In order to get write access, they probably need the transaction to
> > commit:
> >
> > > [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000
> > > [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
> > > [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
> > > [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
> > > [ 454.839453] Call Trace:
> > > [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> > > [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> > > [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> > > [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12
> > > [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
> > > [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
> > > [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> > > [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
> > > [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
> > > [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
> >
> > But that seems to be waiting for a page lock. Probably the same page
> > lock held by the flusher thread? Looks like tar is stuck in the same
> > boat down below.
>
> To test the theory, Chris asked me to try with data=ordered.
> Unfortunately, the deadlock still shows up. This is what I get.
>
> James
>
> ---
>
> [ 263.749738] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46]
> [ 263.751207] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 263.756081] CPU 2
> [ 263.756091] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 263.763033]
> [ 263.764761] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 263.766509] RIP: 0010:[<ffffffff810e1fa2>] [<ffffffff810e1fa2>] shrink_slab+0x86/0x166
> [ 263.768256] RSP: 0018:ffff8800709ebda0 EFLAGS: 00000206
> [ 263.769976] RAX: 0000000000000000 RBX: ffff8800709ebde0 RCX: 0000000000000002
> [ 263.771683] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff81a44e50
> [ 263.773378] RBP: ffff8800709ebde0 R08: 0000000000000000 R09: 000000000000a2b2
> [ 263.775053] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff8148300e
> [ 263.776728] R13: ffff8800709ebe58 R14: 0000000000000004 R15: 00000000000178f9
> [ 263.778408] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
> [ 263.780098] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 263.781777] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ 263.783458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 263.785140] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 263.786811] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000)
> [ 263.788491] Stack:
> [ 263.790153] 000000000000003d 0000000000000080 ffff880000000000 ffff8801005e6700
> [ 263.791810] ffff8801005e6000 0000000000000002 0000000000000000 000000000000000c
> [ 263.793426] ffff8800709ebee0 ffffffff810e4bcc 0000000000000003 ffff88006dfa8000
> [ 263.795038] Call Trace:
> [ 263.796614] [<ffffffff810e4bcc>] kswapd+0x533/0x798
> [ 263.798183] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
> [ 263.799755] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 263.801328] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 263.802904] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 263.804434] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 263.805914] Code: 83 eb 10 e9 ce 00 00 00 44 89 f2 31 f6 48 89 df ff 13 48 63 4b 08 4c 63 e8 48 8b 45 c8 31 d2 48 f7 f1 31 d2 49 0f af c5 49 f7 f7
> [ 263.806080] 03 43 20 48 85 c0 48 89 43 20 79 18 48 8b 33 48 89 c2 48 c7
> [ 263.809185] Call Trace:
> [ 263.810748] [<ffffffff810e4bcc>] kswapd+0x533/0x798
> [ 263.812327] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
> [ 263.813913] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 263.815486] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 263.817053] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 263.818621] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 272.421176] SysRq : Show backtrace of all active CPUs
> [ 272.422748] sending NMI to all CPUs:
> [ 272.424304] NMI backtrace for cpu 2
> [ 272.426364] CPU 2
> [ 272.426380] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 272.435128]
> [ 272.437299] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 272.439506] RIP: 0010:[<ffffffff810d9b6e>] [<ffffffff810d9b6e>] zone_watermark_ok_safe+0xa8/0xae
> [ 272.441728] RSP: 0018:ffff8800709ebd80 EFLAGS: 00000282
> [ 272.443916] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> [ 272.446101] RDX: 0000000000000813 RSI: 0000000000000000 RDI: ffff8801005e6000
> [ 272.448258] RBP: ffff8800709ebd90 R08: 0000000000000000 R09: 00000000000007f0
> [ 272.450362] R10: 0000000000000002 R11: 00000000000000cd R12: ffff8801005e6000
> [ 272.452399] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8801005e6000
> [ 272.454367] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
> [ 272.456306] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 272.458188] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ 272.460021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 272.461789] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 272.463503] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000)
> [ 272.465234] Stack:
> [ 272.466929] 0000000000000000 0000000000000000 ffff8800709ebde0 ffffffff810e1e8a
> [ 272.468675] ffff880000000002 ffffffff8106e701 ffff880000000002 ffff8801005e6000
> [ 272.470406] ffff8801005f9e68 0000000000000002 0000000000000000 0000000000000000
> [ 272.472144] Call Trace:
> [ 272.473850] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2
> [ 272.475582] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77
> [ 272.477304] [<ffffffff810e47fe>] kswapd+0x165/0x798
> [ 272.479015] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 272.480712] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
> [ 272.482414] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 272.484106] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 272.485809] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 272.487507] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 272.489175] Code: a6 00 48 8b 55 d8 8b 4d d0 44 8b 45 c8 7c c0 45 31 c9 4d 85 ed 4d 0f 49 cd 44 89 e6 48 89 df e8 5e f4 ff ff 48 83 c4 20 5b 41 5c
> [ 272.489409] 5d 41 5e 5d c3 55 48 89 e5 0f 1f 44 00 00 bf da 00 02 00 e8
> [ 272.492911] Call Trace:
> [ 272.494651] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2
> [ 272.496408] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77
> [ 272.498173] [<ffffffff810e47fe>] kswapd+0x165/0x798
> [ 272.499926] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 272.501678] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
> [ 272.503444] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 272.505198] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 272.506968] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 272.508691] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 272.510355] NMI backtrace for cpu 1
> [ 272.511996] CPU 1
> [ 272.512011] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 272.519303]
> [ 272.521156] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 272.523190] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
> [ 272.525110] RSP: 0018:ffff880071587e68 EFLAGS: 00000046
> [ 272.527018] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
> [ 272.528960] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640
> [ 272.530906] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5
> [ 272.532800] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010
> [ 272.534634] R13: 12187a34a107f726 R14: 0000000000000002 R15: 0000000000000001
> [ 272.536399] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000
> [ 272.538145] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 272.539879] CR2: 00000037da2830e4 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ 272.541614] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 272.543329] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 272.545026] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700)
> [ 272.546756] Stack:
> [ 272.548457] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290
> [ 272.550225] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170
> [ 272.552009] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868
> [ 272.553780] Call Trace:
> [ 272.555514] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 272.557259] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 272.559005] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 272.560755] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 272.562485] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
> [ 272.566380] Call Trace:
> [ 272.568193] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 272.570031] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 272.571855] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 272.573658] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 272.575463] NMI backtrace for cpu 0
> [ 272.576865] CPU 0
> [ 272.576875] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 272.582932]
> [ 272.584474] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 272.586057] RIP: 0010:[<ffffffff81232e6e>] [<ffffffff81232e6e>] __const_udelay+0x23/0x2e
> [ 272.587604] RSP: 0018:ffff880100203bf8 EFLAGS: 00000887
> [ 272.589120] RAX: 00000000fa1c3100 RBX: 0000000000002710 RCX: 0000000000000040
> [ 272.590635] RDX: 000000000026074e RSI: 0000000000000100 RDI: 0000000000418958
> [ 272.592139] RBP: ffff880100203bf8 R08: 000000008b000052 R09: 0000000000000000
> [ 272.593644] R10: 0000000000000000 R11: 0000000000000003 R12: 0000000000000008
> [ 272.595143] R13: 000000000000006c R14: 0000000000000002 R15: 0000000000000001
> [ 272.596635] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000
> [ 272.598139] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 272.599637] CR2: 00000037d909e270 CR3: 000000006f2d7000 CR4: 00000000000406f0
> [ 272.601151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 272.602669] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 272.604187] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020)
> [ 272.605715] Stack:
> [ 272.607238] ffff880100203c18 ffffffff8102166e 0000000000000000 ffffffff81a5e690
> [ 272.608756] ffff880100203c28 ffffffff812be0ad ffff880100203c68 ffffffff812be310
> [ 272.610226] ffff880037d01900 0000000000000026 ffff880037d01900 0000000000000001
> [ 272.611653] Call Trace:
> [ 272.613026] <IRQ>
> [ 272.614388] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
> [ 272.615761] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
> [ 272.617118] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
> [ 272.618465] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
> [ 272.619814] [<ffffffff81365764>] input_pass_event+0x94/0xcc
> [ 272.621158] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
> [ 272.622497] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
> [ 272.623844] [<ffffffff81366cf2>] input_event+0x69/0x87
> [ 272.625172] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
> [ 272.626489] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
> [ 272.627791] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
> [ 272.629085] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
> [ 272.630383] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
> [ 272.631675] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
> [ 272.632946] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
> [ 272.634205] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
> [ 272.635455] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
> [ 272.636699] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
> [ 272.637927] [<ffffffff8147c253>] common_interrupt+0x13/0x13
> [ 272.639155] <EOI>
> [ 272.640374] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
> [ 272.641612] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
> [ 272.642840] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
> [ 272.644012] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 272.645115] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 272.646215] [<ffffffff8145a91e>] rest_init+0x72/0x74
> [ 272.647313] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
> [ 272.648411] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
> [ 272.649514] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
> [ 272.650624] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
> [ 272.651725] Code: ff 15 37 2b 82 00 5d c3 55 48 89 e5 0f 1f 44 00 00 65 48 8b 14 25 98 3a 01 00 48 8d 04 bd 00 00 00 00 48 69 d2 fa 00 00 00 f7 e2
> [ 272.651891] 8d 7a 01 e8 c3 ff ff ff 5d c3 55 48 89 e5 0f 1f 44 00 00 48
> [ 272.654210] Call Trace:
> [ 272.655298] <IRQ> [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
> [ 272.656372] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
> [ 272.657421] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
> [ 272.658456] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
> [ 272.659480] [<ffffffff81365764>] input_pass_event+0x94/0xcc
> [ 272.660486] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
> [ 272.661487] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
> [ 272.662497] [<ffffffff81366cf2>] input_event+0x69/0x87
> [ 272.663502] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
> [ 272.664519] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
> [ 272.665525] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
> [ 272.666522] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
> [ 272.667509] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
> [ 272.668492] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
> [ 272.669467] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
> [ 272.670445] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
> [ 272.671408] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
> [ 272.672357] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
> [ 272.673284] [<ffffffff8147c253>] common_interrupt+0x13/0x13
> [ 272.674225] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
> [ 272.675166] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
> [ 272.676098] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
> [ 272.677032] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 272.677951] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 272.678865] [<ffffffff8145a91e>] rest_init+0x72/0x74
> [ 272.679770] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
> [ 272.680685] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
> [ 272.681603] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
> [ 272.682526] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
> [ 272.683442] NMI backtrace for cpu 3
> [ 272.684688] CPU 3
> [ 272.684705] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 272.690650]
> [ 272.692143] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 272.693726] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
> [ 272.695312] RSP: 0018:ffff8800715dfe68 EFLAGS: 00000046
> [ 272.696892] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001
> [ 272.698508] RDX: 0000000000000000 RSI: ffff8800715dffd8 RDI: ffffffff81a0e640
> [ 272.700117] RBP: ffff8800715dfeb8 R08: 000000000000006d R09: 00000000000003e4
> [ 272.701740] R10: ffffffff00000003 R11: ffff8801002d3b40 R12: 0000000000000030
> [ 272.703369] R13: 12187a34a107edc5 R14: 0000000000000004 R15: 0000000000000003
> [ 272.704985] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000
> [ 272.706626] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 272.708277] CR2: 00000037d904c480 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ 272.709952] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 272.711633] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 272.713299] Process kworker/0:1 (pid: 0, threadinfo ffff8800715de000, task ffff8800715e1700)
> [ 272.715001] Stack:
> [ 272.716681] ffff8800715dfe88 ffffffff810731c0 ffff8801002d1290 0000000000011290
> [ 272.718411] ffff8800715dfeb8 000000038139c97a ffffe8ffffcc0170 ffffe8ffffcc0170
> [ 272.720153] ffffe8ffffcc0300 0000000000000000 ffff8800715dfef8 ffffffff8139b868
> [ 272.721848] Call Trace:
> [ 272.723465] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 272.725095] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 272.726712] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 272.728304] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 272.729891] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
> [ 272.733594] Call Trace:
> [ 272.735317] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 272.737068] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 272.738815] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 272.740554] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 274.865072] SysRq : Show Blocked State
> [ 274.866350] task PC stack pid father
> [ 274.867634] jbd2/dm-1-8 D 0000000000000000 0 363 2 0x00000000
> [ 274.868930] ffff880037d05ba0 0000000000000046 ffff880037d05b30 ffffffff00000000
> [ 274.870242] ffff880037acc500 ffff880037d05fd8 ffff880037d05fd8 0000000000013b40
> [ 274.871555] ffffffff81a0b020 ffff880037acc500 ffff8801002143c0 00000001005bd1d0
> [ 274.872869] Call Trace:
> [ 274.874168] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.875485] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.876791] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 274.878089] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
> [ 274.879382] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
> [ 274.880684] [<ffffffff8147af4e>] out_of_line_wait_on_bit+0x72/0x7d
> [ 274.881986] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.883290] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 274.884601] [<ffffffff811421c6>] __wait_on_buffer+0x26/0x28
> [ 274.885912] [<ffffffff811c8a3c>] wait_on_buffer+0x35/0x39
> [ 274.887221] [<ffffffff811c9633>] jbd2_journal_commit_transaction+0xab3/0x1194
> [ 274.888547] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
> [ 274.889864] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
> [ 274.891178] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
> [ 274.892500] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
> [ 274.893776] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
> [ 274.895001] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 274.896179] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
> [ 274.897328] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 274.898465] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 274.899591] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 274.900708] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 274.901813] flush-253:1 D 0000000000000000 0 419 2 0x00000000
> [ 274.902938] ffff8800706bf760 0000000000000046 ffff8800706bf6f0 ffffffff00000000
> [ 274.904067] ffff880037afc500 ffff8800706bffd8 ffff8800706bffd8 0000000000013b40
> [ 274.905198] ffffffff81a0b020 ffff880037afc500 ffff8800706bf720 00000001706bf720
> [ 274.906327] Call Trace:
> [ 274.907430] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 274.908536] [<ffffffff81217b00>] get_request_wait+0x102/0x18b
> [ 274.909633] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 274.910729] [<ffffffff81218841>] __make_request+0x18a/0x2b8
> [ 274.911826] [<ffffffff81217538>] generic_make_request+0x2a9/0x323
> [ 274.912908] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17
> [ 274.913979] [<ffffffff81217690>] submit_bio+0xde/0xfd
> [ 274.915040] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3
> [ 274.916104] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29
> [ 274.917161] [<ffffffff81141c42>] submit_bh+0xe6/0x105
> [ 274.918208] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7
> [ 274.919261] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
> [ 274.920306] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
> [ 274.921361] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
> [ 274.922411] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
> [ 274.923396] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97
> [ 274.924324] [<ffffffff81144873>] block_write_full_page+0x15/0x17
> [ 274.925248] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a
> [ 274.926168] [<ffffffff810dd3c1>] __writepage+0x15/0x2e
> [ 274.927083] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330
> [ 274.928001] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33
> [ 274.928923] [<ffffffff8104e4c1>] ? find_busiest_group+0x253/0x8b9
> [ 274.929834] [<ffffffff81198ba9>] ? ext4_da_writepages+0x85/0x44d
> [ 274.930744] [<ffffffff810dd363>] generic_writepages+0x40/0x56
> [ 274.931646] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
> [ 274.932536] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
> [ 274.933430] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
> [ 274.934317] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
> [ 274.935194] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
> [ 274.936061] [<ffffffff81045c65>] ? hrtick_update+0x32/0x34
> [ 274.936941] [<ffffffff810dd79a>] ? global_dirty_limits+0x2b/0xd1
> [ 274.936943] [<ffffffff8113d8cd>] wb_do_writeback+0x17f/0x19d
> [ 274.936945] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
> [ 274.936947] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
> [ 274.936949] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 274.936951] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 274.936953] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 274.936956] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 274.936958] jbd2/dm-2-8 D ffff88006e17c700 0 787 2 0x00000000
> [ 274.936960] ffff88006ffe5cd0 0000000000000046 0000000000000000 0000000000000000
> [ 274.936962] ffff88006f0e0000 ffff88006ffe5fd8 ffff88006ffe5fd8 0000000000013b40
> [ 274.936964] ffff8800715fc500 ffff88006f0e0000 ffff88006ffe5cd0 ffffffff8106e7c3
> [ 274.936966] Call Trace:
> [ 274.936968] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
> [ 274.936970] [<ffffffff811c8d44>] jbd2_journal_commit_transaction+0x1c4/0x1194
> [ 274.936972] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61
> [ 274.936974] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd
> [ 274.936977] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
> [ 274.936979] [<ffffffff8100804e>] ? load_TLS+0x10/0x14
> [ 274.936981] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
> [ 274.936983] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 274.936985] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
> [ 274.936987] [<ffffffff8147be8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
> [ 274.936989] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81
> [ 274.936991] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
> [ 274.936993] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 274.936995] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
> [ 274.936997] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 274.936998] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 274.937001] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 274.937003] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 274.937004] flush-253:2 D ffff88006d5a0000 0 851 2 0x00000000
> [ 274.937006] ffff88006e45d760 0000000000000046 ffff880037a30000 0000000000000246
> [ 274.937008] ffff88006d5a0000 ffff88006e45dfd8 ffff88006e45dfd8 0000000000013b40
> [ 274.937010] ffff880071588000 ffff88006d5a0000 ffff88006e45d720 ffff88006e45d720
> [ 274.937011] Call Trace:
> [ 274.937013] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 274.937015] [<ffffffff81217b00>] get_request_wait+0x102/0x18b
> [ 274.937017] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 274.937019] [<ffffffff81218841>] __make_request+0x18a/0x2b8
> [ 274.937020] [<ffffffff81217538>] generic_make_request+0x2a9/0x323
> [ 274.937022] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17
> [ 274.937024] [<ffffffff81217690>] submit_bio+0xde/0xfd
> [ 274.937026] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3
> [ 274.937028] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29
> [ 274.937030] [<ffffffff81141c42>] submit_bh+0xe6/0x105
> [ 274.937031] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7
> [ 274.937033] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
> [ 274.937035] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
> [ 274.937037] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
> [ 274.937039] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
> [ 274.937040] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97
> [ 274.937042] [<ffffffff81144873>] block_write_full_page+0x15/0x17
> [ 274.937044] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a
> [ 274.937046] [<ffffffff810dd3c1>] __writepage+0x15/0x2e
> [ 274.937048] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330
> [ 274.937049] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33
> [ 274.937052] [<ffffffff810dd363>] generic_writepages+0x40/0x56
> [ 274.937054] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
> [ 274.937055] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
> [ 274.937057] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
> [ 274.937059] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
> [ 274.937061] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
> [ 274.937063] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
> [ 274.937065] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
> [ 274.937067] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d
> [ 274.937069] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde
> [ 274.937071] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
> [ 274.937073] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
> [ 274.937075] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 274.937077] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 274.937079] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 274.937081] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 274.937082] tar D 0000000000000000 0 980 826 0x00000000
> [ 274.937084] ffff88006f19f9e8 0000000000000086 00000000ffff9fd7 0000000000000000
> [ 274.937086] ffff88006d670000 ffff88006f19ffd8 ffff88006f19ffd8 0000000000013b40
> [ 274.937088] ffffffff81a0b020 ffff88006d670000 ffff88006f19f9b8 0000000100000282
> [ 274.937090] Call Trace:
> [ 274.937091] [<ffffffff8147acbe>] schedule_timeout+0xa7/0xde
> [ 274.937093] [<ffffffff8106015c>] ? del_timer+0x7a/0x7a
> [ 274.937095] [<ffffffff8147abc2>] io_schedule_timeout+0x6f/0x98
> [ 274.937097] [<ffffffff810ddc26>] balance_dirty_pages_ratelimited_nr+0x341/0x3b6
> [ 274.937100] [<ffffffff810d5efb>] generic_file_buffered_write+0x1dc/0x23a
> [ 274.937102] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
> [ 274.937105] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
> [ 274.937108] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
> [ 274.937111] [<ffffffff8111edab>] do_sync_write+0xbf/0xff
> [ 274.937112] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
> [ 274.937115] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
> [ 274.937118] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 274.937120] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 274.937122] [<ffffffff8111f421>] vfs_write+0xac/0xf3
> [ 274.937123] [<ffffffff8111f610>] sys_write+0x4a/0x6e
> [ 274.937126] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 274.937128] make D 0000000000000000 0 2252 2143 0x00000004
> [ 274.937130] ffff88006d631948 0000000000000082 ffff88006d6318d8 ffffffff00000000
> [ 274.937132] ffff880037d9dc00 ffff88006d631fd8 ffff88006d631fd8 0000000000013b40
> [ 274.937133] ffffffff81a0b020 ffff880037d9dc00 ffff88006d631908 000000016d631908
> [ 274.937135] Call Trace:
> [ 274.937137] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.937139] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 274.937140] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
> [ 274.937142] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
> [ 274.937144] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d
> [ 274.937146] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.937148] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 274.937149] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c
> [ 274.937151] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d
> [ 274.937153] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d
> [ 274.937155] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d
> [ 274.937156] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
> [ 274.937158] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 274.937161] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 274.937163] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
> [ 274.937165] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
> [ 274.937167] [<ffffffff8147c515>] ? page_fault+0x25/0x30
> [ 274.937169] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
> [ 274.937171] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
> [ 274.937173] [<ffffffff811322c7>] touch_atime+0x10e/0x131
> [ 274.937176] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640
> [ 274.937178] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
> [ 274.937180] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 274.937181] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 274.937183] [<ffffffff8111f511>] vfs_read+0xa9/0xf0
> [ 274.937185] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e
> [ 274.937187] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 274.937188] make D 0000000000000000 0 2262 2254 0x00000004
> [ 274.937190] ffff880070b2b948 0000000000000082 ffff880070b2b8d8 ffffffff00000000
> [ 274.937192] ffff88006d428000 ffff880070b2bfd8 ffff880070b2bfd8 0000000000013b40
> [ 274.937194] ffffffff81a0b020 ffff88006d428000 ffff880070b2b908 0000000170b2b908
> [ 274.937195] Call Trace:
> [ 274.937197] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.937199] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 274.937200] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
> [ 274.937202] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
> [ 274.937204] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d
> [ 274.937205] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.937208] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 274.937209] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c
> [ 274.937211] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d
> [ 274.937212] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d
> [ 274.937214] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d
> [ 274.937216] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
> [ 274.937217] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 274.937219] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 274.937221] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
> [ 274.937223] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
> [ 274.937225] [<ffffffff8147c515>] ? page_fault+0x25/0x30
> [ 274.937227] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
> [ 274.937228] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
> [ 274.937230] [<ffffffff811322c7>] touch_atime+0x10e/0x131
> [ 274.937232] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640
> [ 274.937234] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
> [ 274.937236] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 274.937238] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 274.937240] [<ffffffff8111f511>] vfs_read+0xa9/0xf0
> [ 274.937241] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e
> [ 274.937243] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 274.937244] sh D 0000000000000000 0 2270 2253 0x00000004
> [ 274.937246] ffff8800709fd918 0000000000000086 ffff8800709fd8a8 ffffffff00000000
> [ 274.937248] ffff880037f60000 ffff8800709fdfd8 ffff8800709fdfd8 0000000000013b40
> [ 274.937250] ffff880071589700 ffff880037f60000 ffff8801002543c0 00000001005b89d0
> [ 274.937252] Call Trace:
> [ 274.937253] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.937255] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 274.937256] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
> [ 274.937258] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
> [ 274.937260] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d
> [ 274.937261] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.937264] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 274.937265] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c
> [ 274.937267] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d
> [ 274.937268] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d
> [ 274.937270] [<ffffffff81196fe4>] ? ext4_getblk+0x8e/0x153
> [ 274.937272] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 274.937274] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 274.937276] [<ffffffff8119c0ce>] ext4_add_entry+0x11f/0x8cf
> [ 274.937277] [<ffffffff811429f3>] ? __brelse+0x15/0x33
> [ 274.937280] [<ffffffff81191bac>] ? ext4_new_inode+0xc63/0xd0a
> [ 274.937282] [<ffffffff8119c89c>] ext4_add_nondir+0x1e/0x67
> [ 274.937283] [<ffffffff8119cb1b>] ext4_create+0xf5/0x13e
> [ 274.937286] [<ffffffff81129b12>] vfs_create+0x6c/0x8e
> [ 274.937288] [<ffffffff81129dad>] do_last+0x279/0x5ab
> [ 274.937289] [<ffffffff8112ac73>] path_openat+0xc8/0x31c
> [ 274.937291] [<ffffffff810f2332>] ? handle_mm_fault+0x1ac/0x1bf
> [ 274.937293] [<ffffffff8112aeff>] do_filp_open+0x38/0x86
> [ 274.937295] [<ffffffff81233de1>] ? might_fault+0x21/0x23
> [ 274.937298] [<ffffffff8113477b>] ? alloc_fd+0x72/0x11d
> [ 274.937299] [<ffffffff8111e995>] do_sys_open+0x6e/0x100
> [ 274.937301] [<ffffffff8111ea47>] sys_open+0x20/0x22
> [ 274.937303] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 274.937304] gcc D 0000000000000000 0 2272 2271 0x00000004
> [ 274.937306] ffff88006f197c58 0000000000000086 ffff88006f197d14 0000000000000000
> [ 274.937307] ffff880037f64500 ffff88006f197fd8 ffff88006f197fd8 0000000000013b40
> [ 274.937309] ffffffff81a0b020 ffff880037f64500 ffff88006f197c58 000000018106e7c3
> [ 274.937311] Call Trace:
> [ 274.937313] [<ffffffff811c6fd5>] start_this_handle+0x2e8/0x465
> [ 274.937315] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 274.937317] [<ffffffff811c7418>] jbd2__journal_start+0x94/0xda
> [ 274.937318] [<ffffffff811c7471>] jbd2_journal_start+0x13/0x15
> [ 274.937320] [<ffffffff811ab92a>] ext4_journal_start_sb+0x108/0x120
> [ 274.937323] [<ffffffff8119cc24>] ext4_symlink+0xc0/0x212
> [ 274.937325] [<ffffffff81129928>] vfs_symlink+0x54/0x74
> [ 274.937326] [<ffffffff8112b40a>] sys_symlinkat+0x96/0xef
> [ 274.937329] [<ffffffff81122d89>] ? sys_newstat+0x2a/0x33
> [ 274.937330] [<ffffffff8112b479>] sys_symlink+0x16/0x18
> [ 274.937332] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 274.937335] Sched Debug Version: v0.10, 2.6.39-rc4+ #1
> [ 274.937336] ktime : 275382.978028
> [ 274.937337] sched_clk : 274937.334155
> [ 274.937338] cpu_clk : 274937.334189
> [ 274.937339] jiffies : 4294942678
> [ 274.937340] sched_clock_stable : 1
> [ 274.937341]
> [ 274.937342] sysctl_sched
> [ 274.937342] .sysctl_sched_latency : 18.000000
> [ 274.937344] .sysctl_sched_min_granularity : 2.250000
> [ 274.937345] .sysctl_sched_wakeup_granularity : 3.000000
> [ 274.937346] .sysctl_sched_child_runs_first : 0
> [ 274.937347] .sysctl_sched_features : 7279
> [ 274.937348] .sysctl_sched_tunable_scaling : 1 (logaritmic)
> [ 274.937349]
> [ 274.937350] cpu#0, 2492.220 MHz
> [ 274.937350] .nr_running : 0
> [ 274.937351] .load : 0
> [ 274.937352] .nr_switches : 105039
> [ 274.937353] .nr_load_updates : 121182
> [ 274.937354] .nr_uninterruptible : 8
> [ 274.937355] .next_balance : 4294.942607
> [ 274.937356] .curr->pid : 0
> [ 274.937357] .clock : 274864.523786
> [ 274.937359] .cpu_load[0] : 0
> [ 274.937360] .cpu_load[1] : 0
> [ 274.937361] .cpu_load[2] : 0
> [ 274.937362] .cpu_load[3] : 0
> [ 274.937363] .cpu_load[4] : 0
> [ 274.937364] .yld_count : 0
> [ 274.937364] .sched_switch : 0
> [ 274.937365] .sched_count : 107487
> [ 274.937366] .sched_goidle : 43368
> [ 274.937367] .avg_idle : 1000000
> [ 274.937368] .ttwu_count : 56534
> [ 274.937369] .ttwu_local : 53925
> [ 274.937370] .bkl_count : 0
> [ 274.937371]
> [ 274.937372] cfs_rq[0]:/system
> [ 274.937372] .exec_clock : 5906.036519
> [ 274.937374] .MIN_vruntime : 0.000001
> [ 274.937375] .min_vruntime : 7138.254657
> [ 274.937376] .max_vruntime : 0.000001
> [ 274.937377] .spread : 0.000000
> [ 274.937378] .spread0 : -14302.149798
> [ 274.937379] .nr_spread_over : 0
> [ 274.937379] .nr_running : 0
> [ 274.937380] .load : 0
> [ 274.937381] .load_avg : 159.999997
> [ 274.937382] .load_period : 5.496191
> [ 274.937383] .load_contrib : 29
> [ 274.937384] .load_tg : 29
> [ 274.937385] .se->exec_start : 272446.423795
> [ 274.937386] .se->vruntime : 21431.398168
> [ 274.937387] .se->sum_exec_runtime : 5906.111805
> [ 274.937388] .se->statistics.wait_start : 0.000000
> [ 274.937389] .se->statistics.sleep_start : 0.000000
> [ 274.937390] .se->statistics.block_start : 0.000000
> [ 274.937391] .se->statistics.sleep_max : 0.000000
> [ 274.937392] .se->statistics.block_max : 0.000000
> [ 274.937393] .se->statistics.exec_max : 15.266617
> [ 274.937394] .se->statistics.slice_max : 5.258344
> [ 274.937395] .se->statistics.wait_max : 7.305679
> [ 274.937396] .se->statistics.wait_sum : 190.506686
> [ 274.937397] .se->statistics.wait_count : 18946
> [ 274.937398] .se->load.weight : 2
> [ 274.937399]
> [ 274.937399] cfs_rq[0]:/
> [ 274.937400] .exec_clock : 17409.691339
> [ 274.937401] .MIN_vruntime : 0.000001
> [ 274.937402] .min_vruntime : 21440.404455
> [ 274.937403] .max_vruntime : 0.000001
> [ 274.937403] .spread : 0.000000
> [ 274.937404] .spread0 : 0.000000
> [ 274.937405] .nr_spread_over : 58
> [ 274.937406] .nr_running : 0
> [ 274.937407] .load : 0
> [ 274.937408] .load_avg : 0.000000
> [ 274.937409] .load_period : 0.000000
> [ 274.937409] .load_contrib : 0
> [ 274.937410] .load_tg : 0
> [ 274.937411]
> [ 274.937412] cfs_rq[0]:/system/rescue.service
> [ 274.937413] .exec_clock : 4319.868509
> [ 274.937414] .MIN_vruntime : 0.000001
> [ 274.937414] .min_vruntime : 4255.887524
> [ 274.937415] .max_vruntime : 0.000001
> [ 274.937416] .spread : 0.000000
> [ 274.937417] .spread0 : -17184.516931
> [ 274.937418] .nr_spread_over : 24
> [ 274.937419] .nr_running : 0
> [ 274.937420] .load : 0
> [ 274.937421] .load_avg : 160.004112
> [ 274.937421] .load_period : 5.496149
> [ 274.937422] .load_contrib : 29
> [ 274.937423] .load_tg : 29
> [ 274.937424] .se->exec_start : 272446.423795
> [ 274.937425] .se->vruntime : 7138.254657
> [ 274.937426] .se->sum_exec_runtime : 4319.796162
> [ 274.937427] .se->statistics.wait_start : 0.000000
> [ 274.937428] .se->statistics.sleep_start : 0.000000
> [ 274.937429] .se->statistics.block_start : 0.000000
> [ 274.937430] .se->statistics.sleep_max : 0.000000
> [ 274.937431] .se->statistics.block_max : 0.000000
> [ 274.937432] .se->statistics.exec_max : 6.213083
> [ 274.937432] .se->statistics.slice_max : 7.894226
> [ 274.937433] .se->statistics.wait_max : 4.870567
> [ 274.937434] .se->statistics.wait_sum : 91.737803
> [ 274.937435] .se->statistics.wait_count : 15504
> [ 274.937436] .se->load.weight : 2
> [ 274.937437]
> [ 274.937438] runnable tasks:
> [ 274.937438] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 274.937439] ----------------------------------------------------------------------------------------------------------
> [ 274.937444]
> [ 274.937445] cpu#1, 2492.220 MHz
> [ 274.937445] .nr_running : 0
> [ 274.937446] .load : 0
> [ 274.937447] .nr_switches : 35274
> [ 274.937448] .nr_load_updates : 105259
> [ 274.937449] .nr_uninterruptible : 1
> [ 274.937449] .next_balance : 4294.942681
> [ 274.937450] .curr->pid : 0
> [ 274.937451] .clock : 274937.357922
> [ 274.937452] .cpu_load[0] : 0
> [ 274.937453] .cpu_load[1] : 0
> [ 274.937454] .cpu_load[2] : 0
> [ 274.937454] .cpu_load[3] : 0
> [ 274.937455] .cpu_load[4] : 0
> [ 274.937456] .yld_count : 0
> [ 274.937457] .sched_switch : 0
> [ 274.937458] .sched_count : 35560
> [ 274.937458] .sched_goidle : 14561
> [ 274.937459] .avg_idle : 1000000
> [ 274.937460] .ttwu_count : 16938
> [ 274.937461] .ttwu_local : 14491
> [ 274.937462] .bkl_count : 0
> [ 274.937463]
> [ 274.937463] cfs_rq[1]:/
> [ 274.937464] .exec_clock : 5221.725565
> [ 274.937465] .MIN_vruntime : 0.000001
> [ 274.937466] .min_vruntime : 9077.562937
> [ 274.937466] .max_vruntime : 0.000001
> [ 274.937467] .spread : 0.000000
> [ 274.937468] .spread0 : -12362.841518
> [ 274.937469] .nr_spread_over : 69
> [ 274.937470] .nr_running : 0
> [ 274.937471] .load : 0
> [ 274.937472] .load_avg : 0.000000
> [ 274.937473] .load_period : 0.000000
> [ 274.937473] .load_contrib : 0
> [ 274.937474] .load_tg : 0
> [ 274.937475]
> [ 274.937476] runnable tasks:
> [ 274.937476] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 274.937477] ----------------------------------------------------------------------------------------------------------
> [ 274.937481]
> [ 274.937482] cpu#2, 2492.220 MHz
> [ 274.937482] .nr_running : 4
> [ 274.937483] .load : 2048
> [ 274.937484] .nr_switches : 7092
> [ 274.937485] .nr_load_updates : 100502
> [ 274.937486] .nr_uninterruptible : 0
> [ 274.937487] .next_balance : 4294.942806
> [ 274.937487] .curr->pid : 46
> [ 274.937488] .clock : 183206.600407
> [ 274.937489] .cpu_load[0] : 2048
> [ 274.937490] .cpu_load[1] : 2048
> [ 274.937491] .cpu_load[2] : 2048
> [ 274.937492] .cpu_load[3] : 2048
> [ 274.937492] .cpu_load[4] : 2048
> [ 274.937493] .yld_count : 0
> [ 274.937494] .sched_switch : 0
> [ 274.937495] .sched_count : 7323
> [ 274.937496] .sched_goidle : 2831
> [ 274.937496] .avg_idle : 1000000
> [ 274.937497] .ttwu_count : 3275
> [ 274.937498] .ttwu_local : 2028
> [ 274.937499] .bkl_count : 0
> [ 274.937500]
> [ 274.937500] cfs_rq[2]:/
> [ 274.937501] .exec_clock : 2187.192139
> [ 274.937502] .MIN_vruntime : 5109.505143
> [ 274.937503] .min_vruntime : 5118.505143
> [ 274.937504] .max_vruntime : 5109.505143
> [ 274.937505] .spread : 0.000000
> [ 274.937506] .spread0 : -16321.899312
> [ 274.937506] .nr_spread_over : 31
> [ 274.937507] .nr_running : 2
> [ 274.937508] .load : 2048
> [ 274.937509] .load_avg : 0.000000
> [ 274.937510] .load_period : 0.000000
> [ 274.937511] .load_contrib : 0
> [ 274.937511] .load_tg : 0
> [ 274.937512]
> [ 274.937513] rt_rq[2]:/
> [ 274.937513] .rt_nr_running : 1
> [ 274.937514] .rt_throttled : 0
> [ 274.937515] .rt_time : 0.000000
> [ 274.937516] .rt_runtime : 950.000000
> [ 274.937517]
> [ 274.937517] runnable tasks:
> [ 274.937518] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 274.937519] ----------------------------------------------------------------------------------------------------------
> [ 274.937520] migration/2 13 0.000000 113 0 0.000000 0.000815 0.000000 /
> [ 274.937524] watchdog/2 16 0.000000 17 0 0.000000 1.785026 0.002124 /
> [ 274.937527] R kswapd0 46 5118.505143 370 120 5118.505143 415.766870 181943.841718 /
> [ 274.937531] kworker/2:1 76 5109.505143 1083 120 5109.505143 25.615305 181183.059001 /
> [ 274.937535]
> [ 274.937536] cpu#3, 2492.220 MHz
> [ 274.937536] .nr_running : 1
> [ 274.937537] .load : 1024
> [ 274.937538] .nr_switches : 22324
> [ 274.937539] .nr_load_updates : 59855
> [ 274.937540] .nr_uninterruptible : 0
> [ 274.937541] .next_balance : 4294.942679
> [ 274.937542] .curr->pid : 73
> [ 274.937543] .clock : 274936.938702
> [ 274.937543] .cpu_load[0] : 0
> [ 274.937544] .cpu_load[1] : 0
> [ 274.937545] .cpu_load[2] : 0
> [ 274.937546] .cpu_load[3] : 0
> [ 274.937546] .cpu_load[4] : 0
> [ 274.937547] .yld_count : 123
> [ 274.937548] .sched_switch : 0
> [ 274.937549] .sched_count : 22774
> [ 274.937550] .sched_goidle : 9952
> [ 274.937551] .avg_idle : 946660
> [ 274.937551] .ttwu_count : 10075
> [ 274.937552] .ttwu_local : 8964
> [ 274.937553] .bkl_count : 0
> [ 274.937554]
> [ 274.937554] cfs_rq[3]:/
> [ 274.937555] .exec_clock : 3244.912177
> [ 274.937556] .MIN_vruntime : 0.000001
> [ 274.937557] .min_vruntime : 7517.482154
> [ 274.937558] .max_vruntime : 0.000001
> [ 274.937559] .spread : 0.000000
> [ 274.937559] .spread0 : -13922.922301
> [ 274.937560] .nr_spread_over : 128
> [ 274.937561] .nr_running : 1
> [ 274.937562] .load : 1024
> [ 274.937563] .load_avg : 0.000000
> [ 274.937564] .load_period : 0.000000
> [ 274.937565] .load_contrib : 0
> [ 274.937565] .load_tg : 0
> [ 274.937567]
> [ 274.937567] runnable tasks:
> [ 274.937567] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 274.937568] ----------------------------------------------------------------------------------------------------------
> [ 274.937571] R kworker/3:1 73 7517.482154 7294 120 7517.482154 264.876541 272702.297840 /
> [ 274.937575]
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-27 18:25 ` Colin Ian King
0 siblings, 0 replies; 138+ messages in thread
From: Colin Ian King @ 2011-04-27 18:25 UTC (permalink / raw)
To: James Bottomley
Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4
Hi,
Just like to add that I've seen almost identical issues with 2.6.38
copying large amounts of data to an ext4 filesystem with systems with
small amounts of memory.
I found that increasing /sys/fs/ext4/sdaX/max_writeback_mb_bump worked
around the issue.
Colin
On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote:
> On Wed, 2011-04-27 at 13:34 -0400, Chris Mason wrote:
> > [ cc'd linux-ext4 ]
> >
> > James is hitting softlockups in kswapd while doing writes to a large
> > ext4 file.
> >
> > Excerpts from James Bottomley's message of 2011-04-27 13:21:42 -0400:
> > > On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
> > > > Ok, I'd try turning it on so we catch the sleeping with a spinlock held
> > > > case better.
> > >
> > > Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
> > >
> > > This is the trace with sysrq-l and sysrq-w
> >
> > >
> > > The repro this time doesn't have a soft lockup, just the tar is hung and
> > > one of my CPUs is in 99% system.
> > > [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000
> > > [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
> > > [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
> > > [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
> > > [ 454.753048] Call Trace:
> > > [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> > > [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> > > [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> > > [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
> > > [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> > > [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
> > > [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
> > > [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
> > > [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
> > > [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
> > > [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
> > > [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
> > > [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
> > > [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
> >
> > So our flusher threads are stuck waiting in
> > jbd2_journal_get_write_access, which means they aren't cleaning dirty
> > pages.
> >
> > In order to get write access, they probably need the transaction to
> > commit:
> >
> > > [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000
> > > [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
> > > [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
> > > [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
> > > [ 454.839453] Call Trace:
> > > [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> > > [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> > > [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> > > [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12
> > > [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
> > > [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
> > > [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> > > [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
> > > [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
> > > [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
> >
> > But that seems to be waiting for a page lock. Probably the same page
> > lock held by the flusher thread? Looks like tar is stuck in the same
> > boat down below.
>
> To test the theory, Chris asked me to try with data=ordered.
> Unfortunately, the deadlock still shows up. This is what I get.
>
> James
>
> ---
>
> [ 263.749738] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46]
> [ 263.751207] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 263.756081] CPU 2
> [ 263.756091] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 263.763033]
> [ 263.764761] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 263.766509] RIP: 0010:[<ffffffff810e1fa2>] [<ffffffff810e1fa2>] shrink_slab+0x86/0x166
> [ 263.768256] RSP: 0018:ffff8800709ebda0 EFLAGS: 00000206
> [ 263.769976] RAX: 0000000000000000 RBX: ffff8800709ebde0 RCX: 0000000000000002
> [ 263.771683] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff81a44e50
> [ 263.773378] RBP: ffff8800709ebde0 R08: 0000000000000000 R09: 000000000000a2b2
> [ 263.775053] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff8148300e
> [ 263.776728] R13: ffff8800709ebe58 R14: 0000000000000004 R15: 00000000000178f9
> [ 263.778408] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
> [ 263.780098] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 263.781777] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ 263.783458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 263.785140] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 263.786811] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000)
> [ 263.788491] Stack:
> [ 263.790153] 000000000000003d 0000000000000080 ffff880000000000 ffff8801005e6700
> [ 263.791810] ffff8801005e6000 0000000000000002 0000000000000000 000000000000000c
> [ 263.793426] ffff8800709ebee0 ffffffff810e4bcc 0000000000000003 ffff88006dfa8000
> [ 263.795038] Call Trace:
> [ 263.796614] [<ffffffff810e4bcc>] kswapd+0x533/0x798
> [ 263.798183] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
> [ 263.799755] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 263.801328] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 263.802904] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 263.804434] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 263.805914] Code: 83 eb 10 e9 ce 00 00 00 44 89 f2 31 f6 48 89 df ff 13 48 63 4b 08 4c 63 e8 48 8b 45 c8 31 d2 48 f7 f1 31 d2 49 0f af c5 49 f7 f7
> [ 263.806080] 03 43 20 48 85 c0 48 89 43 20 79 18 48 8b 33 48 89 c2 48 c7
> [ 263.809185] Call Trace:
> [ 263.810748] [<ffffffff810e4bcc>] kswapd+0x533/0x798
> [ 263.812327] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
> [ 263.813913] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 263.815486] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 263.817053] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 263.818621] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 272.421176] SysRq : Show backtrace of all active CPUs
> [ 272.422748] sending NMI to all CPUs:
> [ 272.424304] NMI backtrace for cpu 2
> [ 272.426364] CPU 2
> [ 272.426380] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 272.435128]
> [ 272.437299] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 272.439506] RIP: 0010:[<ffffffff810d9b6e>] [<ffffffff810d9b6e>] zone_watermark_ok_safe+0xa8/0xae
> [ 272.441728] RSP: 0018:ffff8800709ebd80 EFLAGS: 00000282
> [ 272.443916] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> [ 272.446101] RDX: 0000000000000813 RSI: 0000000000000000 RDI: ffff8801005e6000
> [ 272.448258] RBP: ffff8800709ebd90 R08: 0000000000000000 R09: 00000000000007f0
> [ 272.450362] R10: 0000000000000002 R11: 00000000000000cd R12: ffff8801005e6000
> [ 272.452399] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8801005e6000
> [ 272.454367] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
> [ 272.456306] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 272.458188] CR2: 0000000001091018 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ 272.460021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 272.461789] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 272.463503] Process kswapd0 (pid: 46, threadinfo ffff8800709ea000, task ffff88006dfa8000)
> [ 272.465234] Stack:
> [ 272.466929] 0000000000000000 0000000000000000 ffff8800709ebde0 ffffffff810e1e8a
> [ 272.468675] ffff880000000002 ffffffff8106e701 ffff880000000002 ffff8801005e6000
> [ 272.470406] ffff8801005f9e68 0000000000000002 0000000000000000 0000000000000000
> [ 272.472144] Call Trace:
> [ 272.473850] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2
> [ 272.475582] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77
> [ 272.477304] [<ffffffff810e47fe>] kswapd+0x165/0x798
> [ 272.479015] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 272.480712] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
> [ 272.482414] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 272.484106] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 272.485809] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 272.487507] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 272.489175] Code: a6 00 48 8b 55 d8 8b 4d d0 44 8b 45 c8 7c c0 45 31 c9 4d 85 ed 4d 0f 49 cd 44 89 e6 48 89 df e8 5e f4 ff ff 48 83 c4 20 5b 41 5c
> [ 272.489409] 5d 41 5e 5d c3 55 48 89 e5 0f 1f 44 00 00 bf da 00 02 00 e8
> [ 272.492911] Call Trace:
> [ 272.494651] [<ffffffff810e1e8a>] sleeping_prematurely.part.11+0x6e/0xd2
> [ 272.496408] [<ffffffff8106e701>] ? prepare_to_wait_exclusive+0x67/0x77
> [ 272.498173] [<ffffffff810e47fe>] kswapd+0x165/0x798
> [ 272.499926] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 272.501678] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
> [ 272.503444] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 272.505198] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 272.506968] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 272.508691] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 272.510355] NMI backtrace for cpu 1
> [ 272.511996] CPU 1
> [ 272.512011] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 272.519303]
> [ 272.521156] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 272.523190] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
> [ 272.525110] RSP: 0018:ffff880071587e68 EFLAGS: 00000046
> [ 272.527018] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
> [ 272.528960] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640
> [ 272.530906] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5
> [ 272.532800] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010
> [ 272.534634] R13: 12187a34a107f726 R14: 0000000000000002 R15: 0000000000000001
> [ 272.536399] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000
> [ 272.538145] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 272.539879] CR2: 00000037da2830e4 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ 272.541614] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 272.543329] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 272.545026] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700)
> [ 272.546756] Stack:
> [ 272.548457] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290
> [ 272.550225] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170
> [ 272.552009] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868
> [ 272.553780] Call Trace:
> [ 272.555514] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 272.557259] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 272.559005] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 272.560755] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 272.562485] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
> [ 272.566380] Call Trace:
> [ 272.568193] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 272.570031] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 272.571855] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 272.573658] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 272.575463] NMI backtrace for cpu 0
> [ 272.576865] CPU 0
> [ 272.576875] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 272.582932]
> [ 272.584474] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 272.586057] RIP: 0010:[<ffffffff81232e6e>] [<ffffffff81232e6e>] __const_udelay+0x23/0x2e
> [ 272.587604] RSP: 0018:ffff880100203bf8 EFLAGS: 00000887
> [ 272.589120] RAX: 00000000fa1c3100 RBX: 0000000000002710 RCX: 0000000000000040
> [ 272.590635] RDX: 000000000026074e RSI: 0000000000000100 RDI: 0000000000418958
> [ 272.592139] RBP: ffff880100203bf8 R08: 000000008b000052 R09: 0000000000000000
> [ 272.593644] R10: 0000000000000000 R11: 0000000000000003 R12: 0000000000000008
> [ 272.595143] R13: 000000000000006c R14: 0000000000000002 R15: 0000000000000001
> [ 272.596635] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000
> [ 272.598139] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 272.599637] CR2: 00000037d909e270 CR3: 000000006f2d7000 CR4: 00000000000406f0
> [ 272.601151] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 272.602669] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 272.604187] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020)
> [ 272.605715] Stack:
> [ 272.607238] ffff880100203c18 ffffffff8102166e 0000000000000000 ffffffff81a5e690
> [ 272.608756] ffff880100203c28 ffffffff812be0ad ffff880100203c68 ffffffff812be310
> [ 272.610226] ffff880037d01900 0000000000000026 ffff880037d01900 0000000000000001
> [ 272.611653] Call Trace:
> [ 272.613026] <IRQ>
> [ 272.614388] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
> [ 272.615761] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
> [ 272.617118] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
> [ 272.618465] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
> [ 272.619814] [<ffffffff81365764>] input_pass_event+0x94/0xcc
> [ 272.621158] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
> [ 272.622497] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
> [ 272.623844] [<ffffffff81366cf2>] input_event+0x69/0x87
> [ 272.625172] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
> [ 272.626489] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
> [ 272.627791] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
> [ 272.629085] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
> [ 272.630383] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
> [ 272.631675] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
> [ 272.632946] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
> [ 272.634205] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
> [ 272.635455] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
> [ 272.636699] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
> [ 272.637927] [<ffffffff8147c253>] common_interrupt+0x13/0x13
> [ 272.639155] <EOI>
> [ 272.640374] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
> [ 272.641612] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
> [ 272.642840] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
> [ 272.644012] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 272.645115] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 272.646215] [<ffffffff8145a91e>] rest_init+0x72/0x74
> [ 272.647313] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
> [ 272.648411] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
> [ 272.649514] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
> [ 272.650624] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
> [ 272.651725] Code: ff 15 37 2b 82 00 5d c3 55 48 89 e5 0f 1f 44 00 00 65 48 8b 14 25 98 3a 01 00 48 8d 04 bd 00 00 00 00 48 69 d2 fa 00 00 00 f7 e2
> [ 272.651891] 8d 7a 01 e8 c3 ff ff ff 5d c3 55 48 89 e5 0f 1f 44 00 00 48
> [ 272.654210] Call Trace:
> [ 272.655298] <IRQ> [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
> [ 272.656372] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
> [ 272.657421] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
> [ 272.658456] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
> [ 272.659480] [<ffffffff81365764>] input_pass_event+0x94/0xcc
> [ 272.660486] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
> [ 272.661487] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
> [ 272.662497] [<ffffffff81366cf2>] input_event+0x69/0x87
> [ 272.663502] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
> [ 272.664519] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
> [ 272.665525] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
> [ 272.666522] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
> [ 272.667509] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
> [ 272.668492] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
> [ 272.669467] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
> [ 272.670445] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
> [ 272.671408] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
> [ 272.672357] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
> [ 272.673284] [<ffffffff8147c253>] common_interrupt+0x13/0x13
> [ 272.674225] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
> [ 272.675166] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
> [ 272.676098] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
> [ 272.677032] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 272.677951] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 272.678865] [<ffffffff8145a91e>] rest_init+0x72/0x74
> [ 272.679770] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
> [ 272.680685] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
> [ 272.681603] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
> [ 272.682526] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
> [ 272.683442] NMI backtrace for cpu 3
> [ 272.684688] CPU 3
> [ 272.684705] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep iwlagn snd_seq snd_seq_device uvcvideo mac80211 e1000e btusb videodev snd_pcm v4l2_compat_ioctl32 snd_timer bluetooth i2c_i801 snd cfg80211 microcode iTCO_wdt xhci_hcd iTCO_vendor_support wmi pcspkr joydev soundcore snd_page_alloc rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 272.690650]
> [ 272.692143] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 272.693726] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
> [ 272.695312] RSP: 0018:ffff8800715dfe68 EFLAGS: 00000046
> [ 272.696892] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001
> [ 272.698508] RDX: 0000000000000000 RSI: ffff8800715dffd8 RDI: ffffffff81a0e640
> [ 272.700117] RBP: ffff8800715dfeb8 R08: 000000000000006d R09: 00000000000003e4
> [ 272.701740] R10: ffffffff00000003 R11: ffff8801002d3b40 R12: 0000000000000030
> [ 272.703369] R13: 12187a34a107edc5 R14: 0000000000000004 R15: 0000000000000003
> [ 272.704985] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000
> [ 272.706626] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 272.708277] CR2: 00000037d904c480 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ 272.709952] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 272.711633] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 272.713299] Process kworker/0:1 (pid: 0, threadinfo ffff8800715de000, task ffff8800715e1700)
> [ 272.715001] Stack:
> [ 272.716681] ffff8800715dfe88 ffffffff810731c0 ffff8801002d1290 0000000000011290
> [ 272.718411] ffff8800715dfeb8 000000038139c97a ffffe8ffffcc0170 ffffe8ffffcc0170
> [ 272.720153] ffffe8ffffcc0300 0000000000000000 ffff8800715dfef8 ffffffff8139b868
> [ 272.721848] Call Trace:
> [ 272.723465] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 272.725095] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 272.726712] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 272.728304] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 272.729891] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
> [ 272.733594] Call Trace:
> [ 272.735317] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 272.737068] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 272.738815] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 272.740554] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 274.865072] SysRq : Show Blocked State
> [ 274.866350] task PC stack pid father
> [ 274.867634] jbd2/dm-1-8 D 0000000000000000 0 363 2 0x00000000
> [ 274.868930] ffff880037d05ba0 0000000000000046 ffff880037d05b30 ffffffff00000000
> [ 274.870242] ffff880037acc500 ffff880037d05fd8 ffff880037d05fd8 0000000000013b40
> [ 274.871555] ffffffff81a0b020 ffff880037acc500 ffff8801002143c0 00000001005bd1d0
> [ 274.872869] Call Trace:
> [ 274.874168] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.875485] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.876791] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 274.878089] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
> [ 274.879382] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
> [ 274.880684] [<ffffffff8147af4e>] out_of_line_wait_on_bit+0x72/0x7d
> [ 274.881986] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.883290] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 274.884601] [<ffffffff811421c6>] __wait_on_buffer+0x26/0x28
> [ 274.885912] [<ffffffff811c8a3c>] wait_on_buffer+0x35/0x39
> [ 274.887221] [<ffffffff811c9633>] jbd2_journal_commit_transaction+0xab3/0x1194
> [ 274.888547] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
> [ 274.889864] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
> [ 274.891178] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
> [ 274.892500] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
> [ 274.893776] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
> [ 274.895001] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 274.896179] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
> [ 274.897328] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 274.898465] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 274.899591] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 274.900708] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 274.901813] flush-253:1 D 0000000000000000 0 419 2 0x00000000
> [ 274.902938] ffff8800706bf760 0000000000000046 ffff8800706bf6f0 ffffffff00000000
> [ 274.904067] ffff880037afc500 ffff8800706bffd8 ffff8800706bffd8 0000000000013b40
> [ 274.905198] ffffffff81a0b020 ffff880037afc500 ffff8800706bf720 00000001706bf720
> [ 274.906327] Call Trace:
> [ 274.907430] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 274.908536] [<ffffffff81217b00>] get_request_wait+0x102/0x18b
> [ 274.909633] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 274.910729] [<ffffffff81218841>] __make_request+0x18a/0x2b8
> [ 274.911826] [<ffffffff81217538>] generic_make_request+0x2a9/0x323
> [ 274.912908] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17
> [ 274.913979] [<ffffffff81217690>] submit_bio+0xde/0xfd
> [ 274.915040] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3
> [ 274.916104] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29
> [ 274.917161] [<ffffffff81141c42>] submit_bh+0xe6/0x105
> [ 274.918208] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7
> [ 274.919261] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
> [ 274.920306] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
> [ 274.921361] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
> [ 274.922411] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
> [ 274.923396] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97
> [ 274.924324] [<ffffffff81144873>] block_write_full_page+0x15/0x17
> [ 274.925248] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a
> [ 274.926168] [<ffffffff810dd3c1>] __writepage+0x15/0x2e
> [ 274.927083] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330
> [ 274.928001] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33
> [ 274.928923] [<ffffffff8104e4c1>] ? find_busiest_group+0x253/0x8b9
> [ 274.929834] [<ffffffff81198ba9>] ? ext4_da_writepages+0x85/0x44d
> [ 274.930744] [<ffffffff810dd363>] generic_writepages+0x40/0x56
> [ 274.931646] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
> [ 274.932536] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
> [ 274.933430] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
> [ 274.934317] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
> [ 274.935194] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
> [ 274.936061] [<ffffffff81045c65>] ? hrtick_update+0x32/0x34
> [ 274.936941] [<ffffffff810dd79a>] ? global_dirty_limits+0x2b/0xd1
> [ 274.936943] [<ffffffff8113d8cd>] wb_do_writeback+0x17f/0x19d
> [ 274.936945] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
> [ 274.936947] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
> [ 274.936949] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 274.936951] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 274.936953] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 274.936956] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 274.936958] jbd2/dm-2-8 D ffff88006e17c700 0 787 2 0x00000000
> [ 274.936960] ffff88006ffe5cd0 0000000000000046 0000000000000000 0000000000000000
> [ 274.936962] ffff88006f0e0000 ffff88006ffe5fd8 ffff88006ffe5fd8 0000000000013b40
> [ 274.936964] ffff8800715fc500 ffff88006f0e0000 ffff88006ffe5cd0 ffffffff8106e7c3
> [ 274.936966] Call Trace:
> [ 274.936968] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
> [ 274.936970] [<ffffffff811c8d44>] jbd2_journal_commit_transaction+0x1c4/0x1194
> [ 274.936972] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61
> [ 274.936974] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd
> [ 274.936977] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
> [ 274.936979] [<ffffffff8100804e>] ? load_TLS+0x10/0x14
> [ 274.936981] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
> [ 274.936983] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 274.936985] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
> [ 274.936987] [<ffffffff8147be8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
> [ 274.936989] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81
> [ 274.936991] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
> [ 274.936993] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 274.936995] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
> [ 274.936997] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 274.936998] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 274.937001] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 274.937003] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 274.937004] flush-253:2 D ffff88006d5a0000 0 851 2 0x00000000
> [ 274.937006] ffff88006e45d760 0000000000000046 ffff880037a30000 0000000000000246
> [ 274.937008] ffff88006d5a0000 ffff88006e45dfd8 ffff88006e45dfd8 0000000000013b40
> [ 274.937010] ffff880071588000 ffff88006d5a0000 ffff88006e45d720 ffff88006e45d720
> [ 274.937011] Call Trace:
> [ 274.937013] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 274.937015] [<ffffffff81217b00>] get_request_wait+0x102/0x18b
> [ 274.937017] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 274.937019] [<ffffffff81218841>] __make_request+0x18a/0x2b8
> [ 274.937020] [<ffffffff81217538>] generic_make_request+0x2a9/0x323
> [ 274.937022] [<ffffffff810d79c9>] ? mempool_alloc_slab+0x15/0x17
> [ 274.937024] [<ffffffff81217690>] submit_bio+0xde/0xfd
> [ 274.937026] [<ffffffff81145fc0>] ? bio_alloc_bioset+0x4c/0xc3
> [ 274.937028] [<ffffffff810ea6e1>] ? inc_zone_page_state+0x27/0x29
> [ 274.937030] [<ffffffff81141c42>] submit_bh+0xe6/0x105
> [ 274.937031] [<ffffffff8114317e>] __block_write_full_page+0x1e7/0x2d7
> [ 274.937033] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
> [ 274.937035] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
> [ 274.937037] [<ffffffff81144967>] ? bit_spin_lock.constprop.20+0x2c/0x2c
> [ 274.937039] [<ffffffff81147540>] ? thaw_bdev+0x79/0x79
> [ 274.937040] [<ffffffff81144851>] block_write_full_page_endio+0x8a/0x97
> [ 274.937042] [<ffffffff81144873>] block_write_full_page+0x15/0x17
> [ 274.937044] [<ffffffff81147297>] blkdev_writepage+0x18/0x1a
> [ 274.937046] [<ffffffff810dd3c1>] __writepage+0x15/0x2e
> [ 274.937048] [<ffffffff810dd1fc>] write_cache_pages+0x209/0x330
> [ 274.937049] [<ffffffff810dd3ac>] ? set_page_dirty_lock+0x33/0x33
> [ 274.937052] [<ffffffff810dd363>] generic_writepages+0x40/0x56
> [ 274.937054] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
> [ 274.937055] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
> [ 274.937057] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
> [ 274.937059] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
> [ 274.937061] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
> [ 274.937063] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
> [ 274.937065] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
> [ 274.937067] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d
> [ 274.937069] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde
> [ 274.937071] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
> [ 274.937073] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
> [ 274.937075] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 274.937077] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 274.937079] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 274.937081] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 274.937082] tar D 0000000000000000 0 980 826 0x00000000
> [ 274.937084] ffff88006f19f9e8 0000000000000086 00000000ffff9fd7 0000000000000000
> [ 274.937086] ffff88006d670000 ffff88006f19ffd8 ffff88006f19ffd8 0000000000013b40
> [ 274.937088] ffffffff81a0b020 ffff88006d670000 ffff88006f19f9b8 0000000100000282
> [ 274.937090] Call Trace:
> [ 274.937091] [<ffffffff8147acbe>] schedule_timeout+0xa7/0xde
> [ 274.937093] [<ffffffff8106015c>] ? del_timer+0x7a/0x7a
> [ 274.937095] [<ffffffff8147abc2>] io_schedule_timeout+0x6f/0x98
> [ 274.937097] [<ffffffff810ddc26>] balance_dirty_pages_ratelimited_nr+0x341/0x3b6
> [ 274.937100] [<ffffffff810d5efb>] generic_file_buffered_write+0x1dc/0x23a
> [ 274.937102] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
> [ 274.937105] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
> [ 274.937108] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
> [ 274.937111] [<ffffffff8111edab>] do_sync_write+0xbf/0xff
> [ 274.937112] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
> [ 274.937115] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
> [ 274.937118] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 274.937120] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 274.937122] [<ffffffff8111f421>] vfs_write+0xac/0xf3
> [ 274.937123] [<ffffffff8111f610>] sys_write+0x4a/0x6e
> [ 274.937126] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 274.937128] make D 0000000000000000 0 2252 2143 0x00000004
> [ 274.937130] ffff88006d631948 0000000000000082 ffff88006d6318d8 ffffffff00000000
> [ 274.937132] ffff880037d9dc00 ffff88006d631fd8 ffff88006d631fd8 0000000000013b40
> [ 274.937133] ffffffff81a0b020 ffff880037d9dc00 ffff88006d631908 000000016d631908
> [ 274.937135] Call Trace:
> [ 274.937137] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.937139] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 274.937140] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
> [ 274.937142] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
> [ 274.937144] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d
> [ 274.937146] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.937148] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 274.937149] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c
> [ 274.937151] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d
> [ 274.937153] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d
> [ 274.937155] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d
> [ 274.937156] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
> [ 274.937158] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 274.937161] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 274.937163] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
> [ 274.937165] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
> [ 274.937167] [<ffffffff8147c515>] ? page_fault+0x25/0x30
> [ 274.937169] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
> [ 274.937171] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
> [ 274.937173] [<ffffffff811322c7>] touch_atime+0x10e/0x131
> [ 274.937176] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640
> [ 274.937178] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
> [ 274.937180] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 274.937181] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 274.937183] [<ffffffff8111f511>] vfs_read+0xa9/0xf0
> [ 274.937185] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e
> [ 274.937187] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 274.937188] make D 0000000000000000 0 2262 2254 0x00000004
> [ 274.937190] ffff880070b2b948 0000000000000082 ffff880070b2b8d8 ffffffff00000000
> [ 274.937192] ffff88006d428000 ffff880070b2bfd8 ffff880070b2bfd8 0000000000013b40
> [ 274.937194] ffffffff81a0b020 ffff88006d428000 ffff880070b2b908 0000000170b2b908
> [ 274.937195] Call Trace:
> [ 274.937197] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.937199] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 274.937200] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
> [ 274.937202] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
> [ 274.937204] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d
> [ 274.937205] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.937208] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 274.937209] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c
> [ 274.937211] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d
> [ 274.937212] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d
> [ 274.937214] [<ffffffff811927b4>] ? __ext4_get_inode_loc+0x118/0x36d
> [ 274.937216] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
> [ 274.937217] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 274.937219] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 274.937221] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
> [ 274.937223] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
> [ 274.937225] [<ffffffff8147c515>] ? page_fault+0x25/0x30
> [ 274.937227] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
> [ 274.937228] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
> [ 274.937230] [<ffffffff811322c7>] touch_atime+0x10e/0x131
> [ 274.937232] [<ffffffff810d7360>] generic_file_aio_read+0x5d9/0x640
> [ 274.937234] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
> [ 274.937236] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 274.937238] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 274.937240] [<ffffffff8111f511>] vfs_read+0xa9/0xf0
> [ 274.937241] [<ffffffff8111f5a2>] sys_read+0x4a/0x6e
> [ 274.937243] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 274.937244] sh D 0000000000000000 0 2270 2253 0x00000004
> [ 274.937246] ffff8800709fd918 0000000000000086 ffff8800709fd8a8 ffffffff00000000
> [ 274.937248] ffff880037f60000 ffff8800709fdfd8 ffff8800709fdfd8 0000000000013b40
> [ 274.937250] ffff880071589700 ffff880037f60000 ffff8801002543c0 00000001005b89d0
> [ 274.937252] Call Trace:
> [ 274.937253] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.937255] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 274.937256] [<ffffffff81142210>] sleep_on_buffer+0xe/0x12
> [ 274.937258] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
> [ 274.937260] [<ffffffff8147ae56>] out_of_line_wait_on_bit_lock+0x72/0x7d
> [ 274.937261] [<ffffffff81142202>] ? wait_on_buffer+0x3a/0x3a
> [ 274.937264] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 274.937265] [<ffffffff8114240b>] __lock_buffer+0x38/0x3c
> [ 274.937267] [<ffffffff811c6ce9>] lock_buffer+0x39/0x3d
> [ 274.937268] [<ffffffff811c8162>] do_get_write_access+0x70/0x38d
> [ 274.937270] [<ffffffff81196fe4>] ? ext4_getblk+0x8e/0x153
> [ 274.937272] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 274.937274] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 274.937276] [<ffffffff8119c0ce>] ext4_add_entry+0x11f/0x8cf
> [ 274.937277] [<ffffffff811429f3>] ? __brelse+0x15/0x33
> [ 274.937280] [<ffffffff81191bac>] ? ext4_new_inode+0xc63/0xd0a
> [ 274.937282] [<ffffffff8119c89c>] ext4_add_nondir+0x1e/0x67
> [ 274.937283] [<ffffffff8119cb1b>] ext4_create+0xf5/0x13e
> [ 274.937286] [<ffffffff81129b12>] vfs_create+0x6c/0x8e
> [ 274.937288] [<ffffffff81129dad>] do_last+0x279/0x5ab
> [ 274.937289] [<ffffffff8112ac73>] path_openat+0xc8/0x31c
> [ 274.937291] [<ffffffff810f2332>] ? handle_mm_fault+0x1ac/0x1bf
> [ 274.937293] [<ffffffff8112aeff>] do_filp_open+0x38/0x86
> [ 274.937295] [<ffffffff81233de1>] ? might_fault+0x21/0x23
> [ 274.937298] [<ffffffff8113477b>] ? alloc_fd+0x72/0x11d
> [ 274.937299] [<ffffffff8111e995>] do_sys_open+0x6e/0x100
> [ 274.937301] [<ffffffff8111ea47>] sys_open+0x20/0x22
> [ 274.937303] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 274.937304] gcc D 0000000000000000 0 2272 2271 0x00000004
> [ 274.937306] ffff88006f197c58 0000000000000086 ffff88006f197d14 0000000000000000
> [ 274.937307] ffff880037f64500 ffff88006f197fd8 ffff88006f197fd8 0000000000013b40
> [ 274.937309] ffffffff81a0b020 ffff880037f64500 ffff88006f197c58 000000018106e7c3
> [ 274.937311] Call Trace:
> [ 274.937313] [<ffffffff811c6fd5>] start_this_handle+0x2e8/0x465
> [ 274.937315] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 274.937317] [<ffffffff811c7418>] jbd2__journal_start+0x94/0xda
> [ 274.937318] [<ffffffff811c7471>] jbd2_journal_start+0x13/0x15
> [ 274.937320] [<ffffffff811ab92a>] ext4_journal_start_sb+0x108/0x120
> [ 274.937323] [<ffffffff8119cc24>] ext4_symlink+0xc0/0x212
> [ 274.937325] [<ffffffff81129928>] vfs_symlink+0x54/0x74
> [ 274.937326] [<ffffffff8112b40a>] sys_symlinkat+0x96/0xef
> [ 274.937329] [<ffffffff81122d89>] ? sys_newstat+0x2a/0x33
> [ 274.937330] [<ffffffff8112b479>] sys_symlink+0x16/0x18
> [ 274.937332] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 274.937335] Sched Debug Version: v0.10, 2.6.39-rc4+ #1
> [ 274.937336] ktime : 275382.978028
> [ 274.937337] sched_clk : 274937.334155
> [ 274.937338] cpu_clk : 274937.334189
> [ 274.937339] jiffies : 4294942678
> [ 274.937340] sched_clock_stable : 1
> [ 274.937341]
> [ 274.937342] sysctl_sched
> [ 274.937342] .sysctl_sched_latency : 18.000000
> [ 274.937344] .sysctl_sched_min_granularity : 2.250000
> [ 274.937345] .sysctl_sched_wakeup_granularity : 3.000000
> [ 274.937346] .sysctl_sched_child_runs_first : 0
> [ 274.937347] .sysctl_sched_features : 7279
> [ 274.937348] .sysctl_sched_tunable_scaling : 1 (logaritmic)
> [ 274.937349]
> [ 274.937350] cpu#0, 2492.220 MHz
> [ 274.937350] .nr_running : 0
> [ 274.937351] .load : 0
> [ 274.937352] .nr_switches : 105039
> [ 274.937353] .nr_load_updates : 121182
> [ 274.937354] .nr_uninterruptible : 8
> [ 274.937355] .next_balance : 4294.942607
> [ 274.937356] .curr->pid : 0
> [ 274.937357] .clock : 274864.523786
> [ 274.937359] .cpu_load[0] : 0
> [ 274.937360] .cpu_load[1] : 0
> [ 274.937361] .cpu_load[2] : 0
> [ 274.937362] .cpu_load[3] : 0
> [ 274.937363] .cpu_load[4] : 0
> [ 274.937364] .yld_count : 0
> [ 274.937364] .sched_switch : 0
> [ 274.937365] .sched_count : 107487
> [ 274.937366] .sched_goidle : 43368
> [ 274.937367] .avg_idle : 1000000
> [ 274.937368] .ttwu_count : 56534
> [ 274.937369] .ttwu_local : 53925
> [ 274.937370] .bkl_count : 0
> [ 274.937371]
> [ 274.937372] cfs_rq[0]:/system
> [ 274.937372] .exec_clock : 5906.036519
> [ 274.937374] .MIN_vruntime : 0.000001
> [ 274.937375] .min_vruntime : 7138.254657
> [ 274.937376] .max_vruntime : 0.000001
> [ 274.937377] .spread : 0.000000
> [ 274.937378] .spread0 : -14302.149798
> [ 274.937379] .nr_spread_over : 0
> [ 274.937379] .nr_running : 0
> [ 274.937380] .load : 0
> [ 274.937381] .load_avg : 159.999997
> [ 274.937382] .load_period : 5.496191
> [ 274.937383] .load_contrib : 29
> [ 274.937384] .load_tg : 29
> [ 274.937385] .se->exec_start : 272446.423795
> [ 274.937386] .se->vruntime : 21431.398168
> [ 274.937387] .se->sum_exec_runtime : 5906.111805
> [ 274.937388] .se->statistics.wait_start : 0.000000
> [ 274.937389] .se->statistics.sleep_start : 0.000000
> [ 274.937390] .se->statistics.block_start : 0.000000
> [ 274.937391] .se->statistics.sleep_max : 0.000000
> [ 274.937392] .se->statistics.block_max : 0.000000
> [ 274.937393] .se->statistics.exec_max : 15.266617
> [ 274.937394] .se->statistics.slice_max : 5.258344
> [ 274.937395] .se->statistics.wait_max : 7.305679
> [ 274.937396] .se->statistics.wait_sum : 190.506686
> [ 274.937397] .se->statistics.wait_count : 18946
> [ 274.937398] .se->load.weight : 2
> [ 274.937399]
> [ 274.937399] cfs_rq[0]:/
> [ 274.937400] .exec_clock : 17409.691339
> [ 274.937401] .MIN_vruntime : 0.000001
> [ 274.937402] .min_vruntime : 21440.404455
> [ 274.937403] .max_vruntime : 0.000001
> [ 274.937403] .spread : 0.000000
> [ 274.937404] .spread0 : 0.000000
> [ 274.937405] .nr_spread_over : 58
> [ 274.937406] .nr_running : 0
> [ 274.937407] .load : 0
> [ 274.937408] .load_avg : 0.000000
> [ 274.937409] .load_period : 0.000000
> [ 274.937409] .load_contrib : 0
> [ 274.937410] .load_tg : 0
> [ 274.937411]
> [ 274.937412] cfs_rq[0]:/system/rescue.service
> [ 274.937413] .exec_clock : 4319.868509
> [ 274.937414] .MIN_vruntime : 0.000001
> [ 274.937414] .min_vruntime : 4255.887524
> [ 274.937415] .max_vruntime : 0.000001
> [ 274.937416] .spread : 0.000000
> [ 274.937417] .spread0 : -17184.516931
> [ 274.937418] .nr_spread_over : 24
> [ 274.937419] .nr_running : 0
> [ 274.937420] .load : 0
> [ 274.937421] .load_avg : 160.004112
> [ 274.937421] .load_period : 5.496149
> [ 274.937422] .load_contrib : 29
> [ 274.937423] .load_tg : 29
> [ 274.937424] .se->exec_start : 272446.423795
> [ 274.937425] .se->vruntime : 7138.254657
> [ 274.937426] .se->sum_exec_runtime : 4319.796162
> [ 274.937427] .se->statistics.wait_start : 0.000000
> [ 274.937428] .se->statistics.sleep_start : 0.000000
> [ 274.937429] .se->statistics.block_start : 0.000000
> [ 274.937430] .se->statistics.sleep_max : 0.000000
> [ 274.937431] .se->statistics.block_max : 0.000000
> [ 274.937432] .se->statistics.exec_max : 6.213083
> [ 274.937432] .se->statistics.slice_max : 7.894226
> [ 274.937433] .se->statistics.wait_max : 4.870567
> [ 274.937434] .se->statistics.wait_sum : 91.737803
> [ 274.937435] .se->statistics.wait_count : 15504
> [ 274.937436] .se->load.weight : 2
> [ 274.937437]
> [ 274.937438] runnable tasks:
> [ 274.937438] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 274.937439] ----------------------------------------------------------------------------------------------------------
> [ 274.937444]
> [ 274.937445] cpu#1, 2492.220 MHz
> [ 274.937445] .nr_running : 0
> [ 274.937446] .load : 0
> [ 274.937447] .nr_switches : 35274
> [ 274.937448] .nr_load_updates : 105259
> [ 274.937449] .nr_uninterruptible : 1
> [ 274.937449] .next_balance : 4294.942681
> [ 274.937450] .curr->pid : 0
> [ 274.937451] .clock : 274937.357922
> [ 274.937452] .cpu_load[0] : 0
> [ 274.937453] .cpu_load[1] : 0
> [ 274.937454] .cpu_load[2] : 0
> [ 274.937454] .cpu_load[3] : 0
> [ 274.937455] .cpu_load[4] : 0
> [ 274.937456] .yld_count : 0
> [ 274.937457] .sched_switch : 0
> [ 274.937458] .sched_count : 35560
> [ 274.937458] .sched_goidle : 14561
> [ 274.937459] .avg_idle : 1000000
> [ 274.937460] .ttwu_count : 16938
> [ 274.937461] .ttwu_local : 14491
> [ 274.937462] .bkl_count : 0
> [ 274.937463]
> [ 274.937463] cfs_rq[1]:/
> [ 274.937464] .exec_clock : 5221.725565
> [ 274.937465] .MIN_vruntime : 0.000001
> [ 274.937466] .min_vruntime : 9077.562937
> [ 274.937466] .max_vruntime : 0.000001
> [ 274.937467] .spread : 0.000000
> [ 274.937468] .spread0 : -12362.841518
> [ 274.937469] .nr_spread_over : 69
> [ 274.937470] .nr_running : 0
> [ 274.937471] .load : 0
> [ 274.937472] .load_avg : 0.000000
> [ 274.937473] .load_period : 0.000000
> [ 274.937473] .load_contrib : 0
> [ 274.937474] .load_tg : 0
> [ 274.937475]
> [ 274.937476] runnable tasks:
> [ 274.937476] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 274.937477] ----------------------------------------------------------------------------------------------------------
> [ 274.937481]
> [ 274.937482] cpu#2, 2492.220 MHz
> [ 274.937482] .nr_running : 4
> [ 274.937483] .load : 2048
> [ 274.937484] .nr_switches : 7092
> [ 274.937485] .nr_load_updates : 100502
> [ 274.937486] .nr_uninterruptible : 0
> [ 274.937487] .next_balance : 4294.942806
> [ 274.937487] .curr->pid : 46
> [ 274.937488] .clock : 183206.600407
> [ 274.937489] .cpu_load[0] : 2048
> [ 274.937490] .cpu_load[1] : 2048
> [ 274.937491] .cpu_load[2] : 2048
> [ 274.937492] .cpu_load[3] : 2048
> [ 274.937492] .cpu_load[4] : 2048
> [ 274.937493] .yld_count : 0
> [ 274.937494] .sched_switch : 0
> [ 274.937495] .sched_count : 7323
> [ 274.937496] .sched_goidle : 2831
> [ 274.937496] .avg_idle : 1000000
> [ 274.937497] .ttwu_count : 3275
> [ 274.937498] .ttwu_local : 2028
> [ 274.937499] .bkl_count : 0
> [ 274.937500]
> [ 274.937500] cfs_rq[2]:/
> [ 274.937501] .exec_clock : 2187.192139
> [ 274.937502] .MIN_vruntime : 5109.505143
> [ 274.937503] .min_vruntime : 5118.505143
> [ 274.937504] .max_vruntime : 5109.505143
> [ 274.937505] .spread : 0.000000
> [ 274.937506] .spread0 : -16321.899312
> [ 274.937506] .nr_spread_over : 31
> [ 274.937507] .nr_running : 2
> [ 274.937508] .load : 2048
> [ 274.937509] .load_avg : 0.000000
> [ 274.937510] .load_period : 0.000000
> [ 274.937511] .load_contrib : 0
> [ 274.937511] .load_tg : 0
> [ 274.937512]
> [ 274.937513] rt_rq[2]:/
> [ 274.937513] .rt_nr_running : 1
> [ 274.937514] .rt_throttled : 0
> [ 274.937515] .rt_time : 0.000000
> [ 274.937516] .rt_runtime : 950.000000
> [ 274.937517]
> [ 274.937517] runnable tasks:
> [ 274.937518] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 274.937519] ----------------------------------------------------------------------------------------------------------
> [ 274.937520] migration/2 13 0.000000 113 0 0.000000 0.000815 0.000000 /
> [ 274.937524] watchdog/2 16 0.000000 17 0 0.000000 1.785026 0.002124 /
> [ 274.937527] R kswapd0 46 5118.505143 370 120 5118.505143 415.766870 181943.841718 /
> [ 274.937531] kworker/2:1 76 5109.505143 1083 120 5109.505143 25.615305 181183.059001 /
> [ 274.937535]
> [ 274.937536] cpu#3, 2492.220 MHz
> [ 274.937536] .nr_running : 1
> [ 274.937537] .load : 1024
> [ 274.937538] .nr_switches : 22324
> [ 274.937539] .nr_load_updates : 59855
> [ 274.937540] .nr_uninterruptible : 0
> [ 274.937541] .next_balance : 4294.942679
> [ 274.937542] .curr->pid : 73
> [ 274.937543] .clock : 274936.938702
> [ 274.937543] .cpu_load[0] : 0
> [ 274.937544] .cpu_load[1] : 0
> [ 274.937545] .cpu_load[2] : 0
> [ 274.937546] .cpu_load[3] : 0
> [ 274.937546] .cpu_load[4] : 0
> [ 274.937547] .yld_count : 123
> [ 274.937548] .sched_switch : 0
> [ 274.937549] .sched_count : 22774
> [ 274.937550] .sched_goidle : 9952
> [ 274.937551] .avg_idle : 946660
> [ 274.937551] .ttwu_count : 10075
> [ 274.937552] .ttwu_local : 8964
> [ 274.937553] .bkl_count : 0
> [ 274.937554]
> [ 274.937554] cfs_rq[3]:/
> [ 274.937555] .exec_clock : 3244.912177
> [ 274.937556] .MIN_vruntime : 0.000001
> [ 274.937557] .min_vruntime : 7517.482154
> [ 274.937558] .max_vruntime : 0.000001
> [ 274.937559] .spread : 0.000000
> [ 274.937559] .spread0 : -13922.922301
> [ 274.937560] .nr_spread_over : 128
> [ 274.937561] .nr_running : 1
> [ 274.937562] .load : 1024
> [ 274.937563] .load_avg : 0.000000
> [ 274.937564] .load_period : 0.000000
> [ 274.937565] .load_contrib : 0
> [ 274.937565] .load_tg : 0
> [ 274.937567]
> [ 274.937567] runnable tasks:
> [ 274.937567] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 274.937568] ----------------------------------------------------------------------------------------------------------
> [ 274.937571] R kworker/3:1 73 7517.482154 7294 120 7517.482154 264.876541 272702.297840 /
> [ 274.937575]
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-27 17:50 ` James Bottomley
@ 2011-04-27 20:05 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-27 20:05 UTC (permalink / raw)
To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel, linux-ext4
On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote:
> To test the theory, Chris asked me to try with data=ordered.
> Unfortunately, the deadlock still shows up. This is what I get.
As another data point: I'm trying the same kernel with CONFIG_PREEMPT
enabled. This time the deadlock doesn't happen. Instead, kswapd0 gets
pegged at 99% CPU for much of the untar, but it does eventually
complete.
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-27 20:05 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-27 20:05 UTC (permalink / raw)
To: Chris Mason; +Cc: linux-fsdevel, linux-mm, linux-kernel, linux-ext4
On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote:
> To test the theory, Chris asked me to try with data=ordered.
> Unfortunately, the deadlock still shows up. This is what I get.
As another data point: I'm trying the same kernel with CONFIG_PREEMPT
enabled. This time the deadlock doesn't happen. Instead, kswapd0 gets
pegged at 99% CPU for much of the untar, but it does eventually
complete.
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-27 17:21 ` James Bottomley
@ 2011-04-28 0:37 ` Dave Chinner
-1 siblings, 0 replies; 138+ messages in thread
From: Dave Chinner @ 2011-04-28 0:37 UTC (permalink / raw)
To: James Bottomley; +Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel
On Wed, Apr 27, 2011 at 12:21:42PM -0500, James Bottomley wrote:
> On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
> > Ok, I'd try turning it on so we catch the sleeping with a spinlock held
> > case better.
>
> Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
>
> This is the trace with sysrq-l and sysrq-w
>
> The repro this time doesn't have a soft lockup, just the tar is hung and
> one of my CPUs is in 99% system.
>
> James
>
> ---
> [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000
> [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
> [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
> [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
> [ 454.753048] Call Trace:
> [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
> [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
> [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
> [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
> [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
> [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
> [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
> [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
> [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
> [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
> [ 454.791703] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
> [ 454.794334] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
> [ 454.796962] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
> [ 454.799582] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
> [ 454.802204] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
> [ 454.804803] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
> [ 454.807427] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d
> [ 454.810077] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde
> [ 454.812776] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
> [ 454.815464] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
> [ 454.818129] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 454.820808] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 454.823452] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 454.826103] [<ffffffff81483760>] ? gs_change+0x13/0x13
Looks like it is blocked waiting for the journal.
> [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000
> [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
> [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
> [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
> [ 454.839453] Call Trace:
> [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12
> [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
> [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
> [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
> [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
> [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
> [ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
> [ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
> [ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
> [ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13
which is blocked waiting for _data_ IO completion.
> [ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000
> [ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282
> [ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40
> [ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3
> [ 454.899530] Call Trace:
> [ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
> [ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> [ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
> [ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
> [ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
> [ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
> [ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
> [ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e
> [ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204
> [ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a
> [ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
> [ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
> [ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
> [ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff
> [ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
> [ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
> [ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3
> [ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e
> [ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
And tar is blocked waiting for the journal, too.
Looks like a case of a ordered mode data writeback journal stall
(the same basic problem as the ext3 fsync sync-the-world issue).
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 0:37 ` Dave Chinner
0 siblings, 0 replies; 138+ messages in thread
From: Dave Chinner @ 2011-04-28 0:37 UTC (permalink / raw)
To: James Bottomley; +Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel
On Wed, Apr 27, 2011 at 12:21:42PM -0500, James Bottomley wrote:
> On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
> > Ok, I'd try turning it on so we catch the sleeping with a spinlock held
> > case better.
>
> Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
>
> This is the trace with sysrq-l and sysrq-w
>
> The repro this time doesn't have a soft lockup, just the tar is hung and
> one of my CPUs is in 99% system.
>
> James
>
> ---
> [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000
> [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
> [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
> [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
> [ 454.753048] Call Trace:
> [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
> [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
> [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
> [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
> [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
> [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
> [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
> [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
> [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
> [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
> [ 454.791703] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
> [ 454.794334] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
> [ 454.796962] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
> [ 454.799582] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
> [ 454.802204] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
> [ 454.804803] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
> [ 454.807427] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d
> [ 454.810077] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde
> [ 454.812776] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
> [ 454.815464] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
> [ 454.818129] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 454.820808] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 454.823452] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 454.826103] [<ffffffff81483760>] ? gs_change+0x13/0x13
Looks like it is blocked waiting for the journal.
> [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000
> [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
> [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
> [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
> [ 454.839453] Call Trace:
> [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12
> [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
> [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
> [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
> [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
> [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
> [ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
> [ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
> [ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
> [ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13
which is blocked waiting for _data_ IO completion.
> [ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000
> [ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282
> [ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40
> [ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3
> [ 454.899530] Call Trace:
> [ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
> [ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> [ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
> [ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
> [ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
> [ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
> [ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
> [ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e
> [ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204
> [ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a
> [ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
> [ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
> [ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
> [ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff
> [ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
> [ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
> [ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3
> [ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e
> [ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
And tar is blocked waiting for the journal, too.
Looks like a case of a ordered mode data writeback journal stall
(the same basic problem as the ext3 fsync sync-the-world issue).
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-27 20:05 ` James Bottomley
@ 2011-04-28 11:36 ` Colin Ian King
-1 siblings, 0 replies; 138+ messages in thread
From: Colin Ian King @ 2011-04-28 11:36 UTC (permalink / raw)
To: James Bottomley
Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4
One more data point to add, I've been looking at an identical issue when
copying large amounts of data. I bisected this - and the lockups occur
with commit
3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
issue. With this commit, my file copy test locks up after ~8-10
iterations, before this commit I can copy > 100 times and don't see the
lockup.
On Wed, 2011-04-27 at 15:05 -0500, James Bottomley wrote:
> On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote:
> > To test the theory, Chris asked me to try with data=ordered.
> > Unfortunately, the deadlock still shows up. This is what I get.
>
> As another data point: I'm trying the same kernel with CONFIG_PREEMPT
> enabled. This time the deadlock doesn't happen. Instead, kswapd0 gets
> pegged at 99% CPU for much of the untar, but it does eventually
> complete.
>
> James
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 11:36 ` Colin Ian King
0 siblings, 0 replies; 138+ messages in thread
From: Colin Ian King @ 2011-04-28 11:36 UTC (permalink / raw)
To: James Bottomley
Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4
One more data point to add, I've been looking at an identical issue when
copying large amounts of data. I bisected this - and the lockups occur
with commit
3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
issue. With this commit, my file copy test locks up after ~8-10
iterations, before this commit I can copy > 100 times and don't see the
lockup.
On Wed, 2011-04-27 at 15:05 -0500, James Bottomley wrote:
> On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote:
> > To test the theory, Chris asked me to try with data=ordered.
> > Unfortunately, the deadlock still shows up. This is what I get.
>
> As another data point: I'm trying the same kernel with CONFIG_PREEMPT
> enabled. This time the deadlock doesn't happen. Instead, kswapd0 gets
> pegged at 99% CPU for much of the untar, but it does eventually
> complete.
>
> James
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 11:36 ` Colin Ian King
@ 2011-04-28 12:29 ` Chris Mason
-1 siblings, 0 replies; 138+ messages in thread
From: Chris Mason @ 2011-04-28 12:29 UTC (permalink / raw)
To: Colin Ian King
Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4
Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400:
> One more data point to add, I've been looking at an identical issue when
> copying large amounts of data. I bisected this - and the lockups occur
> with commit
> 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> issue. With this commit, my file copy test locks up after ~8-10
> iterations, before this commit I can copy > 100 times and don't see the
> lockup.
Well, that's really interesting. I tried with compaction on here and
couldn't trigger it, but this (very very lightly) tested patch might
help.
It moves the writeout throttle before the goto restart, and also makes
sure we do at least one cond_resched before we loop.
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 6771ea7..cb08b41 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1934,12 +1934,14 @@ restart:
if (inactive_anon_is_low(zone, sc))
shrink_active_list(SWAP_CLUSTER_MAX, zone, sc, priority, 0);
+ throttle_vm_writeout(sc->gfp_mask);
+
/* reclaim/compaction might need reclaim to continue */
if (should_continue_reclaim(zone, nr_reclaimed,
- sc->nr_scanned - nr_scanned, sc))
+ sc->nr_scanned - nr_scanned, sc)) {
+ cond_resched();
goto restart;
-
- throttle_vm_writeout(sc->gfp_mask);
+ }
}
/*
^ permalink raw reply related [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 12:29 ` Chris Mason
0 siblings, 0 replies; 138+ messages in thread
From: Chris Mason @ 2011-04-28 12:29 UTC (permalink / raw)
To: Colin Ian King
Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4
Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400:
> One more data point to add, I've been looking at an identical issue when
> copying large amounts of data. I bisected this - and the lockups occur
> with commit
> 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> issue. With this commit, my file copy test locks up after ~8-10
> iterations, before this commit I can copy > 100 times and don't see the
> lockup.
Well, that's really interesting. I tried with compaction on here and
couldn't trigger it, but this (very very lightly) tested patch might
help.
It moves the writeout throttle before the goto restart, and also makes
sure we do at least one cond_resched before we loop.
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 6771ea7..cb08b41 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1934,12 +1934,14 @@ restart:
if (inactive_anon_is_low(zone, sc))
shrink_active_list(SWAP_CLUSTER_MAX, zone, sc, priority, 0);
+ throttle_vm_writeout(sc->gfp_mask);
+
/* reclaim/compaction might need reclaim to continue */
if (should_continue_reclaim(zone, nr_reclaimed,
- sc->nr_scanned - nr_scanned, sc))
+ sc->nr_scanned - nr_scanned, sc)) {
+ cond_resched();
goto restart;
-
- throttle_vm_writeout(sc->gfp_mask);
+ }
}
/*
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 12:29 ` Chris Mason
@ 2011-04-28 13:42 ` Colin Ian King
-1 siblings, 0 replies; 138+ messages in thread
From: Colin Ian King @ 2011-04-28 13:42 UTC (permalink / raw)
To: Chris Mason
Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4
On Thu, 2011-04-28 at 08:29 -0400, Chris Mason wrote:
> Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400:
> > One more data point to add, I've been looking at an identical issue when
> > copying large amounts of data. I bisected this - and the lockups occur
> > with commit
> > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > issue. With this commit, my file copy test locks up after ~8-10
> > iterations, before this commit I can copy > 100 times and don't see the
> > lockup.
>
> Well, that's really interesting. I tried with compaction on here and
> couldn't trigger it, but this (very very lightly) tested patch might
> help.
>
Thanks Chris,
I've given this a soak test but I still see the same lockup.
> It moves the writeout throttle before the goto restart, and also makes
> sure we do at least one cond_resched before we loop.
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 6771ea7..cb08b41 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -1934,12 +1934,14 @@ restart:
> if (inactive_anon_is_low(zone, sc))
> shrink_active_list(SWAP_CLUSTER_MAX, zone, sc, priority, 0);
>
> + throttle_vm_writeout(sc->gfp_mask);
> +
> /* reclaim/compaction might need reclaim to continue */
> if (should_continue_reclaim(zone, nr_reclaimed,
> - sc->nr_scanned - nr_scanned, sc))
> + sc->nr_scanned - nr_scanned, sc)) {
> + cond_resched();
> goto restart;
> -
> - throttle_vm_writeout(sc->gfp_mask);
> + }
> }
>
> /*
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 13:42 ` Colin Ian King
0 siblings, 0 replies; 138+ messages in thread
From: Colin Ian King @ 2011-04-28 13:42 UTC (permalink / raw)
To: Chris Mason
Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4
On Thu, 2011-04-28 at 08:29 -0400, Chris Mason wrote:
> Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400:
> > One more data point to add, I've been looking at an identical issue when
> > copying large amounts of data. I bisected this - and the lockups occur
> > with commit
> > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > issue. With this commit, my file copy test locks up after ~8-10
> > iterations, before this commit I can copy > 100 times and don't see the
> > lockup.
>
> Well, that's really interesting. I tried with compaction on here and
> couldn't trigger it, but this (very very lightly) tested patch might
> help.
>
Thanks Chris,
I've given this a soak test but I still see the same lockup.
> It moves the writeout throttle before the goto restart, and also makes
> sure we do at least one cond_resched before we loop.
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 6771ea7..cb08b41 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -1934,12 +1934,14 @@ restart:
> if (inactive_anon_is_low(zone, sc))
> shrink_active_list(SWAP_CLUSTER_MAX, zone, sc, priority, 0);
>
> + throttle_vm_writeout(sc->gfp_mask);
> +
> /* reclaim/compaction might need reclaim to continue */
> if (should_continue_reclaim(zone, nr_reclaimed,
> - sc->nr_scanned - nr_scanned, sc))
> + sc->nr_scanned - nr_scanned, sc)) {
> + cond_resched();
> goto restart;
> -
> - throttle_vm_writeout(sc->gfp_mask);
> + }
> }
>
> /*
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 13:42 ` Colin Ian King
@ 2011-04-28 13:45 ` Chris Mason
-1 siblings, 0 replies; 138+ messages in thread
From: Chris Mason @ 2011-04-28 13:45 UTC (permalink / raw)
To: Colin Ian King
Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4
Excerpts from Colin Ian King's message of 2011-04-28 09:42:20 -0400:
>
> On Thu, 2011-04-28 at 08:29 -0400, Chris Mason wrote:
> > Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400:
> > > One more data point to add, I've been looking at an identical issue when
> > > copying large amounts of data. I bisected this - and the lockups occur
> > > with commit
> > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > > issue. With this commit, my file copy test locks up after ~8-10
> > > iterations, before this commit I can copy > 100 times and don't see the
> > > lockup.
> >
> > Well, that's really interesting. I tried with compaction on here and
> > couldn't trigger it, but this (very very lightly) tested patch might
> > help.
> >
> Thanks Chris,
>
> I've given this a soak test but I still see the same lockup.
Could you post the soft lockups you're seeing?
-chris
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 13:45 ` Chris Mason
0 siblings, 0 replies; 138+ messages in thread
From: Chris Mason @ 2011-04-28 13:45 UTC (permalink / raw)
To: Colin Ian King
Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4
Excerpts from Colin Ian King's message of 2011-04-28 09:42:20 -0400:
>
> On Thu, 2011-04-28 at 08:29 -0400, Chris Mason wrote:
> > Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400:
> > > One more data point to add, I've been looking at an identical issue when
> > > copying large amounts of data. I bisected this - and the lockups occur
> > > with commit
> > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > > issue. With this commit, my file copy test locks up after ~8-10
> > > iterations, before this commit I can copy > 100 times and don't see the
> > > lockup.
> >
> > Well, that's really interesting. I tried with compaction on here and
> > couldn't trigger it, but this (very very lightly) tested patch might
> > help.
> >
> Thanks Chris,
>
> I've given this a soak test but I still see the same lockup.
Could you post the soft lockups you're seeing?
-chris
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 11:36 ` Colin Ian King
@ 2011-04-28 13:52 ` Jan Kara
-1 siblings, 0 replies; 138+ messages in thread
From: Jan Kara @ 2011-04-28 13:52 UTC (permalink / raw)
To: colin.king
Cc: James Bottomley, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu 28-04-11 12:36:30, Colin Ian King wrote:
> One more data point to add, I've been looking at an identical issue when
> copying large amounts of data. I bisected this - and the lockups occur
> with commit
> 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> issue. With this commit, my file copy test locks up after ~8-10
> iterations, before this commit I can copy > 100 times and don't see the
> lockup.
Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit
of yours causes kswapd on non-preempt kernels spin for a *long* time...
Honza
> On Wed, 2011-04-27 at 15:05 -0500, James Bottomley wrote:
> > On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote:
> > > To test the theory, Chris asked me to try with data=ordered.
> > > Unfortunately, the deadlock still shows up. This is what I get.
> >
> > As another data point: I'm trying the same kernel with CONFIG_PREEMPT
> > enabled. This time the deadlock doesn't happen. Instead, kswapd0 gets
> > pegged at 99% CPU for much of the untar, but it does eventually
> > complete.
> >
> > James
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at http://www.tux.org/lkml/
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 13:52 ` Jan Kara
0 siblings, 0 replies; 138+ messages in thread
From: Jan Kara @ 2011-04-28 13:52 UTC (permalink / raw)
To: colin.king
Cc: James Bottomley, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu 28-04-11 12:36:30, Colin Ian King wrote:
> One more data point to add, I've been looking at an identical issue when
> copying large amounts of data. I bisected this - and the lockups occur
> with commit
> 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> issue. With this commit, my file copy test locks up after ~8-10
> iterations, before this commit I can copy > 100 times and don't see the
> lockup.
Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit
of yours causes kswapd on non-preempt kernels spin for a *long* time...
Honza
> On Wed, 2011-04-27 at 15:05 -0500, James Bottomley wrote:
> > On Wed, 2011-04-27 at 12:50 -0500, James Bottomley wrote:
> > > To test the theory, Chris asked me to try with data=ordered.
> > > Unfortunately, the deadlock still shows up. This is what I get.
> >
> > As another data point: I'm trying the same kernel with CONFIG_PREEMPT
> > enabled. This time the deadlock doesn't happen. Instead, kswapd0 gets
> > pegged at 99% CPU for much of the untar, but it does eventually
> > complete.
> >
> > James
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at http://www.tux.org/lkml/
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 13:45 ` Chris Mason
(?)
@ 2011-04-28 14:01 ` Colin Ian King
2011-04-28 14:04 ` Chris Mason
2011-04-28 14:25 ` Jan Kara
-1 siblings, 2 replies; 138+ messages in thread
From: Colin Ian King @ 2011-04-28 14:01 UTC (permalink / raw)
To: Chris Mason
Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4
[-- Attachment #1: Type: text/plain, Size: 349 bytes --]
> Could you post the soft lockups you're seeing?
As requested, attached
Colin
>
> -chris
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
[-- Attachment #2: call-trace.log --]
[-- Type: text/x-log, Size: 26550 bytes --]
Apr 28 14:35:35 ubuntu kernel: [ 257.112169] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds.
Apr 28 14:46:15 ubuntu kernel: [ 257.112179] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 14:46:15 ubuntu kernel: [ 257.112182] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000
Apr 28 14:46:15 ubuntu kernel: [ 257.112186] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000
Apr 28 14:46:15 ubuntu kernel: [ 257.112190] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00
Apr 28 14:46:15 ubuntu kernel: [ 257.112194] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00
Apr 28 14:46:15 ubuntu kernel: [ 257.112198] Call Trace:
Apr 28 14:46:15 ubuntu kernel: [ 257.112205] [<ffffffff8110c070>] ? sync_page+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 257.112210] [<ffffffff815c0990>] io_schedule+0x70/0xc0
Apr 28 14:46:15 ubuntu kernel: [ 257.112213] [<ffffffff8110c0b0>] sync_page+0x40/0x50
Apr 28 14:46:15 ubuntu kernel: [ 257.112216] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
Apr 28 14:46:15 ubuntu kernel: [ 257.112220] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
Apr 28 14:46:15 ubuntu kernel: [ 257.112223] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 257.112227] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
Apr 28 14:46:15 ubuntu kernel: [ 257.112230] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
Apr 28 14:46:15 ubuntu kernel: [ 257.112234] [<ffffffff8112d144>] __do_fault+0x54/0x520
Apr 28 14:46:15 ubuntu kernel: [ 257.112237] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
Apr 28 14:46:15 ubuntu kernel: [ 257.112240] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
Apr 28 14:46:15 ubuntu kernel: [ 257.112242] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
Apr 28 14:46:15 ubuntu kernel: [ 257.112245] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
Apr 28 14:46:15 ubuntu kernel: [ 257.112249] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
Apr 28 14:46:15 ubuntu kernel: [ 257.112252] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
Apr 28 14:46:15 ubuntu kernel: [ 257.112255] [<ffffffff815c34d5>] page_fault+0x25/0x30
Apr 28 14:46:15 ubuntu kernel: [ 257.112260] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds.
Apr 28 14:46:15 ubuntu kernel: [ 257.112262] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 14:46:15 ubuntu kernel: [ 257.112264] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000
Apr 28 14:46:15 ubuntu kernel: [ 257.112267] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000
Apr 28 14:46:15 ubuntu kernel: [ 257.112271] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00
Apr 28 14:46:15 ubuntu kernel: [ 257.112275] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00
Apr 28 14:46:15 ubuntu kernel: [ 257.112279] Call Trace:
Apr 28 14:46:15 ubuntu kernel: [ 257.112282] [<ffffffff8110c070>] ? sync_page+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 257.112285] [<ffffffff815c0990>] io_schedule+0x70/0xc0
Apr 28 14:46:15 ubuntu kernel: [ 257.112288] [<ffffffff8110c0b0>] sync_page+0x40/0x50
Apr 28 14:46:15 ubuntu kernel: [ 257.112291] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
Apr 28 14:46:15 ubuntu kernel: [ 257.112294] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
Apr 28 14:46:15 ubuntu kernel: [ 257.112297] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 257.112300] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190
Apr 28 14:46:15 ubuntu kernel: [ 257.112304] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30
Apr 28 14:46:15 ubuntu kernel: [ 257.112307] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170
Apr 28 14:46:15 ubuntu kernel: [ 257.112311] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190
Apr 28 14:46:15 ubuntu kernel: [ 257.112315] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0
Apr 28 14:46:15 ubuntu kernel: [ 257.112319] [<ffffffff81247e9b>] kjournald2+0xbb/0x220
Apr 28 14:46:15 ubuntu kernel: [ 257.112322] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
Apr 28 14:46:15 ubuntu kernel: [ 257.112325] [<ffffffff81247de0>] ? kjournald2+0x0/0x220
Apr 28 14:46:15 ubuntu kernel: [ 257.112328] [<ffffffff810877e6>] kthread+0x96/0xa0
Apr 28 14:46:15 ubuntu kernel: [ 257.112332] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
Apr 28 14:46:15 ubuntu kernel: [ 257.112335] [<ffffffff81087750>] ? kthread+0x0/0xa0
Apr 28 14:46:15 ubuntu kernel: [ 257.112337] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
Apr 28 14:46:15 ubuntu kernel: [ 257.112340] INFO: task dirname:5969 blocked for more than 30 seconds.
Apr 28 14:46:15 ubuntu kernel: [ 257.112341] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 14:46:15 ubuntu kernel: [ 257.112343] dirname D 0000000000000002 0 5969 5214 0x00000000
Apr 28 14:46:15 ubuntu kernel: [ 257.112347] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000
Apr 28 14:46:15 ubuntu kernel: [ 257.112351] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00
Apr 28 14:46:15 ubuntu kernel: [ 257.112354] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00
Apr 28 14:46:15 ubuntu kernel: [ 257.112358] Call Trace:
Apr 28 14:46:15 ubuntu kernel: [ 257.112361] [<ffffffff8110c070>] ? sync_page+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 257.112364] [<ffffffff815c0990>] io_schedule+0x70/0xc0
Apr 28 14:46:15 ubuntu kernel: [ 257.112367] [<ffffffff8110c0b0>] sync_page+0x40/0x50
Apr 28 14:46:15 ubuntu kernel: [ 257.112370] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
Apr 28 14:46:15 ubuntu kernel: [ 257.112373] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
Apr 28 14:46:15 ubuntu kernel: [ 257.112376] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 257.112380] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
Apr 28 14:46:15 ubuntu kernel: [ 257.112383] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
Apr 28 14:46:15 ubuntu kernel: [ 257.112386] [<ffffffff8112d144>] __do_fault+0x54/0x520
Apr 28 14:46:15 ubuntu kernel: [ 257.112389] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
Apr 28 14:46:15 ubuntu kernel: [ 257.112392] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
Apr 28 14:46:15 ubuntu kernel: [ 257.112395] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
Apr 28 14:46:15 ubuntu kernel: [ 257.112398] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
Apr 28 14:46:15 ubuntu kernel: [ 257.112400] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
Apr 28 14:46:15 ubuntu kernel: [ 257.112403] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
Apr 28 14:46:15 ubuntu kernel: [ 257.112407] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500
Apr 28 14:46:15 ubuntu kernel: [ 257.112409] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0
Apr 28 14:46:15 ubuntu kernel: [ 257.112412] [<ffffffff815c34d5>] page_fault+0x25/0x30
Apr 28 14:46:15 ubuntu kernel: [ 257.112416] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70
Apr 28 14:46:15 ubuntu kernel: [ 257.112419] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70
Apr 28 14:46:15 ubuntu kernel: [ 257.112422] [<ffffffff812e6dc6>] clear_user+0x36/0x40
Apr 28 14:46:15 ubuntu kernel: [ 257.112426] [<ffffffff811b0b6d>] padzero+0x2d/0x40
Apr 28 14:46:15 ubuntu kernel: [ 257.112428] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00
Apr 28 14:46:15 ubuntu kernel: [ 257.112432] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300
Apr 28 14:46:15 ubuntu kernel: [ 257.112435] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00
Apr 28 14:46:15 ubuntu kernel: [ 257.112438] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0
Apr 28 14:46:15 ubuntu kernel: [ 257.112441] [<ffffffff8101521a>] sys_execve+0x4a/0x80
Apr 28 14:46:15 ubuntu kernel: [ 257.112444] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0
Apr 28 14:46:15 ubuntu kernel: [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds.
Apr 28 14:46:15 ubuntu kernel: [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 14:46:15 ubuntu kernel: [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000
Apr 28 14:46:15 ubuntu kernel: [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000
Apr 28 14:46:15 ubuntu kernel: [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00
Apr 28 14:46:15 ubuntu kernel: [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00
Apr 28 14:46:15 ubuntu kernel: [ 287.088400] Call Trace:
Apr 28 14:46:15 ubuntu kernel: [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0
Apr 28 14:46:15 ubuntu kernel: [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50
Apr 28 14:46:15 ubuntu kernel: [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
Apr 28 14:46:15 ubuntu kernel: [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
Apr 28 14:46:15 ubuntu kernel: [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
Apr 28 14:46:15 ubuntu kernel: [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
Apr 28 14:46:15 ubuntu kernel: [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520
Apr 28 14:46:15 ubuntu kernel: [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170
Apr 28 14:46:15 ubuntu kernel: [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0
Apr 28 14:46:15 ubuntu kernel: [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
Apr 28 14:46:15 ubuntu kernel: [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
Apr 28 14:46:15 ubuntu kernel: [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
Apr 28 14:46:15 ubuntu kernel: [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
Apr 28 14:46:15 ubuntu kernel: [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
Apr 28 14:46:15 ubuntu kernel: [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
Apr 28 14:46:15 ubuntu kernel: [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
Apr 28 14:46:15 ubuntu kernel: [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230
Apr 28 14:46:15 ubuntu kernel: [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30
Apr 28 14:46:15 ubuntu kernel: [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds.
Apr 28 14:46:15 ubuntu kernel: [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 14:46:15 ubuntu kernel: [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000
Apr 28 14:46:15 ubuntu kernel: [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000
Apr 28 14:46:15 ubuntu kernel: [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00
Apr 28 14:46:15 ubuntu kernel: [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00
Apr 28 14:46:15 ubuntu kernel: [ 287.088484] Call Trace:
Apr 28 14:46:15 ubuntu kernel: [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0
Apr 28 14:46:15 ubuntu kernel: [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50
Apr 28 14:46:15 ubuntu kernel: [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
Apr 28 14:46:15 ubuntu kernel: [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
Apr 28 14:46:15 ubuntu kernel: [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
Apr 28 14:46:15 ubuntu kernel: [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
Apr 28 14:46:15 ubuntu kernel: [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60
Apr 28 14:46:15 ubuntu kernel: [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520
Apr 28 14:46:15 ubuntu kernel: [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60
Apr 28 14:46:15 ubuntu kernel: [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
Apr 28 14:46:15 ubuntu kernel: [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
Apr 28 14:46:15 ubuntu kernel: [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
Apr 28 14:46:15 ubuntu kernel: [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
Apr 28 14:46:15 ubuntu kernel: [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
Apr 28 14:46:15 ubuntu kernel: [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20
Apr 28 14:46:15 ubuntu kernel: [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0
Apr 28 14:46:15 ubuntu kernel: [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90
Apr 28 14:46:15 ubuntu kernel: [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30
Apr 28 14:46:15 ubuntu kernel: [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds.
Apr 28 14:46:15 ubuntu kernel: [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 14:46:15 ubuntu kernel: [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000
Apr 28 14:46:15 ubuntu kernel: [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000
Apr 28 14:46:15 ubuntu kernel: [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00
Apr 28 14:46:15 ubuntu kernel: [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00
Apr 28 14:46:15 ubuntu kernel: [ 287.088577] Call Trace:
Apr 28 14:46:15 ubuntu kernel: [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0
Apr 28 14:46:15 ubuntu kernel: [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50
Apr 28 14:46:15 ubuntu kernel: [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
Apr 28 14:46:15 ubuntu kernel: [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
Apr 28 14:46:15 ubuntu kernel: [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
Apr 28 14:46:15 ubuntu kernel: [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
Apr 28 14:46:15 ubuntu kernel: [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520
Apr 28 14:46:15 ubuntu kernel: [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
Apr 28 14:46:15 ubuntu kernel: [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
Apr 28 14:46:15 ubuntu kernel: [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
Apr 28 14:46:15 ubuntu kernel: [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
Apr 28 14:46:15 ubuntu kernel: [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
Apr 28 14:46:15 ubuntu kernel: [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
Apr 28 14:46:15 ubuntu kernel: [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30
Apr 28 14:46:15 ubuntu kernel: [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds.
Apr 28 14:46:15 ubuntu kernel: [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 14:46:15 ubuntu kernel: [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000
Apr 28 14:46:15 ubuntu kernel: [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000
Apr 28 14:46:15 ubuntu kernel: [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00
Apr 28 14:46:15 ubuntu kernel: [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00
Apr 28 14:46:15 ubuntu kernel: [ 287.088647] Call Trace:
Apr 28 14:46:15 ubuntu kernel: [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0
Apr 28 14:46:15 ubuntu kernel: [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50
Apr 28 14:46:15 ubuntu kernel: [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
Apr 28 14:46:15 ubuntu kernel: [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
Apr 28 14:46:15 ubuntu kernel: [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190
Apr 28 14:46:15 ubuntu kernel: [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30
Apr 28 14:46:15 ubuntu kernel: [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170
Apr 28 14:46:15 ubuntu kernel: [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190
Apr 28 14:46:15 ubuntu kernel: [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0
Apr 28 14:46:15 ubuntu kernel: [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220
Apr 28 14:46:15 ubuntu kernel: [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
Apr 28 14:46:15 ubuntu kernel: [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220
Apr 28 14:46:15 ubuntu kernel: [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0
Apr 28 14:46:15 ubuntu kernel: [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
Apr 28 14:46:15 ubuntu kernel: [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0
Apr 28 14:46:15 ubuntu kernel: [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
Apr 28 14:46:15 ubuntu kernel: [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds.
Apr 28 14:46:15 ubuntu kernel: [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 14:46:15 ubuntu kernel: [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000
Apr 28 14:46:15 ubuntu kernel: [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000
Apr 28 14:46:15 ubuntu kernel: [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00
Apr 28 14:46:15 ubuntu kernel: [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00
Apr 28 14:46:15 ubuntu kernel: [ 287.088723] Call Trace:
Apr 28 14:46:15 ubuntu kernel: [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0
Apr 28 14:46:15 ubuntu kernel: [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50
Apr 28 14:46:15 ubuntu kernel: [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
Apr 28 14:46:15 ubuntu kernel: [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
Apr 28 14:46:15 ubuntu kernel: [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
Apr 28 14:46:15 ubuntu kernel: [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
Apr 28 14:46:15 ubuntu kernel: [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520
Apr 28 14:46:15 ubuntu kernel: [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
Apr 28 14:46:15 ubuntu kernel: [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
Apr 28 14:46:15 ubuntu kernel: [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
Apr 28 14:46:15 ubuntu kernel: [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
Apr 28 14:46:15 ubuntu kernel: [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
Apr 28 14:46:15 ubuntu kernel: [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
Apr 28 14:46:15 ubuntu kernel: [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500
Apr 28 14:46:15 ubuntu kernel: [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0
Apr 28 14:46:15 ubuntu kernel: [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30
Apr 28 14:46:15 ubuntu kernel: [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70
Apr 28 14:46:15 ubuntu kernel: [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70
Apr 28 14:46:15 ubuntu kernel: [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40
Apr 28 14:46:15 ubuntu kernel: [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40
Apr 28 14:46:15 ubuntu kernel: [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00
Apr 28 14:46:15 ubuntu kernel: [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300
Apr 28 14:46:15 ubuntu kernel: [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00
Apr 28 14:46:15 ubuntu kernel: [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0
Apr 28 14:46:15 ubuntu kernel: [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80
Apr 28 14:46:15 ubuntu kernel: [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0
Apr 28 14:46:15 ubuntu kernel: [ 317.064582] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds.
Apr 28 14:46:15 ubuntu kernel: [ 317.064585] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 14:46:15 ubuntu kernel: [ 317.064587] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000
Apr 28 14:46:15 ubuntu kernel: [ 317.064591] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000
Apr 28 14:46:15 ubuntu kernel: [ 317.064596] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00
Apr 28 14:46:15 ubuntu kernel: [ 317.064607] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00
Apr 28 14:46:15 ubuntu kernel: [ 317.064611] Call Trace:
Apr 28 14:46:15 ubuntu kernel: [ 317.064616] [<ffffffff8110c070>] ? sync_page+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 317.064620] [<ffffffff815c0990>] io_schedule+0x70/0xc0
Apr 28 14:46:15 ubuntu kernel: [ 317.064623] [<ffffffff8110c0b0>] sync_page+0x40/0x50
Apr 28 14:46:15 ubuntu kernel: [ 317.064627] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
Apr 28 14:46:15 ubuntu kernel: [ 317.064630] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
Apr 28 14:46:15 ubuntu kernel: [ 317.064633] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 317.064637] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
Apr 28 14:46:15 ubuntu kernel: [ 317.064640] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
Apr 28 14:46:15 ubuntu kernel: [ 317.064643] [<ffffffff8112d144>] __do_fault+0x54/0x520
Apr 28 14:46:15 ubuntu kernel: [ 317.064646] [<ffffffff81134a43>] ? unmap_region+0x113/0x170
Apr 28 14:46:15 ubuntu kernel: [ 317.064649] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0
Apr 28 14:46:15 ubuntu kernel: [ 317.064652] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
Apr 28 14:46:15 ubuntu kernel: [ 317.064655] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
Apr 28 14:46:15 ubuntu kernel: [ 317.064658] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
Apr 28 14:46:15 ubuntu kernel: [ 317.064661] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
Apr 28 14:46:15 ubuntu kernel: [ 317.064664] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
Apr 28 14:46:15 ubuntu kernel: [ 317.064667] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
Apr 28 14:46:15 ubuntu kernel: [ 317.064670] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
Apr 28 14:46:15 ubuntu kernel: [ 317.064673] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230
Apr 28 14:46:15 ubuntu kernel: [ 317.064676] [<ffffffff815c34d5>] page_fault+0x25/0x30
Apr 28 14:46:15 ubuntu kernel: [ 317.064679] INFO: task NetworkManager:764 blocked for more than 30 seconds.
Apr 28 14:46:15 ubuntu kernel: [ 317.064681] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 14:46:15 ubuntu kernel: [ 317.064683] NetworkManager D 0000000000000002 0 764 1 0x00000000
Apr 28 14:46:15 ubuntu kernel: [ 317.064686] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000
Apr 28 14:46:15 ubuntu kernel: [ 317.064690] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00
Apr 28 14:46:15 ubuntu kernel: [ 317.064693] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00
Apr 28 14:46:15 ubuntu kernel: [ 317.064697] Call Trace:
Apr 28 14:46:15 ubuntu kernel: [ 317.064701] [<ffffffff8110c070>] ? sync_page+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 317.064704] [<ffffffff815c0990>] io_schedule+0x70/0xc0
Apr 28 14:46:15 ubuntu kernel: [ 317.064707] [<ffffffff8110c0b0>] sync_page+0x40/0x50
Apr 28 14:46:15 ubuntu kernel: [ 317.064710] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
Apr 28 14:46:15 ubuntu kernel: [ 317.064713] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
Apr 28 14:46:15 ubuntu kernel: [ 317.064716] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
Apr 28 14:46:15 ubuntu kernel: [ 317.064719] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
Apr 28 14:46:15 ubuntu kernel: [ 317.064722] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
Apr 28 14:46:15 ubuntu kernel: [ 317.064726] [<ffffffff81177110>] ? pollwake+0x0/0x60
Apr 28 14:46:15 ubuntu kernel: [ 317.064728] [<ffffffff8112d144>] __do_fault+0x54/0x520
Apr 28 14:46:15 ubuntu kernel: [ 317.064731] [<ffffffff81177110>] ? pollwake+0x0/0x60
Apr 28 14:46:15 ubuntu kernel: [ 317.064734] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
Apr 28 14:46:15 ubuntu kernel: [ 317.064737] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
Apr 28 14:46:15 ubuntu kernel: [ 317.064739] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
Apr 28 14:46:15 ubuntu kernel: [ 317.064742] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
Apr 28 14:46:15 ubuntu kernel: [ 317.064745] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
Apr 28 14:46:15 ubuntu kernel: [ 317.064749] [<ffffffff81013859>] ? read_tsc+0x9/0x20
Apr 28 14:46:15 ubuntu kernel: [ 317.064752] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0
Apr 28 14:46:15 ubuntu kernel: [ 317.064755] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90
Apr 28 14:46:15 ubuntu kernel: [ 317.064758] [<ffffffff815c34d5>] page_fault+0x25/0x30
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 14:01 ` Colin Ian King
@ 2011-04-28 14:04 ` Chris Mason
2011-04-28 14:25 ` Jan Kara
1 sibling, 0 replies; 138+ messages in thread
From: Chris Mason @ 2011-04-28 14:04 UTC (permalink / raw)
To: Colin Ian King
Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4
Excerpts from Colin Ian King's message of 2011-04-28 10:01:22 -0400:
>
> > Could you post the soft lockups you're seeing?
>
> As requested, attached
These are not good, but they aren't the lockup James was seeing. Were
these messages with my patch? If yes, please post the messages from
without my patch.
-chris
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 14:04 ` Chris Mason
0 siblings, 0 replies; 138+ messages in thread
From: Chris Mason @ 2011-04-28 14:04 UTC (permalink / raw)
To: Colin Ian King
Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4
Excerpts from Colin Ian King's message of 2011-04-28 10:01:22 -0400:
>
> > Could you post the soft lockups you're seeing?
>
> As requested, attached
These are not good, but they aren't the lockup James was seeing. Were
these messages with my patch? If yes, please post the messages from
without my patch.
-chris
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 13:52 ` Jan Kara
@ 2011-04-28 14:07 ` Mel Gorman
-1 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-04-28 14:07 UTC (permalink / raw)
To: Jan Kara
Cc: colin.king, James Bottomley, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4, mgorman
On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote:
> On Thu 28-04-11 12:36:30, Colin Ian King wrote:
> > One more data point to add, I've been looking at an identical issue when
> > copying large amounts of data. I bisected this - and the lockups occur
> > with commit
> > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > issue. With this commit, my file copy test locks up after ~8-10
> > iterations, before this commit I can copy > 100 times and don't see the
> > lockup.
> Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit
> of yours causes kswapd on non-preempt kernels spin for a *long* time...
>
I'm still thinking about the traces which do not point the finger
directly at compaction per-se but it's possible that the change means
kswapd is not reclaiming like it should be.
To test this theory, does applying
[d527caf2: mm: compaction: prevent kswapd compacting memory to reduce
CPU usage] help?
--
Mel Gorman
SUSE Labs
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 14:07 ` Mel Gorman
0 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-04-28 14:07 UTC (permalink / raw)
To: Jan Kara
Cc: colin.king, James Bottomley, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4, mgorman
On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote:
> On Thu 28-04-11 12:36:30, Colin Ian King wrote:
> > One more data point to add, I've been looking at an identical issue when
> > copying large amounts of data. I bisected this - and the lockups occur
> > with commit
> > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > issue. With this commit, my file copy test locks up after ~8-10
> > iterations, before this commit I can copy > 100 times and don't see the
> > lockup.
> Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit
> of yours causes kswapd on non-preempt kernels spin for a *long* time...
>
I'm still thinking about the traces which do not point the finger
directly at compaction per-se but it's possible that the change means
kswapd is not reclaiming like it should be.
To test this theory, does applying
[d527caf2: mm: compaction: prevent kswapd compacting memory to reduce
CPU usage] help?
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 14:07 ` Mel Gorman
@ 2011-04-28 14:25 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 14:25 UTC (permalink / raw)
To: Mel Gorman
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 15:07 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote:
> > On Thu 28-04-11 12:36:30, Colin Ian King wrote:
> > > One more data point to add, I've been looking at an identical issue when
> > > copying large amounts of data. I bisected this - and the lockups occur
> > > with commit
> > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > > issue. With this commit, my file copy test locks up after ~8-10
> > > iterations, before this commit I can copy > 100 times and don't see the
> > > lockup.
> > Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit
> > of yours causes kswapd on non-preempt kernels spin for a *long* time...
> >
>
> I'm still thinking about the traces which do not point the finger
> directly at compaction per-se but it's possible that the change means
> kswapd is not reclaiming like it should be.
>
> To test this theory, does applying
> [d527caf2: mm: compaction: prevent kswapd compacting memory to reduce
> CPU usage] help?
I can answer definitively no to this. The upstream kernel I reproduced
this on has that patch included.
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 14:25 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 14:25 UTC (permalink / raw)
To: Mel Gorman
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 15:07 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote:
> > On Thu 28-04-11 12:36:30, Colin Ian King wrote:
> > > One more data point to add, I've been looking at an identical issue when
> > > copying large amounts of data. I bisected this - and the lockups occur
> > > with commit
> > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > > issue. With this commit, my file copy test locks up after ~8-10
> > > iterations, before this commit I can copy > 100 times and don't see the
> > > lockup.
> > Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit
> > of yours causes kswapd on non-preempt kernels spin for a *long* time...
> >
>
> I'm still thinking about the traces which do not point the finger
> directly at compaction per-se but it's possible that the change means
> kswapd is not reclaiming like it should be.
>
> To test this theory, does applying
> [d527caf2: mm: compaction: prevent kswapd compacting memory to reduce
> CPU usage] help?
I can answer definitively no to this. The upstream kernel I reproduced
this on has that patch included.
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 14:01 ` Colin Ian King
@ 2011-04-28 14:25 ` Jan Kara
2011-04-28 14:25 ` Jan Kara
1 sibling, 0 replies; 138+ messages in thread
From: Jan Kara @ 2011-04-28 14:25 UTC (permalink / raw)
To: Colin Ian King
Cc: Chris Mason, James Bottomley, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4
On Thu 28-04-11 15:01:22, Colin Ian King wrote:
>
> > Could you post the soft lockups you're seeing?
>
> As requested, attached
Hum, what keeps puzzling me is that in all the cases of hangs I've seen
so far, we are stuck waiting for IO to finish for a long time - e.g. in the
traces below kjournald waits for PageWriteback bit to get cleared. Also we
are stuck waiting for page locks which might be because those pages are
being read in? All in all it seems that the IO is just incredibly slow.
But it's not clear to me what pushes us into that situation (especially
since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the
underlying blocks are not already allocated.
Honza
[ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds.
[ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000
[ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000
[ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00
[ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00
[ 287.088400] Call Trace:
[ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50
[ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0
[ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50
[ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
[ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
[ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
[ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
[ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
[ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520
[ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170
[ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0
[ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
[ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
[ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
[ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
[ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
[ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
[ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
[ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230
[ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30
[ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds.
[ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000
[ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000
[ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00
[ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00
[ 287.088484] Call Trace:
[ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50
[ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0
[ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50
[ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
[ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
[ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
[ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
[ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
[ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60
[ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520
[ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60
[ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
[ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
[ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
[ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
[ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
[ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20
[ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0
[ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90
[ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30
[ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds.
[ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000
[ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000
[ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00
[ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00
[ 287.088577] Call Trace:
[ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50
[ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0
[ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50
[ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
[ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
[ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
[ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
[ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
[ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520
[ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
[ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
[ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
[ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
[ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
[ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
[ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30
[ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds.
[ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000
[ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000
[ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00
[ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00
[ 287.088647] Call Trace:
[ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50
[ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0
[ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50
[ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
[ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
[ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
[ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190
[ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30
[ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170
[ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190
[ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0
[ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220
[ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
[ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220
[ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0
[ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
[ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0
[ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
[ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds.
[ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000
[ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000
[ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00
[ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00
[ 287.088723] Call Trace:
[ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50
[ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0
[ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50
[ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
[ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
[ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
[ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
[ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
[ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520
[ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
[ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
[ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
[ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
[ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
[ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
[ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500
[ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0
[ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30
[ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70
[ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70
[ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40
[ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40
[ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00
[ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300
[ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00
[ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0
[ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80
[ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 14:25 ` Jan Kara
0 siblings, 0 replies; 138+ messages in thread
From: Jan Kara @ 2011-04-28 14:25 UTC (permalink / raw)
To: Colin Ian King
Cc: Chris Mason, James Bottomley, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4
On Thu 28-04-11 15:01:22, Colin Ian King wrote:
>
> > Could you post the soft lockups you're seeing?
>
> As requested, attached
Hum, what keeps puzzling me is that in all the cases of hangs I've seen
so far, we are stuck waiting for IO to finish for a long time - e.g. in the
traces below kjournald waits for PageWriteback bit to get cleared. Also we
are stuck waiting for page locks which might be because those pages are
being read in? All in all it seems that the IO is just incredibly slow.
But it's not clear to me what pushes us into that situation (especially
since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the
underlying blocks are not already allocated.
Honza
[ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds.
[ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000
[ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000
[ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00
[ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00
[ 287.088400] Call Trace:
[ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50
[ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0
[ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50
[ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
[ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
[ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
[ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
[ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
[ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520
[ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170
[ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0
[ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
[ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
[ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
[ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
[ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
[ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
[ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
[ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230
[ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30
[ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds.
[ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000
[ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000
[ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00
[ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00
[ 287.088484] Call Trace:
[ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50
[ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0
[ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50
[ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
[ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
[ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
[ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
[ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
[ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60
[ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520
[ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60
[ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
[ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
[ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
[ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
[ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
[ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20
[ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0
[ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90
[ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30
[ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds.
[ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000
[ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000
[ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00
[ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00
[ 287.088577] Call Trace:
[ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50
[ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0
[ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50
[ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
[ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
[ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
[ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
[ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
[ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520
[ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
[ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
[ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
[ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
[ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
[ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
[ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30
[ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds.
[ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000
[ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000
[ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00
[ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00
[ 287.088647] Call Trace:
[ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50
[ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0
[ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50
[ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
[ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
[ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
[ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190
[ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30
[ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170
[ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190
[ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0
[ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220
[ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
[ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220
[ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0
[ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
[ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0
[ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
[ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds.
[ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000
[ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000
[ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00
[ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00
[ 287.088723] Call Trace:
[ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50
[ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0
[ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50
[ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
[ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
[ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
[ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
[ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
[ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520
[ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
[ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
[ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
[ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
[ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
[ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
[ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500
[ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0
[ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30
[ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70
[ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70
[ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40
[ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40
[ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00
[ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300
[ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00
[ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0
[ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80
[ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 14:25 ` Jan Kara
@ 2011-04-28 14:33 ` Jan Kara
-1 siblings, 0 replies; 138+ messages in thread
From: Jan Kara @ 2011-04-28 14:33 UTC (permalink / raw)
To: Colin Ian King
Cc: Chris Mason, James Bottomley, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4
On Thu 28-04-11 16:25:51, Jan Kara wrote:
> On Thu 28-04-11 15:01:22, Colin Ian King wrote:
> >
> > > Could you post the soft lockups you're seeing?
> >
> > As requested, attached
> Hum, what keeps puzzling me is that in all the cases of hangs I've seen
> so far, we are stuck waiting for IO to finish for a long time - e.g. in the
> traces below kjournald waits for PageWriteback bit to get cleared. Also we
> are stuck waiting for page locks which might be because those pages are
> being read in? All in all it seems that the IO is just incredibly slow.
>
> But it's not clear to me what pushes us into that situation (especially
> since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the
> underlying blocks are not already allocated.
Hmm, maybe because the system is under memory pressure (and kswapd is not
able to get rid of dirty pages), we page out clean pages. Thus also pages
of executables which need to be paged in soon anyway thus putting heavy
read load on the system which makes writes crawl? I'm not sure why
compaction should make this any worse but maybe it can.
James, Colin, can you capture output of 'vmstat 1' while you do the
copying? Thanks.
Honza
> [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds.
> [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000
> [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000
> [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00
> [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00
> [ 287.088400] Call Trace:
> [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520
> [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170
> [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0
> [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230
> [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30
> [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds.
> [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000
> [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000
> [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00
> [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00
> [ 287.088484] Call Trace:
> [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60
> [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520
> [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60
> [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20
> [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0
> [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90
> [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30
> [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds.
> [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000
> [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000
> [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00
> [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00
> [ 287.088577] Call Trace:
> [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520
> [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30
> [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds.
> [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000
> [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000
> [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00
> [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00
> [ 287.088647] Call Trace:
> [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190
> [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30
> [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170
> [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190
> [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0
> [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220
> [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
> [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220
> [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0
> [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
> [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0
> [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
> [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds.
> [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000
> [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000
> [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00
> [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00
> [ 287.088723] Call Trace:
> [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520
> [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500
> [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0
> [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30
> [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70
> [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70
> [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40
> [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40
> [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00
> [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300
> [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00
> [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0
> [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80
> [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0
> --
> Jan Kara <jack@suse.cz>
> SUSE Labs, CR
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 14:33 ` Jan Kara
0 siblings, 0 replies; 138+ messages in thread
From: Jan Kara @ 2011-04-28 14:33 UTC (permalink / raw)
To: Colin Ian King
Cc: Chris Mason, James Bottomley, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4
On Thu 28-04-11 16:25:51, Jan Kara wrote:
> On Thu 28-04-11 15:01:22, Colin Ian King wrote:
> >
> > > Could you post the soft lockups you're seeing?
> >
> > As requested, attached
> Hum, what keeps puzzling me is that in all the cases of hangs I've seen
> so far, we are stuck waiting for IO to finish for a long time - e.g. in the
> traces below kjournald waits for PageWriteback bit to get cleared. Also we
> are stuck waiting for page locks which might be because those pages are
> being read in? All in all it seems that the IO is just incredibly slow.
>
> But it's not clear to me what pushes us into that situation (especially
> since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the
> underlying blocks are not already allocated.
Hmm, maybe because the system is under memory pressure (and kswapd is not
able to get rid of dirty pages), we page out clean pages. Thus also pages
of executables which need to be paged in soon anyway thus putting heavy
read load on the system which makes writes crawl? I'm not sure why
compaction should make this any worse but maybe it can.
James, Colin, can you capture output of 'vmstat 1' while you do the
copying? Thanks.
Honza
> [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds.
> [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000
> [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000
> [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00
> [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00
> [ 287.088400] Call Trace:
> [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520
> [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170
> [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0
> [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230
> [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30
> [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds.
> [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000
> [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000
> [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00
> [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00
> [ 287.088484] Call Trace:
> [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60
> [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520
> [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60
> [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20
> [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0
> [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90
> [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30
> [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds.
> [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000
> [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000
> [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00
> [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00
> [ 287.088577] Call Trace:
> [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520
> [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30
> [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds.
> [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000
> [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000
> [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00
> [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00
> [ 287.088647] Call Trace:
> [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190
> [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30
> [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170
> [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190
> [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0
> [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220
> [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
> [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220
> [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0
> [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
> [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0
> [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
> [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds.
> [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000
> [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000
> [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00
> [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00
> [ 287.088723] Call Trace:
> [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520
> [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500
> [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0
> [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30
> [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70
> [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70
> [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40
> [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40
> [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00
> [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300
> [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00
> [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0
> [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80
> [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0
> --
> Jan Kara <jack@suse.cz>
> SUSE Labs, CR
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 12:29 ` Chris Mason
@ 2011-04-28 14:49 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 14:49 UTC (permalink / raw)
To: Chris Mason
Cc: Colin Ian King, linux-fsdevel, linux-mm, linux-kernel, linux-ext4
On Thu, 2011-04-28 at 08:29 -0400, Chris Mason wrote:
> Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400:
> > One more data point to add, I've been looking at an identical issue when
> > copying large amounts of data. I bisected this - and the lockups occur
> > with commit
> > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > issue. With this commit, my file copy test locks up after ~8-10
> > iterations, before this commit I can copy > 100 times and don't see the
> > lockup.
>
> Well, that's really interesting. I tried with compaction on here and
> couldn't trigger it, but this (very very lightly) tested patch might
> help.
>
> It moves the writeout throttle before the goto restart, and also makes
> sure we do at least one cond_resched before we loop.
It seems to take longer, but with a PREEMPT kernel, kswapd eventually
shoots up to 99% during the tar.
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 14:49 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 14:49 UTC (permalink / raw)
To: Chris Mason
Cc: Colin Ian King, linux-fsdevel, linux-mm, linux-kernel, linux-ext4
On Thu, 2011-04-28 at 08:29 -0400, Chris Mason wrote:
> Excerpts from Colin Ian King's message of 2011-04-28 07:36:30 -0400:
> > One more data point to add, I've been looking at an identical issue when
> > copying large amounts of data. I bisected this - and the lockups occur
> > with commit
> > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > issue. With this commit, my file copy test locks up after ~8-10
> > iterations, before this commit I can copy > 100 times and don't see the
> > lockup.
>
> Well, that's really interesting. I tried with compaction on here and
> couldn't trigger it, but this (very very lightly) tested patch might
> help.
>
> It moves the writeout throttle before the goto restart, and also makes
> sure we do at least one cond_resched before we loop.
It seems to take longer, but with a PREEMPT kernel, kswapd eventually
shoots up to 99% during the tar.
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 14:33 ` Jan Kara
(?)
@ 2011-04-28 14:58 ` Colin Ian King
2011-04-28 22:40 ` Jan Kara
-1 siblings, 1 reply; 138+ messages in thread
From: Colin Ian King @ 2011-04-28 14:58 UTC (permalink / raw)
To: Jan Kara
Cc: Chris Mason, James Bottomley, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4
[-- Attachment #1: Type: text/plain, Size: 11166 bytes --]
On Thu, 2011-04-28 at 16:33 +0200, Jan Kara wrote:
> On Thu 28-04-11 16:25:51, Jan Kara wrote:
> > On Thu 28-04-11 15:01:22, Colin Ian King wrote:
> > >
> > > > Could you post the soft lockups you're seeing?
> > >
> > > As requested, attached
> > Hum, what keeps puzzling me is that in all the cases of hangs I've seen
> > so far, we are stuck waiting for IO to finish for a long time - e.g. in the
> > traces below kjournald waits for PageWriteback bit to get cleared. Also we
> > are stuck waiting for page locks which might be because those pages are
> > being read in? All in all it seems that the IO is just incredibly slow.
> >
> > But it's not clear to me what pushes us into that situation (especially
> > since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the
> > underlying blocks are not already allocated.
> Hmm, maybe because the system is under memory pressure (and kswapd is not
> able to get rid of dirty pages), we page out clean pages. Thus also pages
> of executables which need to be paged in soon anyway thus putting heavy
> read load on the system which makes writes crawl? I'm not sure why
> compaction should make this any worse but maybe it can.
>
> James, Colin, can you capture output of 'vmstat 1' while you do the
> copying? Thanks.
Attached.
>
> Honza
>
> > [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds.
> > [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000
> > [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000
> > [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00
> > [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00
> > [ 287.088400] Call Trace:
> > [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170
> > [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0
> > [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> > [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> > [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> > [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> > [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230
> > [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds.
> > [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000
> > [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000
> > [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00
> > [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00
> > [ 287.088484] Call Trace:
> > [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60
> > [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60
> > [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> > [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> > [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20
> > [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0
> > [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90
> > [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds.
> > [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000
> > [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000
> > [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00
> > [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00
> > [ 287.088577] Call Trace:
> > [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> > [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> > [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> > [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds.
> > [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000
> > [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000
> > [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00
> > [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00
> > [ 287.088647] Call Trace:
> > [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190
> > [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30
> > [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170
> > [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190
> > [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0
> > [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220
> > [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
> > [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220
> > [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0
> > [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
> > [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0
> > [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
> > [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds.
> > [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000
> > [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000
> > [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00
> > [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00
> > [ 287.088723] Call Trace:
> > [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> > [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> > [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> > [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500
> > [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0
> > [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70
> > [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70
> > [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40
> > [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40
> > [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00
> > [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300
> > [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00
> > [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0
> > [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80
> > [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0
> > --
> > Jan Kara <jack@suse.cz>
> > SUSE Labs, CR
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: vmstat.log.gz --]
[-- Type: application/x-gzip, Size: 14197 bytes --]
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 14:25 ` James Bottomley
@ 2011-04-28 15:08 ` Mel Gorman
-1 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-04-28 15:08 UTC (permalink / raw)
To: James Bottomley
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, Apr 28, 2011 at 09:25:14AM -0500, James Bottomley wrote:
> On Thu, 2011-04-28 at 15:07 +0100, Mel Gorman wrote:
> > On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote:
> > > On Thu 28-04-11 12:36:30, Colin Ian King wrote:
> > > > One more data point to add, I've been looking at an identical issue when
> > > > copying large amounts of data. I bisected this - and the lockups occur
> > > > with commit
> > > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > > > issue. With this commit, my file copy test locks up after ~8-10
> > > > iterations, before this commit I can copy > 100 times and don't see the
> > > > lockup.
> > > Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit
> > > of yours causes kswapd on non-preempt kernels spin for a *long* time...
> > >
> >
> > I'm still thinking about the traces which do not point the finger
> > directly at compaction per-se but it's possible that the change means
> > kswapd is not reclaiming like it should be.
> >
> > To test this theory, does applying
> > [d527caf2: mm: compaction: prevent kswapd compacting memory to reduce
> > CPU usage] help?
>
> I can answer definitively no to this. The upstream kernel I reproduced
> this on has that patch included.
>
So it is.
Another consequence of this patch is that when high order allocations
are in progress (is the test case fork heavy in any way for
example? alternatively, it might be something in the storage stack
that requires high-order allocs) we are no longer necessarily going
to sleep because of should_reclaim_continue() check. This could
explain kswapd-at-99% but would only apply if CONFIG_COMPACTION is
set (does unsetting CONFIG_COMPACTION help). If the bug only triggers
for CONFIG_COMPACTION, does the following *untested* patch help any?
(as a warning, I'm offline Friday until Tuesday morning. I'll try
check mail over the weekend but it's unlikely I'll find a terminal
or be allowed to use it without an ass kicking)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 148c6e6..c74a501 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1842,15 +1842,22 @@ static inline bool should_continue_reclaim(struct zone *zone,
return false;
/*
- * If we failed to reclaim and have scanned the full list, stop.
- * NOTE: Checking just nr_reclaimed would exit reclaim/compaction far
- * faster but obviously would be less likely to succeed
- * allocation. If this is desirable, use GFP_REPEAT to decide
- * if both reclaimed and scanned should be checked or just
- * reclaimed
+ * For direct reclaimers
+ * If we failed to reclaim and have scanned the full list, stop.
+ * The caller will check congestion and sleep if necessary until
+ * some IO completes.
+ * For kswapd
+ * Check just nr_reclaimed. If we are failing to reclaim, we
+ * want to stop this reclaim loop, increase the priority and
+ * go to sleep if necessary to allow IO a change to complete.
+ * This avoids kswapd going into a busy loop in shrink_zone()
*/
- if (!nr_reclaimed && !nr_scanned)
- return false;
+ if (!nr_reclaimed) {
+ if (current_is_kswapd())
+ return false;
+ else if (!nr_scanned)
+ return false;
+ }
/*
* If we have not reclaimed enough pages for compaction and the
@@ -1924,8 +1931,13 @@ restart:
/* reclaim/compaction might need reclaim to continue */
if (should_continue_reclaim(zone, nr_reclaimed,
- sc->nr_scanned - nr_scanned, sc))
+ sc->nr_scanned - nr_scanned, sc)) {
+ /* Throttle direct reclaimers if congested */
+ if (!current_is_kswapd())
+ wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
+
goto restart;
+ }
throttle_vm_writeout(sc->gfp_mask);
}
^ permalink raw reply related [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 15:08 ` Mel Gorman
0 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-04-28 15:08 UTC (permalink / raw)
To: James Bottomley
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, Apr 28, 2011 at 09:25:14AM -0500, James Bottomley wrote:
> On Thu, 2011-04-28 at 15:07 +0100, Mel Gorman wrote:
> > On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote:
> > > On Thu 28-04-11 12:36:30, Colin Ian King wrote:
> > > > One more data point to add, I've been looking at an identical issue when
> > > > copying large amounts of data. I bisected this - and the lockups occur
> > > > with commit
> > > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > > > issue. With this commit, my file copy test locks up after ~8-10
> > > > iterations, before this commit I can copy > 100 times and don't see the
> > > > lockup.
> > > Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit
> > > of yours causes kswapd on non-preempt kernels spin for a *long* time...
> > >
> >
> > I'm still thinking about the traces which do not point the finger
> > directly at compaction per-se but it's possible that the change means
> > kswapd is not reclaiming like it should be.
> >
> > To test this theory, does applying
> > [d527caf2: mm: compaction: prevent kswapd compacting memory to reduce
> > CPU usage] help?
>
> I can answer definitively no to this. The upstream kernel I reproduced
> this on has that patch included.
>
So it is.
Another consequence of this patch is that when high order allocations
are in progress (is the test case fork heavy in any way for
example? alternatively, it might be something in the storage stack
that requires high-order allocs) we are no longer necessarily going
to sleep because of should_reclaim_continue() check. This could
explain kswapd-at-99% but would only apply if CONFIG_COMPACTION is
set (does unsetting CONFIG_COMPACTION help). If the bug only triggers
for CONFIG_COMPACTION, does the following *untested* patch help any?
(as a warning, I'm offline Friday until Tuesday morning. I'll try
check mail over the weekend but it's unlikely I'll find a terminal
or be allowed to use it without an ass kicking)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 148c6e6..c74a501 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1842,15 +1842,22 @@ static inline bool should_continue_reclaim(struct zone *zone,
return false;
/*
- * If we failed to reclaim and have scanned the full list, stop.
- * NOTE: Checking just nr_reclaimed would exit reclaim/compaction far
- * faster but obviously would be less likely to succeed
- * allocation. If this is desirable, use GFP_REPEAT to decide
- * if both reclaimed and scanned should be checked or just
- * reclaimed
+ * For direct reclaimers
+ * If we failed to reclaim and have scanned the full list, stop.
+ * The caller will check congestion and sleep if necessary until
+ * some IO completes.
+ * For kswapd
+ * Check just nr_reclaimed. If we are failing to reclaim, we
+ * want to stop this reclaim loop, increase the priority and
+ * go to sleep if necessary to allow IO a change to complete.
+ * This avoids kswapd going into a busy loop in shrink_zone()
*/
- if (!nr_reclaimed && !nr_scanned)
- return false;
+ if (!nr_reclaimed) {
+ if (current_is_kswapd())
+ return false;
+ else if (!nr_scanned)
+ return false;
+ }
/*
* If we have not reclaimed enough pages for compaction and the
@@ -1924,8 +1931,13 @@ restart:
/* reclaim/compaction might need reclaim to continue */
if (should_continue_reclaim(zone, nr_reclaimed,
- sc->nr_scanned - nr_scanned, sc))
+ sc->nr_scanned - nr_scanned, sc)) {
+ /* Throttle direct reclaimers if congested */
+ if (!current_is_kswapd())
+ wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
+
goto restart;
+ }
throttle_vm_writeout(sc->gfp_mask);
}
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 14:04 ` Chris Mason
(?)
@ 2011-04-28 15:23 ` Colin Ian King
-1 siblings, 0 replies; 138+ messages in thread
From: Colin Ian King @ 2011-04-28 15:23 UTC (permalink / raw)
To: Chris Mason
Cc: James Bottomley, linux-fsdevel, linux-mm, linux-kernel, linux-ext4
[-- Attachment #1: Type: text/plain, Size: 449 bytes --]
On Thu, 2011-04-28 at 10:04 -0400, Chris Mason wrote:
> Excerpts from Colin Ian King's message of 2011-04-28 10:01:22 -0400:
> >
> > > Could you post the soft lockups you're seeing?
> >
> > As requested, attached
>
> These are not good, but they aren't the lockup James was seeing. Were
> these messages with my patch? If yes, please post the messages from
> without my patch.
Attached are the messages without your patch.
Colin
>
> -chris
[-- Attachment #2: kern.log --]
[-- Type: text/x-log, Size: 24228 bytes --]
Apr 28 15:47:41 ubuntu kernel: [ 493.876427] INFO: task jbd2/sda1-8:290 blocked for more than 30 seconds.
Apr 28 15:49:04 ubuntu kernel: [ 493.876447] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 15:49:04 ubuntu kernel: [ 493.876455] jbd2/sda1-8 D 0000000000000000 0 290 2 0x00000000
Apr 28 15:49:04 ubuntu kernel: [ 493.876468] ffff8800363e5c20 0000000000000046 ffff8800363e5fd8 ffff8800363e4000
Apr 28 15:49:04 ubuntu kernel: [ 493.876481] 0000000000013d00 ffff8800720b3178 ffff8800363e5fd8 0000000000013d00
Apr 28 15:49:04 ubuntu kernel: [ 493.876507] ffffffff81a0b020 ffff8800720b2dc0 ffff88001fffb1d0 ffff88001fc13d00
Apr 28 15:49:04 ubuntu kernel: [ 493.876522] Call Trace:
Apr 28 15:49:04 ubuntu kernel: [ 493.876542] [<ffffffff81191db0>] ? sync_buffer+0x0/0x50
Apr 28 15:49:04 ubuntu kernel: [ 493.876557] [<ffffffff815c0980>] io_schedule+0x70/0xc0
Apr 28 15:49:04 ubuntu kernel: [ 493.876570] [<ffffffff81191df0>] sync_buffer+0x40/0x50
Apr 28 15:49:04 ubuntu kernel: [ 493.876583] [<ffffffff815c12ff>] __wait_on_bit+0x5f/0x90
Apr 28 15:49:04 ubuntu kernel: [ 493.876596] [<ffffffff81191db0>] ? sync_buffer+0x0/0x50
Apr 28 15:49:04 ubuntu kernel: [ 493.876609] [<ffffffff815c13ac>] out_of_line_wait_on_bit+0x7c/0x90
Apr 28 15:49:04 ubuntu kernel: [ 493.876623] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
Apr 28 15:49:04 ubuntu kernel: [ 493.876636] [<ffffffff81191dae>] __wait_on_buffer+0x2e/0x30
Apr 28 15:49:04 ubuntu kernel: [ 493.876649] [<ffffffff81243c3e>] jbd2_journal_commit_transaction+0x10ae/0x1190
Apr 28 15:49:04 ubuntu kernel: [ 493.876663] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
Apr 28 15:49:04 ubuntu kernel: [ 493.876677] [<ffffffff81247e8b>] kjournald2+0xbb/0x220
Apr 28 15:49:04 ubuntu kernel: [ 493.876688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
Apr 28 15:49:04 ubuntu kernel: [ 493.876700] [<ffffffff81247dd0>] ? kjournald2+0x0/0x220
Apr 28 15:49:04 ubuntu kernel: [ 493.876711] [<ffffffff810877e6>] kthread+0x96/0xa0
Apr 28 15:49:04 ubuntu kernel: [ 493.876725] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
Apr 28 15:49:04 ubuntu kernel: [ 493.876737] [<ffffffff81087750>] ? kthread+0x0/0xa0
Apr 28 15:49:04 ubuntu kernel: [ 493.876747] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
Apr 28 15:49:04 ubuntu kernel: [ 493.876777] INFO: task flush-8:0:1137 blocked for more than 30 seconds.
Apr 28 15:49:04 ubuntu kernel: [ 493.876783] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 15:49:04 ubuntu kernel: [ 493.876792] flush-8:0 D 0000000000000002 0 1137 2 0x00000000
Apr 28 15:49:04 ubuntu kernel: [ 493.876806] ffff88007495f420 0000000000000046 ffff88007495ffd8 ffff88007495e000
Apr 28 15:49:04 ubuntu kernel: [ 493.876820] 0000000000013d00 ffff88007552b178 ffff88007495ffd8 0000000000013d00
Apr 28 15:49:04 ubuntu kernel: [ 493.876836] ffff8801005e5b80 ffff88007552adc0 ffff88005dd9a840 ffff88001fc53d00
Apr 28 15:49:04 ubuntu kernel: [ 493.876850] Call Trace:
Apr 28 15:49:04 ubuntu kernel: [ 493.876863] [<ffffffff815c0980>] io_schedule+0x70/0xc0
Apr 28 15:49:04 ubuntu kernel: [ 493.876878] [<ffffffff812c3499>] get_request_wait+0xc9/0x1a0
Apr 28 15:49:04 ubuntu kernel: [ 493.876890] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
Apr 28 15:49:04 ubuntu kernel: [ 493.876902] [<ffffffff812bc680>] ? elv_merge+0x50/0x120
Apr 28 15:49:04 ubuntu kernel: [ 493.876915] [<ffffffff812c3c56>] __make_request+0x76/0x4c0
Apr 28 15:49:04 ubuntu kernel: [ 493.876929] [<ffffffff812c1148>] generic_make_request+0x2d8/0x5c0
Apr 28 15:49:04 ubuntu kernel: [ 493.876946] [<ffffffff8110e725>] ? mempool_alloc_slab+0x15/0x20
Apr 28 15:49:04 ubuntu kernel: [ 493.876957] [<ffffffff8110ea69>] ? mempool_alloc+0x59/0x140
Apr 28 15:49:04 ubuntu kernel: [ 493.876970] [<ffffffff812c14b9>] submit_bio+0x89/0x120
Apr 28 15:49:04 ubuntu kernel: [ 493.876982] [<ffffffff811976fb>] ? bio_alloc_bioset+0x5b/0xf0
Apr 28 15:49:04 ubuntu kernel: [ 493.876995] [<ffffffff811918eb>] submit_bh+0xeb/0x120
Apr 28 15:49:04 ubuntu kernel: [ 493.877007] [<ffffffff81193630>] __block_write_full_page+0x210/0x3a0
Apr 28 15:49:04 ubuntu kernel: [ 493.877018] [<ffffffff81192720>] ? end_buffer_async_write+0x0/0x170
Apr 28 15:49:04 ubuntu kernel: [ 493.877032] [<ffffffff81206aa0>] ? noalloc_get_block_write+0x0/0x30
Apr 28 15:49:04 ubuntu kernel: [ 493.877044] [<ffffffff81206aa0>] ? noalloc_get_block_write+0x0/0x30
Apr 28 15:49:04 ubuntu kernel: [ 493.877056] [<ffffffff811944d3>] block_write_full_page_endio+0xe3/0x120
Apr 28 15:49:04 ubuntu kernel: [ 493.877067] [<ffffffff81194525>] block_write_full_page+0x15/0x20
Apr 28 15:49:04 ubuntu kernel: [ 493.877079] [<ffffffff81204e9a>] mpage_da_submit_io+0x43a/0x4c0
Apr 28 15:49:04 ubuntu kernel: [ 493.877094] [<ffffffff8122929b>] ? __ext4_handle_dirty_metadata+0x7b/0x120
Apr 28 15:49:04 ubuntu kernel: [ 493.877110] [<ffffffff81208c8e>] mpage_da_map_and_submit+0x1ae/0x440
Apr 28 15:49:04 ubuntu kernel: [ 493.877122] [<ffffffff812dfe2d>] ? radix_tree_gang_lookup_tag_slot+0x8d/0xd0
Apr 28 15:49:04 ubuntu kernel: [ 493.877136] [<ffffffff81208f8d>] mpage_add_bh_to_extent+0x6d/0xf0
Apr 28 15:49:04 ubuntu kernel: [ 493.877149] [<ffffffff81209102>] __mpage_da_writepage+0xf2/0x190
Apr 28 15:49:04 ubuntu kernel: [ 493.877162] [<ffffffff8120935b>] write_cache_pages_da+0x1bb/0x2d0
Apr 28 15:49:04 ubuntu kernel: [ 493.877176] [<ffffffff81209786>] ext4_da_writepages+0x316/0x630
Apr 28 15:49:04 ubuntu kernel: [ 493.877191] [<ffffffff81117151>] do_writepages+0x21/0x40
Apr 28 15:49:04 ubuntu kernel: [ 493.877202] [<ffffffff8118b5df>] writeback_single_inode+0x9f/0x240
Apr 28 15:49:04 ubuntu kernel: [ 493.877215] [<ffffffff8118b9cb>] writeback_sb_inodes+0xcb/0x160
Apr 28 15:49:04 ubuntu kernel: [ 493.877229] [<ffffffff8118bc1b>] writeback_inodes_wb+0x10b/0x1c0
Apr 28 15:49:04 ubuntu kernel: [ 493.877240] [<ffffffff8118c04e>] wb_writeback+0x37e/0x490
Apr 28 15:49:04 ubuntu kernel: [ 493.877254] [<ffffffff815c2e6f>] ? _raw_spin_lock_irqsave+0x2f/0x40
Apr 28 15:49:04 ubuntu kernel: [ 493.877269] [<ffffffff81074fdb>] ? lock_timer_base.clone.20+0x3b/0x70
Apr 28 15:49:04 ubuntu kernel: [ 493.877283] [<ffffffff811166aa>] ? determine_dirtyable_memory+0x1a/0x30
Apr 28 15:49:04 ubuntu kernel: [ 493.877296] [<ffffffff8118c2ef>] wb_do_writeback+0x18f/0x230
Apr 28 15:49:04 ubuntu kernel: [ 493.877309] [<ffffffff8118c412>] bdi_writeback_thread+0x82/0x260
Apr 28 15:49:04 ubuntu kernel: [ 493.877321] [<ffffffff8118c390>] ? bdi_writeback_thread+0x0/0x260
Apr 28 15:49:04 ubuntu kernel: [ 493.877333] [<ffffffff810877e6>] kthread+0x96/0xa0
Apr 28 15:49:04 ubuntu kernel: [ 493.877345] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
Apr 28 15:49:04 ubuntu kernel: [ 493.877357] [<ffffffff81087750>] ? kthread+0x0/0xa0
Apr 28 15:49:04 ubuntu kernel: [ 493.877368] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
Apr 28 15:49:04 ubuntu kernel: [ 493.877386] INFO: task unity-panel-ser:1548 blocked for more than 30 seconds.
Apr 28 15:49:04 ubuntu kernel: [ 493.877393] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 15:49:04 ubuntu kernel: [ 493.877400] unity-panel-ser D 0000000000000002 0 1548 1 0x00000000
Apr 28 15:49:04 ubuntu kernel: [ 493.877414] ffff88006837fb68 0000000000000086 ffff88006837ffd8 ffff88006837e000
Apr 28 15:49:04 ubuntu kernel: [ 493.877429] 0000000000013d00 ffff8800723ac858 ffff88006837ffd8 0000000000013d00
Apr 28 15:49:04 ubuntu kernel: [ 493.877444] ffff8801005e5b80 ffff8800723ac4a0 ffff88001fff41c8 ffff88001fc53d00
Apr 28 15:49:04 ubuntu kernel: [ 493.877459] Call Trace:
Apr 28 15:49:04 ubuntu kernel: [ 493.877473] [<ffffffff8110c070>] ? sync_page+0x0/0x50
Apr 28 15:49:04 ubuntu kernel: [ 493.877484] [<ffffffff815c0980>] io_schedule+0x70/0xc0
Apr 28 15:49:04 ubuntu kernel: [ 493.877497] [<ffffffff8110c0b0>] sync_page+0x40/0x50
Apr 28 15:49:04 ubuntu kernel: [ 493.877509] [<ffffffff815c12ff>] __wait_on_bit+0x5f/0x90
Apr 28 15:49:04 ubuntu kernel: [ 493.877522] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
Apr 28 15:49:04 ubuntu kernel: [ 493.877534] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
Apr 28 15:49:04 ubuntu kernel: [ 493.877548] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
Apr 28 15:49:04 ubuntu kernel: [ 493.877561] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
Apr 28 15:49:04 ubuntu kernel: [ 493.877573] [<ffffffff8112d134>] __do_fault+0x54/0x520
Apr 28 15:49:04 ubuntu kernel: [ 493.877585] [<ffffffff811309ca>] handle_pte_fault+0xfa/0x210
Apr 28 15:49:04 ubuntu kernel: [ 493.877597] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
Apr 28 15:49:04 ubuntu kernel: [ 493.877608] [<ffffffff8112de3f>] ? __pte_alloc+0xdf/0x100
Apr 28 15:49:04 ubuntu kernel: [ 493.877619] [<ffffffff81131d4d>] handle_mm_fault+0x16d/0x250
Apr 28 15:49:04 ubuntu kernel: [ 493.877632] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
Apr 28 15:49:04 ubuntu kernel: [ 493.877644] [<ffffffff81136f75>] ? do_mmap_pgoff+0x335/0x370
Apr 28 15:49:04 ubuntu kernel: [ 493.877656] [<ffffffff815c34d5>] page_fault+0x25/0x30
Apr 28 15:49:04 ubuntu kernel: [ 493.877672] INFO: task jbd2/sda4-8:2990 blocked for more than 30 seconds.
Apr 28 15:49:04 ubuntu kernel: [ 493.877678] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 15:49:04 ubuntu kernel: [ 493.877686] jbd2/sda4-8 D 0000000000000000 0 2990 2 0x00000000
Apr 28 15:49:04 ubuntu kernel: [ 493.877699] ffff880074f4fd20 0000000000000046 ffff880074f4ffd8 ffff880074f4e000
Apr 28 15:49:04 ubuntu kernel: [ 493.877714] 0000000000013d00 ffff88005c2103b8 ffff880074f4ffd8 0000000000013d00
Apr 28 15:49:04 ubuntu kernel: [ 493.877729] ffffffff81a0b020 ffff88005c210000 0000000000000002 ffff8800755ae5a0
Apr 28 15:49:04 ubuntu kernel: [ 493.877744] Call Trace:
Apr 28 15:49:04 ubuntu kernel: [ 493.877756] [<ffffffff81242d40>] jbd2_journal_commit_transaction+0x1b0/0x1190
Apr 28 15:49:04 ubuntu kernel: [ 493.877772] [<ffffffff8100a6e0>] ? __switch_to+0xc0/0x2f0
Apr 28 15:49:04 ubuntu kernel: [ 493.877786] [<ffffffff815c2cbe>] ? _raw_spin_lock+0xe/0x20
Apr 28 15:49:04 ubuntu kernel: [ 493.877799] [<ffffffff81074fdb>] ? lock_timer_base.clone.20+0x3b/0x70
Apr 28 15:49:04 ubuntu kernel: [ 493.877812] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
Apr 28 15:49:04 ubuntu kernel: [ 493.877824] [<ffffffff81247e8b>] kjournald2+0xbb/0x220
Apr 28 15:49:04 ubuntu kernel: [ 493.877836] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
Apr 28 15:49:04 ubuntu kernel: [ 493.877847] [<ffffffff81247dd0>] ? kjournald2+0x0/0x220
Apr 28 15:49:04 ubuntu kernel: [ 493.877858] [<ffffffff810877e6>] kthread+0x96/0xa0
Apr 28 15:49:04 ubuntu kernel: [ 493.877869] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
Apr 28 15:49:04 ubuntu kernel: [ 493.877882] [<ffffffff81087750>] ? kthread+0x0/0xa0
Apr 28 15:49:04 ubuntu kernel: [ 493.877892] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
Apr 28 15:49:04 ubuntu kernel: [ 553.778917] INFO: task jbd2/sda1-8:290 blocked for more than 30 seconds.
Apr 28 15:49:04 ubuntu kernel: [ 553.778936] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 15:49:04 ubuntu kernel: [ 553.778944] jbd2/sda1-8 D 0000000000000002 0 290 2 0x00000000
Apr 28 15:49:04 ubuntu kernel: [ 553.778956] ffff8800363e5c20 0000000000000046 ffff8800363e5fd8 ffff8800363e4000
Apr 28 15:49:04 ubuntu kernel: [ 553.778969] 0000000000013d00 ffff8800720b3178 ffff8800363e5fd8 0000000000013d00
Apr 28 15:49:04 ubuntu kernel: [ 553.778981] ffff8801005e2dc0 ffff8800720b2dc0 ffff88001ffec230 ffff88001fc53d00
Apr 28 15:49:04 ubuntu kernel: [ 553.779007] Call Trace:
Apr 28 15:49:04 ubuntu kernel: [ 553.779027] [<ffffffff81191db0>] ? sync_buffer+0x0/0x50
Apr 28 15:49:04 ubuntu kernel: [ 553.779043] [<ffffffff815c0980>] io_schedule+0x70/0xc0
Apr 28 15:49:04 ubuntu kernel: [ 553.779055] [<ffffffff81191df0>] sync_buffer+0x40/0x50
Apr 28 15:49:04 ubuntu kernel: [ 553.779067] [<ffffffff815c12ff>] __wait_on_bit+0x5f/0x90
Apr 28 15:49:04 ubuntu kernel: [ 553.779080] [<ffffffff81191db0>] ? sync_buffer+0x0/0x50
Apr 28 15:49:04 ubuntu kernel: [ 553.779093] [<ffffffff815c13ac>] out_of_line_wait_on_bit+0x7c/0x90
Apr 28 15:49:04 ubuntu kernel: [ 553.779107] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
Apr 28 15:49:04 ubuntu kernel: [ 553.779121] [<ffffffff81191dae>] __wait_on_buffer+0x2e/0x30
Apr 28 15:49:04 ubuntu kernel: [ 553.779134] [<ffffffff812432ef>] jbd2_journal_commit_transaction+0x75f/0x1190
Apr 28 15:49:04 ubuntu kernel: [ 553.779148] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
Apr 28 15:49:04 ubuntu kernel: [ 553.779162] [<ffffffff81247e8b>] kjournald2+0xbb/0x220
Apr 28 15:49:04 ubuntu kernel: [ 553.779174] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
Apr 28 15:49:04 ubuntu kernel: [ 553.779185] [<ffffffff81247dd0>] ? kjournald2+0x0/0x220
Apr 28 15:49:04 ubuntu kernel: [ 553.779196] [<ffffffff810877e6>] kthread+0x96/0xa0
Apr 28 15:49:04 ubuntu kernel: [ 553.779210] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
Apr 28 15:49:04 ubuntu kernel: [ 553.779222] [<ffffffff81087750>] ? kthread+0x0/0xa0
Apr 28 15:49:04 ubuntu kernel: [ 553.779233] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
Apr 28 15:49:04 ubuntu kernel: [ 553.779245] INFO: task rs:main Q:Reg:775 blocked for more than 30 seconds.
Apr 28 15:49:04 ubuntu kernel: [ 553.779252] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 15:49:04 ubuntu kernel: [ 553.779260] rs:main Q:Reg D 0000000000000000 0 775 1 0x00000000
Apr 28 15:49:04 ubuntu kernel: [ 553.779274] ffff880036317b98 0000000000000086 ffff880036317fd8 ffff880036316000
Apr 28 15:49:04 ubuntu kernel: [ 553.779289] 0000000000013d00 ffff8800759983b8 ffff880036317fd8 0000000000013d00
Apr 28 15:49:04 ubuntu kernel: [ 553.779304] ffff88003632adc0 ffff880075998000 ffff88001ffe6f68 ffff88001fc13d00
Apr 28 15:49:04 ubuntu kernel: [ 553.779318] Call Trace:
Apr 28 15:49:04 ubuntu kernel: [ 553.779333] [<ffffffff8110c070>] ? sync_page+0x0/0x50
Apr 28 15:49:04 ubuntu kernel: [ 553.779345] [<ffffffff815c0980>] io_schedule+0x70/0xc0
Apr 28 15:49:04 ubuntu kernel: [ 553.779357] [<ffffffff8110c0b0>] sync_page+0x40/0x50
Apr 28 15:49:04 ubuntu kernel: [ 553.779369] [<ffffffff815c12ff>] __wait_on_bit+0x5f/0x90
Apr 28 15:49:04 ubuntu kernel: [ 553.779381] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
Apr 28 15:49:04 ubuntu kernel: [ 553.779393] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
Apr 28 15:49:04 ubuntu kernel: [ 553.779405] [<ffffffff812df74e>] ? radix_tree_lookup_slot+0xe/0x10
Apr 28 15:49:04 ubuntu kernel: [ 553.779418] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
Apr 28 15:49:04 ubuntu kernel: [ 553.779431] [<ffffffff8112f090>] do_swap_page.clone.50+0x5e0/0x640
Apr 28 15:49:04 ubuntu kernel: [ 553.779446] [<ffffffff8108bb90>] ? lock_hrtimer_base.clone.25+0x30/0x60
Apr 28 15:49:04 ubuntu kernel: [ 553.779459] [<ffffffff8108bc5c>] ? hrtimer_try_to_cancel+0x4c/0xe0
Apr 28 15:49:04 ubuntu kernel: [ 553.779472] [<ffffffff81130a89>] handle_pte_fault+0x1b9/0x210
Apr 28 15:49:04 ubuntu kernel: [ 553.779484] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
Apr 28 15:49:04 ubuntu kernel: [ 553.779494] [<ffffffff8112de3f>] ? __pte_alloc+0xdf/0x100
Apr 28 15:49:04 ubuntu kernel: [ 553.779505] [<ffffffff81131d4d>] handle_mm_fault+0x16d/0x250
Apr 28 15:49:04 ubuntu kernel: [ 553.779518] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
Apr 28 15:49:04 ubuntu kernel: [ 553.779531] [<ffffffff8109ce7b>] ? do_futex+0xbb/0x210
Apr 28 15:49:04 ubuntu kernel: [ 553.779543] [<ffffffff8109d04b>] ? sys_futex+0x7b/0x180
Apr 28 15:49:04 ubuntu kernel: [ 553.779555] [<ffffffff8116f985>] ? putname+0x35/0x50
Apr 28 15:49:04 ubuntu kernel: [ 553.779567] [<ffffffff81164200>] ? do_sys_open+0x10/0x150
Apr 28 15:49:04 ubuntu kernel: [ 553.779579] [<ffffffff815c34d5>] page_fault+0x25/0x30
Apr 28 15:49:04 ubuntu kernel: [ 553.779589] INFO: task rsyslogd:795 blocked for more than 30 seconds.
Apr 28 15:49:04 ubuntu kernel: [ 553.779595] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 15:49:04 ubuntu kernel: [ 553.779603] rsyslogd D 0000000000000000 0 795 1 0x00000000
Apr 28 15:49:04 ubuntu kernel: [ 553.779616] ffff880075c0db98 0000000000000086 ffff880075c0dfd8 ffff880075c0c000
Apr 28 15:49:04 ubuntu kernel: [ 553.779631] 0000000000013d00 ffff88003623b178 ffff880075c0dfd8 0000000000013d00
Apr 28 15:49:04 ubuntu kernel: [ 553.779646] ffff880075998000 ffff88003623adc0 ffff88001ffe6f68 ffff88001fc13d00
Apr 28 15:49:04 ubuntu kernel: [ 553.779660] Call Trace:
Apr 28 15:49:04 ubuntu kernel: [ 553.779672] [<ffffffff8110c070>] ? sync_page+0x0/0x50
Apr 28 15:49:04 ubuntu kernel: [ 553.779684] [<ffffffff815c0980>] io_schedule+0x70/0xc0
Apr 28 15:49:04 ubuntu kernel: [ 553.779696] [<ffffffff8110c0b0>] sync_page+0x40/0x50
Apr 28 15:49:04 ubuntu kernel: [ 553.779709] [<ffffffff815c12ff>] __wait_on_bit+0x5f/0x90
Apr 28 15:49:04 ubuntu kernel: [ 553.779720] [<ffffffff8114077f>] ? read_swap_cache_async+0x4f/0x160
Apr 28 15:49:04 ubuntu kernel: [ 553.779734] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
Apr 28 15:49:04 ubuntu kernel: [ 553.779746] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
Apr 28 15:49:04 ubuntu kernel: [ 553.779759] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
Apr 28 15:49:04 ubuntu kernel: [ 553.779770] [<ffffffff8112f090>] do_swap_page.clone.50+0x5e0/0x640
Apr 28 15:49:04 ubuntu kernel: [ 553.779783] [<ffffffff81130a89>] handle_pte_fault+0x1b9/0x210
Apr 28 15:49:04 ubuntu kernel: [ 553.779794] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
Apr 28 15:49:04 ubuntu kernel: [ 553.779805] [<ffffffff8112de3f>] ? __pte_alloc+0xdf/0x100
Apr 28 15:49:04 ubuntu kernel: [ 553.779816] [<ffffffff81131d4d>] handle_mm_fault+0x16d/0x250
Apr 28 15:49:04 ubuntu kernel: [ 553.779828] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
Apr 28 15:49:04 ubuntu kernel: [ 553.779841] [<ffffffff81164fd0>] ? vfs_read+0x120/0x180
Apr 28 15:49:04 ubuntu kernel: [ 553.779851] [<ffffffff815c34d5>] page_fault+0x25/0x30
Apr 28 15:49:04 ubuntu kernel: [ 553.779894] INFO: task tee:2977 blocked for more than 30 seconds.
Apr 28 15:49:04 ubuntu kernel: [ 553.779900] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 15:49:04 ubuntu kernel: [ 553.779908] tee D 0000000000000002 0 2977 2875 0x00000000
Apr 28 15:49:04 ubuntu kernel: [ 553.779921] ffff88005c1e39e8 0000000000000082 ffff88005c1e3fd8 ffff88005c1e2000
Apr 28 15:49:04 ubuntu kernel: [ 553.779935] 0000000000013d00 ffff8800755283b8 ffff88005c1e3fd8 0000000000013d00
Apr 28 15:49:04 ubuntu kernel: [ 553.779950] ffff8801005e5b80 ffff880075528000 ffff88005c1e39d8 ffff88005c1e3a38
Apr 28 15:49:04 ubuntu kernel: [ 553.779965] Call Trace:
Apr 28 15:49:04 ubuntu kernel: [ 553.779980] [<ffffffff81241db5>] do_get_write_access+0x255/0x490
Apr 28 15:49:04 ubuntu kernel: [ 553.779992] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
Apr 28 15:49:04 ubuntu kernel: [ 553.780006] [<ffffffff81209e2d>] ? ext4_dirty_inode+0x3d/0x60
Apr 28 15:49:04 ubuntu kernel: [ 553.780019] [<ffffffff81242131>] jbd2_journal_get_write_access+0x31/0x50
Apr 28 15:49:04 ubuntu kernel: [ 553.780034] [<ffffffff81228f9e>] __ext4_journal_get_write_access+0x3e/0x80
Apr 28 15:49:04 ubuntu kernel: [ 553.780047] [<ffffffff81205268>] ext4_reserve_inode_write+0x78/0xa0
Apr 28 15:49:04 ubuntu kernel: [ 553.780059] [<ffffffff812052df>] ext4_mark_inode_dirty+0x4f/0x220
Apr 28 15:49:04 ubuntu kernel: [ 553.780072] [<ffffffff8121dc03>] ? ext4_journal_start_sb+0xb3/0x140
Apr 28 15:49:04 ubuntu kernel: [ 553.780084] [<ffffffff81209e2d>] ext4_dirty_inode+0x3d/0x60
Apr 28 15:49:04 ubuntu kernel: [ 553.780096] [<ffffffff8118b31f>] __mark_inode_dirty+0x3f/0x260
Apr 28 15:49:04 ubuntu kernel: [ 553.780109] [<ffffffff8117f0b5>] file_update_time+0xf5/0x170
Apr 28 15:49:04 ubuntu kernel: [ 553.780124] [<ffffffff8110d928>] __generic_file_aio_write+0x1f8/0x440
Apr 28 15:49:04 ubuntu kernel: [ 553.780137] [<ffffffff8105f652>] ? default_wake_function+0x12/0x20
Apr 28 15:49:04 ubuntu kernel: [ 553.780150] [<ffffffff8110dbd6>] generic_file_aio_write+0x66/0xd0
Apr 28 15:49:04 ubuntu kernel: [ 553.780162] [<ffffffff811fe2a9>] ext4_file_write+0x69/0x280
Apr 28 15:49:04 ubuntu kernel: [ 553.780176] [<ffffffff8138b0c3>] ? pty_write+0x73/0x80
Apr 28 15:49:04 ubuntu kernel: [ 553.780188] [<ffffffff81038c79>] ? default_spin_lock_flags+0x9/0x10
Apr 28 15:49:04 ubuntu kernel: [ 553.780201] [<ffffffff81164682>] do_sync_write+0xd2/0x110
Apr 28 15:49:04 ubuntu kernel: [ 553.780217] [<ffffffff812ae7e8>] ? apparmor_file_permission+0x18/0x20
Apr 28 15:49:04 ubuntu kernel: [ 553.780231] [<ffffffff81279bac>] ? security_file_permission+0x2c/0xb0
Apr 28 15:49:04 ubuntu kernel: [ 553.780244] [<ffffffff81384d70>] ? n_tty_write+0x0/0x280
Apr 28 15:49:04 ubuntu kernel: [ 553.780256] [<ffffffff81164ab1>] ? rw_verify_area+0x61/0xf0
Apr 28 15:49:04 ubuntu kernel: [ 553.780268] [<ffffffff81164df6>] vfs_write+0xc6/0x180
Apr 28 15:49:04 ubuntu kernel: [ 553.780278] [<ffffffff81165111>] sys_write+0x51/0x90
Apr 28 15:49:04 ubuntu kernel: [ 553.780290] [<ffffffff8100c002>] system_call_fastpath+0x16/0x1b
Apr 28 15:49:04 ubuntu kernel: [ 553.780299] INFO: task jbd2/sda4-8:2990 blocked for more than 30 seconds.
Apr 28 15:49:04 ubuntu kernel: [ 553.780307] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 15:49:04 ubuntu kernel: [ 553.780314] jbd2/sda4-8 D 0000000000000000 0 2990 2 0x00000000
Apr 28 15:49:04 ubuntu kernel: [ 553.780327] ffff880074f4faf0 0000000000000046 ffff880074f4ffd8 ffff880074f4e000
Apr 28 15:49:04 ubuntu kernel: [ 553.780342] 0000000000013d00 ffff88005c2103b8 ffff880074f4ffd8 0000000000013d00
Apr 28 15:49:04 ubuntu kernel: [ 553.780357] ffffffff81a0b020 ffff88005c210000 ffff88001ffee048 ffff88001fc13d00
Apr 28 15:49:04 ubuntu kernel: [ 553.780372] Call Trace:
Apr 28 15:49:04 ubuntu kernel: [ 553.780384] [<ffffffff8110c070>] ? sync_page+0x0/0x50
Apr 28 15:49:04 ubuntu kernel: [ 553.780396] [<ffffffff815c0980>] io_schedule+0x70/0xc0
Apr 28 15:49:04 ubuntu kernel: [ 553.780407] [<ffffffff8110c0b0>] sync_page+0x40/0x50
Apr 28 15:49:04 ubuntu kernel: [ 553.780419] [<ffffffff815c12ff>] __wait_on_bit+0x5f/0x90
Apr 28 15:49:04 ubuntu kernel: [ 553.780432] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
Apr 28 15:49:04 ubuntu kernel: [ 553.780444] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
Apr 28 15:49:04 ubuntu kernel: [ 553.780457] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190
Apr 28 15:49:04 ubuntu kernel: [ 553.780472] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30
Apr 28 15:49:04 ubuntu kernel: [ 553.780484] [<ffffffff81242a83>] journal_finish_inode_data_buffers+0x63/0x170
Apr 28 15:49:04 ubuntu kernel: [ 553.780496] [<ffffffff81243274>] jbd2_journal_commit_transaction+0x6e4/0x1190
Apr 28 15:49:04 ubuntu kernel: [ 553.780510] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
Apr 28 15:49:04 ubuntu kernel: [ 553.780522] [<ffffffff81247e8b>] kjournald2+0xbb/0x220
Apr 28 15:49:04 ubuntu kernel: [ 553.780534] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
Apr 28 15:49:04 ubuntu kernel: [ 553.780546] [<ffffffff81247dd0>] ? kjournald2+0x0/0x220
Apr 28 15:49:04 ubuntu kernel: [ 553.780556] [<ffffffff810877e6>] kthread+0x96/0xa0
Apr 28 15:49:04 ubuntu kernel: [ 553.780568] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
Apr 28 15:49:04 ubuntu kernel: [ 553.780580] [<ffffffff81087750>] ? kthread+0x0/0xa0
Apr 28 15:49:04 ubuntu kernel: [ 553.780591] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-27 18:25 ` Colin Ian King
@ 2011-04-28 15:57 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 15:57 UTC (permalink / raw)
To: Colin Ian King
Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4
On Wed, 2011-04-27 at 19:25 +0100, Colin Ian King wrote:
> Just like to add that I've seen almost identical issues with 2.6.38
> copying large amounts of data to an ext4 filesystem with systems with
> small amounts of memory.
>
> I found that increasing /sys/fs/ext4/sdaX/max_writeback_mb_bump worked
> around the issue.
With the PREEMPT kernel, values of 256 and 512 don't prevent kswapd
spinning up to 99% and staying there.
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 15:57 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 15:57 UTC (permalink / raw)
To: Colin Ian King
Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, linux-ext4
On Wed, 2011-04-27 at 19:25 +0100, Colin Ian King wrote:
> Just like to add that I've seen almost identical issues with 2.6.38
> copying large amounts of data to an ext4 filesystem with systems with
> small amounts of memory.
>
> I found that increasing /sys/fs/ext4/sdaX/max_writeback_mb_bump worked
> around the issue.
With the PREEMPT kernel, values of 256 and 512 don't prevent kswapd
spinning up to 99% and staying there.
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 15:08 ` Mel Gorman
@ 2011-04-28 16:01 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 16:01 UTC (permalink / raw)
To: Mel Gorman
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 16:08 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 09:25:14AM -0500, James Bottomley wrote:
> > On Thu, 2011-04-28 at 15:07 +0100, Mel Gorman wrote:
> > > On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote:
> > > > On Thu 28-04-11 12:36:30, Colin Ian King wrote:
> > > > > One more data point to add, I've been looking at an identical issue when
> > > > > copying large amounts of data. I bisected this - and the lockups occur
> > > > > with commit
> > > > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > > > > issue. With this commit, my file copy test locks up after ~8-10
> > > > > iterations, before this commit I can copy > 100 times and don't see the
> > > > > lockup.
> > > > Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit
> > > > of yours causes kswapd on non-preempt kernels spin for a *long* time...
> > > >
> > >
> > > I'm still thinking about the traces which do not point the finger
> > > directly at compaction per-se but it's possible that the change means
> > > kswapd is not reclaiming like it should be.
> > >
> > > To test this theory, does applying
> > > [d527caf2: mm: compaction: prevent kswapd compacting memory to reduce
> > > CPU usage] help?
> >
> > I can answer definitively no to this. The upstream kernel I reproduced
> > this on has that patch included.
> >
>
> So it is.
>
> Another consequence of this patch is that when high order allocations
> are in progress (is the test case fork heavy in any way for
> example?
It's a simple huge untar, so it shouldn't fork.
> alternatively, it might be something in the storage stack
> that requires high-order allocs)
I've tried switching from CFQ to deadline with no apparent changes
> we are no longer necessarily going
> to sleep because of should_reclaim_continue() check. This could
> explain kswapd-at-99% but would only apply if CONFIG_COMPACTION is
> set (does unsetting CONFIG_COMPACTION help). If the bug only triggers
> for CONFIG_COMPACTION, does the following *untested* patch help any?
Turning off COMPACTION and HUGEPAGES doesn't help ... kswapd still goes
to 99% on the PREEMPT kernel, so it doesn't seem to be related
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 16:01 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 16:01 UTC (permalink / raw)
To: Mel Gorman
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 16:08 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 09:25:14AM -0500, James Bottomley wrote:
> > On Thu, 2011-04-28 at 15:07 +0100, Mel Gorman wrote:
> > > On Thu, Apr 28, 2011 at 03:52:28PM +0200, Jan Kara wrote:
> > > > On Thu 28-04-11 12:36:30, Colin Ian King wrote:
> > > > > One more data point to add, I've been looking at an identical issue when
> > > > > copying large amounts of data. I bisected this - and the lockups occur
> > > > > with commit
> > > > > 3e7d344970673c5334cf7b5bb27c8c0942b06126 - before that I don't see the
> > > > > issue. With this commit, my file copy test locks up after ~8-10
> > > > > iterations, before this commit I can copy > 100 times and don't see the
> > > > > lockup.
> > > > Adding Mel to CC, I guess he'll be interested. Mel, it seems this commit
> > > > of yours causes kswapd on non-preempt kernels spin for a *long* time...
> > > >
> > >
> > > I'm still thinking about the traces which do not point the finger
> > > directly at compaction per-se but it's possible that the change means
> > > kswapd is not reclaiming like it should be.
> > >
> > > To test this theory, does applying
> > > [d527caf2: mm: compaction: prevent kswapd compacting memory to reduce
> > > CPU usage] help?
> >
> > I can answer definitively no to this. The upstream kernel I reproduced
> > this on has that patch included.
> >
>
> So it is.
>
> Another consequence of this patch is that when high order allocations
> are in progress (is the test case fork heavy in any way for
> example?
It's a simple huge untar, so it shouldn't fork.
> alternatively, it might be something in the storage stack
> that requires high-order allocs)
I've tried switching from CFQ to deadline with no apparent changes
> we are no longer necessarily going
> to sleep because of should_reclaim_continue() check. This could
> explain kswapd-at-99% but would only apply if CONFIG_COMPACTION is
> set (does unsetting CONFIG_COMPACTION help). If the bug only triggers
> for CONFIG_COMPACTION, does the following *untested* patch help any?
Turning off COMPACTION and HUGEPAGES doesn't help ... kswapd still goes
to 99% on the PREEMPT kernel, so it doesn't seem to be related
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 14:33 ` Jan Kara
@ 2011-04-28 16:11 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 16:11 UTC (permalink / raw)
To: Jan Kara
Cc: Colin Ian King, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4
On Thu, 2011-04-28 at 16:33 +0200, Jan Kara wrote:
> James, Colin, can you capture output of 'vmstat 1' while you do the
> copying? Thanks.
Sure; this is with a PREEMPT kernel so the tar goes to completion
without locking the system.
I started using a memory balloon to clear out the page cache.
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 98708 1676100 4112 39556 0 45 1566 1879 392 168 14 17 59 10 0
0 0 98708 1676100 4112 39604 0 0 0 0 37 35 0 0 100 0 0
0 0 98708 1676100 4112 39652 0 0 0 0 31 35 0 0 100 0 0
0 0 98708 1676100 4112 39652 0 0 0 0 26 27 0 0 100 0 0
0 0 98708 1676100 4112 39652 32 0 32 0 50 48 0 0 100 0 0
1 1 98708 1673984 4616 40576 192 0 1812 0 286 117 0 0 92 8 0
0 1 98708 1590540 4792 124236 0 0 41972 0 901 822 0 4 67 29 0
0 1 98708 1499524 4808 214820 0 0 45328 0 917 890 0 4 77 20 0
0 1 98708 1404532 4808 308316 64 0 46784 0 965 1033 0 3 77 20 0
0 1 98708 1312020 5612 399008 0 0 44836 0 1274 1075 0 6 77 18 0
0 1 98708 1304540 5628 406324 0 0 3600 47260 289 115 0 2 68 30 0
0 1 98708 1230760 5672 479020 0 0 36260 712 914 854 0 3 79 18 0
0 1 98708 1208696 5676 500016 0 0 10500 93888 421 251 0 3 67 31 0
0 1 98708 1157632 5676 551264 0 0 25600 0 658 518 0 2 75 23 0
0 1 98708 1052356 5680 655344 0 0 51972 0 1112 1112 0 4 78 19 0
0 2 98708 1024960 5680 679080 0 0 11904 87304 477 277 0 3 54 43 0
1 1 98708 1027288 5684 679432 0 0 128 2896 195 42 0 0 85 14 0
0 1 98708 947124 5688 758528 0 0 39552 4 932 899 0 4 68 28 0
0 1 98708 849792 5692 854272 0 0 47876 4468 1045 936 0 4 77 20 0
0 1 98708 813000 5820 890492 0 0 18176 18300 512 386 0 3 66 32 0
1 2 98708 794304 5820 908132 0 0 8832 95640 379 324 0 2 77 21 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 1 98708 794272 5824 908128 0 0 0 39456 212 58 0 1 73 26 0
0 1 98708 792344 5832 910452 0 0 1156 44440 193 62 0 1 74 25 0
0 1 98708 764392 5836 938288 0 0 13956 0 441 374 0 1 79 19 0
0 1 98708 660828 5840 1040232 0 0 50948 0 1141 992 0 6 65 29 0
0 1 98708 555156 5840 1144216 0 0 51968 0 1072 1008 0 4 76 20 0
0 1 98708 458692 5856 1239212 0 0 47496 120 995 1046 0 3 78 19 0
0 1 98708 411336 5856 1285896 0 0 23296 17764 536 463 0 3 71 26 0
1 1 98708 367716 5856 1328904 0 0 21504 22272 541 463 0 3 78 20 0
0 1 98708 330048 5864 1366044 0 0 18568 17524 529 515 0 2 78 20 0
0 1 98708 301620 5864 1393912 0 0 13952 28148 382 309 0 2 68 30 0
1 1 98708 245140 5872 1449576 0 0 27776 13224 610 567 0 3 73 24 0
0 2 98708 226868 5872 1467000 0 0 8704 114804 375 322 0 2 77 21 0
0 2 98708 226268 5876 1467552 0 0 260 51720 204 40 0 1 75 24 0
0 1 98708 226548 5880 1467536 0 0 4 13180 149 29 0 1 70 30 0
2 0 98708 159792 5880 1533844 0 0 33152 0 721 747 0 2 81 17 0
0 1 98708 55252 5888 1636692 0 0 51456 104 1021 977 0 5 70 25 0
1 1 98708 22804 2680 1671344 0 0 53632 0 1806 1188 0 20 59 21 0
2 1 98708 22832 2672 1671372 0 0 40704 11196 1967 1126 0 22 60 17 0
1 1 98708 22908 2676 1671196 0 0 24580 23040 1621 705 0 35 36 30 0
1 1 98708 22968 2676 1670796 0 0 21504 22212 1576 674 0 22 60 18 0
1 1 98708 22672 2684 1670996 0 0 17920 16968 1450 687 0 27 51 23 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 1 98708 22904 2452 1671344 0 0 24832 25780 1632 714 0 27 50 23 0
1 2 98708 22764 2452 1670724 0 0 15488 96304 1459 531 0 37 33 30 0
2 2 98708 22640 2452 1670724 0 0 0 60300 1219 356 0 20 45 35 0
1 1 98708 22744 2452 1670688 0 0 768 29312 1200 253 0 29 36 34 0
2 0 98708 22964 2460 1671040 0 0 10752 68 1345 445 0 23 56 21 0
1 1 98708 22652 2536 1671560 0 0 51540 0 2113 1246 0 28 49 23 0
1 1 98708 22676 2528 1671768 0 0 118276 14604 4875 2681 0 27 51 22 0
1 1 98708 22644 2528 1671804 0 0 23680 18172 1539 774 0 29 44 27 0
1 1 98708 22720 2536 1671464 0 0 21888 24664 1494 651 0 24 55 22 0
2 1 98708 22672 2496 1671592 0 0 22528 18144 1493 643 0 30 42 28 0
1 1 98708 22732 2488 1671508 0 0 22528 25192 1537 726 0 23 55 21 0
1 2 98708 22884 2488 1670504 0 0 18816 89440 1464 581 0 31 37 32 0
1 2 98708 22740 2488 1670764 0 0 768 75012 1196 254 0 22 55 23 0
1 2 98708 22848 2492 1670600 0 0 256 15512 1225 358 0 29 27 44 0
1 1 98708 22620 2500 1671628 0 0 19076 11408 1490 575 0 23 54 23 0
1 1 98708 22988 2500 1671108 0 0 55040 0 2118 1203 0 27 51 22 0
1 1 98708 22876 2488 1671044 0 0 53760 0 2138 1323 0 28 49 23 0
1 1 98708 22904 2500 1671012 0 0 48012 5528 2049 1097 0 32 43 25 0
1 1 98708 22856 2508 1670876 0 0 21504 18536 1500 646 0 23 55 22 0
2 0 98708 22684 2504 1670972 0 0 23680 23552 1537 784 0 29 44 26 0
1 1 98708 22960 2500 1670520 0 0 23808 25552 1529 677 0 24 55 21 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 1 98708 22820 2504 1670804 0 0 21764 25104 1507 647 0 30 43 26 0
1 1 98708 22880 2504 1670812 0 0 24192 18900 1541 768 0 23 55 22 0
2 3 98708 22716 2512 1670068 0 0 2816 132592 1257 303 0 30 30 40 0
1 1 98708 22944 2512 1669940 0 0 512 40400 1165 247 0 23 57 20 0
1 1 98708 22672 2512 1670624 0 0 512 0 1186 349 0 28 44 29 0
1 1 98708 22788 2504 1670416 0 0 51200 0 2130 1158 0 25 53 22 0
1 1 98708 22660 2508 1670688 0 0 50564 0 1972 1161 0 31 43 26 0
1 1 98708 22780 2516 1670088 0 0 50176 64 2028 1204 0 24 56 20 0
1 1 98708 22856 2508 1669884 0 0 32640 19040 1723 819 0 31 43 26 0
3 0 98708 22808 2512 1669812 0 0 23812 19176 1535 670 0 23 55 22 0
1 1 98708 22744 2504 1669728 0 0 21888 24872 1535 653 0 30 43 27 0
1 1 98708 22696 2504 1669636 0 0 22784 25032 1529 647 0 23 55 21 0
1 1 98708 22632 2512 1669596 0 0 21896 19228 1515 647 0 30 43 27 0
1 2 98708 22676 2512 1668992 0 0 9600 74484 1366 418 0 24 43 33 0
1 2 98708 22780 2512 1668772 0 0 256 107120 1211 245 0 29 42 30 0
1 1 98708 23196 2512 1668920 0 0 256 40 1158 228 0 23 57 20 0
2 1 98708 22968 2512 1669092 0 0 47360 0 2023 1097 0 26 49 25 0
1 1 98708 22864 2704 1668784 0 0 39992 116 1878 994 0 27 49 24 0
1 1 98708 22704 2708 1668872 0 0 53388 0 2166 1206 0 28 49 23 0
1 1 98708 22796 2712 1668472 0 0 39940 12784 1873 966 0 27 52 21 0
2 1 98708 22624 2712 1668508 0 0 21888 21904 1498 645 0 30 43 27 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 1 98708 22700 2704 1668688 0 0 24320 26240 1548 685 0 23 55 22 0
1 1 98708 22776 2712 1668284 0 0 18048 17532 1438 586 0 30 43 27 0
1 1 98708 22868 2712 1668336 0 0 24192 21764 1552 661 0 23 55 23 0
2 2 98708 22772 2716 1667696 0 0 12420 88772 1433 494 0 30 41 28 0
1 2 98708 23424 2720 1667584 0 0 4 9092 1181 246 0 23 40 37 0
1 1 98708 23120 2724 1667584 0 0 4 86380 1262 269 0 29 42 29 0
1 0 98708 23408 2984 1667708 0 0 364 0 1187 249 0 22 60 18 0
2 0 98708 23416 2992 1667772 0 0 0 120 1045 246 0 25 74 1 0
1 0 98708 23408 2992 1667720 0 0 0 0 1025 221 0 26 74 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1030 225 0 28 72 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1024 224 0 23 77 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1034 226 0 28 72 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1022 221 0 23 77 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1032 227 0 28 72 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1024 221 0 23 77 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1034 228 0 28 72 0 0
kswapd goes up to 99% at nearly the end of the second page. I ^C on the
tar but kswapd stays at 99% for the end.
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 16:11 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 16:11 UTC (permalink / raw)
To: Jan Kara
Cc: Colin Ian King, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4
On Thu, 2011-04-28 at 16:33 +0200, Jan Kara wrote:
> James, Colin, can you capture output of 'vmstat 1' while you do the
> copying? Thanks.
Sure; this is with a PREEMPT kernel so the tar goes to completion
without locking the system.
I started using a memory balloon to clear out the page cache.
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 98708 1676100 4112 39556 0 45 1566 1879 392 168 14 17 59 10 0
0 0 98708 1676100 4112 39604 0 0 0 0 37 35 0 0 100 0 0
0 0 98708 1676100 4112 39652 0 0 0 0 31 35 0 0 100 0 0
0 0 98708 1676100 4112 39652 0 0 0 0 26 27 0 0 100 0 0
0 0 98708 1676100 4112 39652 32 0 32 0 50 48 0 0 100 0 0
1 1 98708 1673984 4616 40576 192 0 1812 0 286 117 0 0 92 8 0
0 1 98708 1590540 4792 124236 0 0 41972 0 901 822 0 4 67 29 0
0 1 98708 1499524 4808 214820 0 0 45328 0 917 890 0 4 77 20 0
0 1 98708 1404532 4808 308316 64 0 46784 0 965 1033 0 3 77 20 0
0 1 98708 1312020 5612 399008 0 0 44836 0 1274 1075 0 6 77 18 0
0 1 98708 1304540 5628 406324 0 0 3600 47260 289 115 0 2 68 30 0
0 1 98708 1230760 5672 479020 0 0 36260 712 914 854 0 3 79 18 0
0 1 98708 1208696 5676 500016 0 0 10500 93888 421 251 0 3 67 31 0
0 1 98708 1157632 5676 551264 0 0 25600 0 658 518 0 2 75 23 0
0 1 98708 1052356 5680 655344 0 0 51972 0 1112 1112 0 4 78 19 0
0 2 98708 1024960 5680 679080 0 0 11904 87304 477 277 0 3 54 43 0
1 1 98708 1027288 5684 679432 0 0 128 2896 195 42 0 0 85 14 0
0 1 98708 947124 5688 758528 0 0 39552 4 932 899 0 4 68 28 0
0 1 98708 849792 5692 854272 0 0 47876 4468 1045 936 0 4 77 20 0
0 1 98708 813000 5820 890492 0 0 18176 18300 512 386 0 3 66 32 0
1 2 98708 794304 5820 908132 0 0 8832 95640 379 324 0 2 77 21 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 1 98708 794272 5824 908128 0 0 0 39456 212 58 0 1 73 26 0
0 1 98708 792344 5832 910452 0 0 1156 44440 193 62 0 1 74 25 0
0 1 98708 764392 5836 938288 0 0 13956 0 441 374 0 1 79 19 0
0 1 98708 660828 5840 1040232 0 0 50948 0 1141 992 0 6 65 29 0
0 1 98708 555156 5840 1144216 0 0 51968 0 1072 1008 0 4 76 20 0
0 1 98708 458692 5856 1239212 0 0 47496 120 995 1046 0 3 78 19 0
0 1 98708 411336 5856 1285896 0 0 23296 17764 536 463 0 3 71 26 0
1 1 98708 367716 5856 1328904 0 0 21504 22272 541 463 0 3 78 20 0
0 1 98708 330048 5864 1366044 0 0 18568 17524 529 515 0 2 78 20 0
0 1 98708 301620 5864 1393912 0 0 13952 28148 382 309 0 2 68 30 0
1 1 98708 245140 5872 1449576 0 0 27776 13224 610 567 0 3 73 24 0
0 2 98708 226868 5872 1467000 0 0 8704 114804 375 322 0 2 77 21 0
0 2 98708 226268 5876 1467552 0 0 260 51720 204 40 0 1 75 24 0
0 1 98708 226548 5880 1467536 0 0 4 13180 149 29 0 1 70 30 0
2 0 98708 159792 5880 1533844 0 0 33152 0 721 747 0 2 81 17 0
0 1 98708 55252 5888 1636692 0 0 51456 104 1021 977 0 5 70 25 0
1 1 98708 22804 2680 1671344 0 0 53632 0 1806 1188 0 20 59 21 0
2 1 98708 22832 2672 1671372 0 0 40704 11196 1967 1126 0 22 60 17 0
1 1 98708 22908 2676 1671196 0 0 24580 23040 1621 705 0 35 36 30 0
1 1 98708 22968 2676 1670796 0 0 21504 22212 1576 674 0 22 60 18 0
1 1 98708 22672 2684 1670996 0 0 17920 16968 1450 687 0 27 51 23 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 1 98708 22904 2452 1671344 0 0 24832 25780 1632 714 0 27 50 23 0
1 2 98708 22764 2452 1670724 0 0 15488 96304 1459 531 0 37 33 30 0
2 2 98708 22640 2452 1670724 0 0 0 60300 1219 356 0 20 45 35 0
1 1 98708 22744 2452 1670688 0 0 768 29312 1200 253 0 29 36 34 0
2 0 98708 22964 2460 1671040 0 0 10752 68 1345 445 0 23 56 21 0
1 1 98708 22652 2536 1671560 0 0 51540 0 2113 1246 0 28 49 23 0
1 1 98708 22676 2528 1671768 0 0 118276 14604 4875 2681 0 27 51 22 0
1 1 98708 22644 2528 1671804 0 0 23680 18172 1539 774 0 29 44 27 0
1 1 98708 22720 2536 1671464 0 0 21888 24664 1494 651 0 24 55 22 0
2 1 98708 22672 2496 1671592 0 0 22528 18144 1493 643 0 30 42 28 0
1 1 98708 22732 2488 1671508 0 0 22528 25192 1537 726 0 23 55 21 0
1 2 98708 22884 2488 1670504 0 0 18816 89440 1464 581 0 31 37 32 0
1 2 98708 22740 2488 1670764 0 0 768 75012 1196 254 0 22 55 23 0
1 2 98708 22848 2492 1670600 0 0 256 15512 1225 358 0 29 27 44 0
1 1 98708 22620 2500 1671628 0 0 19076 11408 1490 575 0 23 54 23 0
1 1 98708 22988 2500 1671108 0 0 55040 0 2118 1203 0 27 51 22 0
1 1 98708 22876 2488 1671044 0 0 53760 0 2138 1323 0 28 49 23 0
1 1 98708 22904 2500 1671012 0 0 48012 5528 2049 1097 0 32 43 25 0
1 1 98708 22856 2508 1670876 0 0 21504 18536 1500 646 0 23 55 22 0
2 0 98708 22684 2504 1670972 0 0 23680 23552 1537 784 0 29 44 26 0
1 1 98708 22960 2500 1670520 0 0 23808 25552 1529 677 0 24 55 21 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 1 98708 22820 2504 1670804 0 0 21764 25104 1507 647 0 30 43 26 0
1 1 98708 22880 2504 1670812 0 0 24192 18900 1541 768 0 23 55 22 0
2 3 98708 22716 2512 1670068 0 0 2816 132592 1257 303 0 30 30 40 0
1 1 98708 22944 2512 1669940 0 0 512 40400 1165 247 0 23 57 20 0
1 1 98708 22672 2512 1670624 0 0 512 0 1186 349 0 28 44 29 0
1 1 98708 22788 2504 1670416 0 0 51200 0 2130 1158 0 25 53 22 0
1 1 98708 22660 2508 1670688 0 0 50564 0 1972 1161 0 31 43 26 0
1 1 98708 22780 2516 1670088 0 0 50176 64 2028 1204 0 24 56 20 0
1 1 98708 22856 2508 1669884 0 0 32640 19040 1723 819 0 31 43 26 0
3 0 98708 22808 2512 1669812 0 0 23812 19176 1535 670 0 23 55 22 0
1 1 98708 22744 2504 1669728 0 0 21888 24872 1535 653 0 30 43 27 0
1 1 98708 22696 2504 1669636 0 0 22784 25032 1529 647 0 23 55 21 0
1 1 98708 22632 2512 1669596 0 0 21896 19228 1515 647 0 30 43 27 0
1 2 98708 22676 2512 1668992 0 0 9600 74484 1366 418 0 24 43 33 0
1 2 98708 22780 2512 1668772 0 0 256 107120 1211 245 0 29 42 30 0
1 1 98708 23196 2512 1668920 0 0 256 40 1158 228 0 23 57 20 0
2 1 98708 22968 2512 1669092 0 0 47360 0 2023 1097 0 26 49 25 0
1 1 98708 22864 2704 1668784 0 0 39992 116 1878 994 0 27 49 24 0
1 1 98708 22704 2708 1668872 0 0 53388 0 2166 1206 0 28 49 23 0
1 1 98708 22796 2712 1668472 0 0 39940 12784 1873 966 0 27 52 21 0
2 1 98708 22624 2712 1668508 0 0 21888 21904 1498 645 0 30 43 27 0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 1 98708 22700 2704 1668688 0 0 24320 26240 1548 685 0 23 55 22 0
1 1 98708 22776 2712 1668284 0 0 18048 17532 1438 586 0 30 43 27 0
1 1 98708 22868 2712 1668336 0 0 24192 21764 1552 661 0 23 55 23 0
2 2 98708 22772 2716 1667696 0 0 12420 88772 1433 494 0 30 41 28 0
1 2 98708 23424 2720 1667584 0 0 4 9092 1181 246 0 23 40 37 0
1 1 98708 23120 2724 1667584 0 0 4 86380 1262 269 0 29 42 29 0
1 0 98708 23408 2984 1667708 0 0 364 0 1187 249 0 22 60 18 0
2 0 98708 23416 2992 1667772 0 0 0 120 1045 246 0 25 74 1 0
1 0 98708 23408 2992 1667720 0 0 0 0 1025 221 0 26 74 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1030 225 0 28 72 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1024 224 0 23 77 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1034 226 0 28 72 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1022 221 0 23 77 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1032 227 0 28 72 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1024 221 0 23 77 0 0
1 0 98708 23408 2992 1667720 0 0 0 0 1034 228 0 28 72 0 0
kswapd goes up to 99% at nearly the end of the second page. I ^C on the
tar but kswapd stays at 99% for the end.
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 16:01 ` James Bottomley
@ 2011-04-28 16:50 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 16:50 UTC (permalink / raw)
To: Mel Gorman
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
This is the output of perf record -g -a -f sleep 5
(hopefully the list won't choke)
James
---
# Events: 1K cycles
#
# Overhead Command Shared Object Symbol
# ........ ........... ................. .......................................
#
9.03% tar [kernel.kallsyms] [k] copy_user_generic_string
|
--- copy_user_generic_string
|
|--60.55%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--39.45%-- generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
3.45% swapper [kernel.kallsyms] [k] intel_idle
|
--- intel_idle
cpuidle_idle_call
cpu_idle
|
|--76.47%-- rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--23.53%-- start_secondary
3.39% tar [kernel.kallsyms] [k] zero_user_segments
|
--- zero_user_segments
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
1.78% tar [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--16.72%-- __percpu_counter_add
| |
| |--59.73%-- __prop_inc_single
| | task_dirty_inc
| | account_page_dirtied
| | __set_page_dirty
| | mark_buffer_dirty
| | __block_commit_write
| | block_write_end
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --40.27%-- ext4_claim_free_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--14.83%-- _raw_spin_lock_irq
| |
| |--63.66%-- add_to_page_cache_locked
| | add_to_page_cache_lru
| | |
| | |--51.51%-- mpage_readpages
| | | ext4_readpages
| | | __do_page_cache_readahead
| | | ra_submit
| | | ondemand_readahead
| | | page_cache_async_readahead
| | | generic_file_aio_read
| | | do_sync_read
| | | vfs_read
| | | sys_read
| | | system_call_fastpath
| | | __GI___libc_read
| | |
| | --48.49%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --36.34%-- blk_throtl_bio
| generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--13.85%-- bit_spin_lock
| jbd2_journal_add_journal_head
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--12.19%-- __lru_cache_add
| add_to_page_cache_lru
| |
| |--70.86%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --29.14%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--8.30%-- bit_spin_lock
| lock_page_cgroup
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| |
| |--58.88%-- mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --41.12%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--7.17%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.90%-- __srcu_read_lock
| fsnotify
| fsnotify_modify
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.61%-- __srcu_read_unlock
| fsnotify
| fsnotify_modify
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.14%-- _raw_read_lock_irqsave
| dm_get_live_table
| dm_merge_bvec
| __bio_add_page.part.2
| bio_add_page
| do_mpage_readpage
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--3.81%-- _raw_read_lock
| start_this_handle
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--3.77%-- balance_dirty_pages_ratelimited_nr
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--3.55%-- __add_bdi_stat
| account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--2.18%-- _raw_spin_lock
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
1.70% tar [kernel.kallsyms] [k] __list_del_entry
|
--- __list_del_entry
|
|--72.43%-- list_del
| |
| |--76.01%-- __rmqueue
| | get_page_from_freelist
| | __alloc_pages_nodemask
| | alloc_pages_current
| | |
| | |--92.23%-- __page_cache_alloc
| | | |
| | | |--60.54%-- __do_page_cache_readahead
| | | | ra_submit
| | | | ondemand_readahead
| | | | page_cache_async_readahead
| | | | generic_file_aio_read
| | | | do_sync_read
| | | | vfs_read
| | | | sys_read
| | | | system_call_fastpath
| | | | __GI___libc_read
| | | |
| | | --39.46%-- grab_cache_page_write_begin
| | | ext4_da_write_begin
| | | generic_file_buffered_write
| | | __generic_file_aio_write
| | | generic_file_aio_write
| | | ext4_file_write
| | | do_sync_write
| | | vfs_write
| | | sys_write
| | | system_call_fastpath
| | | __GI___libc_write
| | |
| | --7.77%-- alloc_slab_page
| | new_slab
| | __slab_alloc
| | kmem_cache_alloc
| | alloc_buffer_head
| | alloc_page_buffers
| | create_empty_buffers
| | __block_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| |--18.55%-- get_page_from_freelist
| | __alloc_pages_nodemask
| | alloc_pages_current
| | __page_cache_alloc
| | |
| | |--60.32%-- __do_page_cache_readahead
| | | ra_submit
| | | ondemand_readahead
| | | page_cache_async_readahead
| | | generic_file_aio_read
| | | do_sync_read
| | | vfs_read
| | | sys_read
| | | system_call_fastpath
| | | __GI___libc_read
| | |
| | --39.68%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --5.44%-- activate_page
| mark_page_accessed
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--17.37%-- __rmqueue
| get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--63.38%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --36.62%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--10.20%-- get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--65.70%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--34.30%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
1.53% tar tar [.] 0x75ff
|
--- 0x407546
|
--- 0x42a9c2
|
--- 0x40753f
|
--- 0x40e790
|
--- 0x407586
|
--- 0x40fe1a
|
--- 0x424b50
|
--- 0x412972
|
--- 0x407501
|
--- 0x433336
|
--- 0x4075ff
|
--- 0x403b40
|
--- 0x4055c7
|
--- 0x4332d9
|
--- 0x405efa
|
--- 0x42a9fe
|
--- 0x405a43
|
--- 0x4332d6
1.44% tar [kernel.kallsyms] [k] __ext4_get_inode_loc
|
--- __ext4_get_inode_loc
ext4_get_inode_loc
|
|--62.07%-- ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| |
| |--94.67%-- generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --5.33%-- file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--37.93%-- ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
1.43% tar [kernel.kallsyms] [k] kmem_cache_alloc
|
--- kmem_cache_alloc
|
|--66.77%-- alloc_buffer_head
| alloc_page_buffers
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--22.22%-- jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| |
| |--65.27%-- ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --34.73%-- ext4_dirty_inode
| __mark_inode_dirty
| file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--6.29%-- radix_tree_preload
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--4.72%-- scsi_pool_alloc_command
scsi_host_alloc_command
__scsi_get_command
scsi_get_command
scsi_get_cmd_from_req
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
1.25% tar [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
|
|--24.47%-- _raw_spin_unlock_irqrestore
| |
| |--30.14%-- scsi_get_command
| | scsi_get_cmd_from_req
| | scsi_setup_fs_cmnd
| | sd_prep_fn
| | blk_peek_request
| | scsi_request_fn
| | __blk_run_queue
| | queue_unplugged
| | blk_flush_plug_list
| | io_schedule
| | sleep_on_page_killable
| | __wait_on_bit_lock
| | __lock_page_killable
| | lock_page_killable
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| |--28.74%-- prepare_to_wait_exclusive
| | __wait_on_bit_lock
| | __lock_page_killable
| | lock_page_killable
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| |--23.48%-- __wake_up
| | jbd2_journal_stop
| | __ext4_journal_stop
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --17.65%-- blkiocg_update_dispatch_stats
| throtl_charge_bio
| blk_throtl_bio
| generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--12.68%-- radix_tree_preload_end
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--12.51%-- pagefault_enable
| iov_iter_copy_from_user_atomic
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--10.90%-- _raw_spin_unlock_irq
| |
| |--52.31%-- add_to_page_cache_locked
| | add_to_page_cache_lru
| | mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --47.69%-- __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--10.37%-- mem_cgroup_charge_statistics
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--9.99%-- _raw_spin_unlock
| |
| |--60.73%-- inode_add_rsv_space
| | __dquot_alloc_space
| | ext4_da_get_block_prep
| | __block_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --39.27%-- ext4_ext_map_blocks
| ext4_map_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--5.89%-- __wake_up
| jbd2_journal_stop
| __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.86%-- balance_dirty_pages_ratelimited_nr
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.52%-- add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--3.81%-- avc_has_perm_noaudit
task_has_capability
selinux_capable
security_capable
ns_capable
generic_permission
inode_permission
may_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
1.24% tar [kernel.kallsyms] [k] bit_spin_lock
|
--- bit_spin_lock
|
|--94.43%-- lock_page_cgroup
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| |
| |--56.27%-- mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --43.73%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--5.57%-- __mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
1.13% swapper [kernel.kallsyms] [k] poll_idle
|
--- poll_idle
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
1.00% tar [kernel.kallsyms] [k] ext4_da_get_block_prep
|
--- ext4_da_get_block_prep
|
|--82.10%-- __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--17.90%-- ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.95% tar [kernel.kallsyms] [k] __find_get_block
|
--- __find_get_block
|
|--85.78%-- __getblk
| __ext4_get_inode_loc
| ext4_get_inode_loc
| |
| |--84.94%-- ext4_reserve_inode_write
| | ext4_mark_inode_dirty
| | ext4_dirty_inode
| | __mark_inode_dirty
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --15.06%-- ext4_xattr_get
| ext4_xattr_security_get
| generic_getxattr
| cap_inode_need_killpriv
| security_inode_need_killpriv
| file_remove_suid
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--7.86%-- ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--6.36%-- ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.92% tar [kernel.kallsyms] [k] radix_tree_lookup_element
|
--- radix_tree_lookup_element
|
|--79.19%-- radix_tree_lookup_slot
| find_get_page
| |
| |--39.67%-- __find_get_block_slow
| | unmap_underlying_metadata
| | __block_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| |--37.78%-- generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --22.55%-- find_lock_page
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--13.79%-- radix_tree_lookup
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--7.02%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.86% tar [kernel.kallsyms] [k] ext4_mark_iloc_dirty
|
--- ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.84% tar [kernel.kallsyms] [k] __alloc_pages_nodemask
|
--- __alloc_pages_nodemask
|
|--88.77%-- alloc_pages_current
| __page_cache_alloc
| |
| |--53.12%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --46.88%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--11.23%-- __page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.80% tar [kernel.kallsyms] [k] get_parent_ip
|
--- get_parent_ip
|
|--27.09%-- add_preempt_count
| |
| |--44.58%-- jbd_lock_bh_state
| | do_get_write_access
| | jbd2_journal_get_write_access
| | __ext4_journal_get_write_access
| | ext4_reserve_inode_write
| | ext4_mark_inode_dirty
| | ext4_dirty_inode
| | __mark_inode_dirty
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| |--32.43%-- bit_spin_lock
| | lock_page_cgroup
| | __mem_cgroup_commit_charge.constprop.28
| | __mem_cgroup_commit_charge_lrucare
| | mem_cgroup_cache_charge
| | add_to_page_cache_locked
| | add_to_page_cache_lru
| | grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --22.99%-- _raw_spin_lock_irqsave
| __wake_up
| jbd2_journal_stop
| __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--26.54%-- sub_preempt_count
| |
| |--43.60%-- __percpu_counter_add
| | ext4_claim_free_blocks
| | ext4_da_get_block_prep
| | __block_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| |--29.86%-- bit_spin_unlock
| | unlock_page_cgroup
| | __mem_cgroup_commit_charge.constprop.28
| | __mem_cgroup_commit_charge_lrucare
| | mem_cgroup_cache_charge
| | add_to_page_cache_locked
| | add_to_page_cache_lru
| | mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --26.53%-- __lru_cache_add
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--13.83%-- radix_tree_preload
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--9.64%-- jbd_lock_bh_state
| jbd2_journal_dirty_metadata
| __ext4_handle_dirty_metadata
| ext4_mark_iloc_dirty
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--8.02%-- bit_spin_unlock
| jbd2_journal_add_journal_head
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--7.60%-- _raw_spin_lock
| inode_add_rsv_space
| __dquot_alloc_space
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--7.27%-- balance_dirty_pages_ratelimited_nr
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.74% tar [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
|
|--55.40%-- _raw_spin_lock_irq
| |
| |--86.22%-- __set_page_dirty
| | mark_buffer_dirty
| | __block_commit_write
| | block_write_end
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --13.78%-- blk_throtl_bio
| generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--44.60%-- _raw_spin_lock
|
|--76.80%-- inode_add_rsv_space
| __dquot_alloc_space
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--23.20%-- ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.71% tar [kernel.kallsyms] [k] put_page_testzero
|
--- put_page_testzero
put_page
|
|--49.42%-- generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--26.92%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--23.66%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.69% tar [kernel.kallsyms] [k] get_page_from_freelist
|
--- get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--63.44%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--36.56%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.67% tar [kernel.kallsyms] [k] __wake_up_bit
|
--- __wake_up_bit
|
|--90.71%-- unlock_page
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--9.29%-- wake_up_bit
unlock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.65% tar [kernel.kallsyms] [k] put_mems_allowed
|
--- put_mems_allowed
|
|--92.20%-- alloc_pages_current
| __page_cache_alloc
| |
| |--54.97%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --45.03%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--7.80%-- __page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.65% tar [kernel.kallsyms] [k] __rmqueue
|
--- __rmqueue
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--70.26%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--29.74%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.62% tar [kernel.kallsyms] [k] start_this_handle
|
--- start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
|
|--71.12%-- ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--28.88%-- ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.58% tar [kernel.kallsyms] [k] __rcu_read_unlock
|
--- __rcu_read_unlock
|
|--35.45%-- __prop_inc_single
| task_dirty_inc
| account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--14.77%-- find_get_page
| find_lock_page
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--11.81%-- __find_get_block_slow
| __find_get_block
| ext4_free_blocks
| ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--11.28%-- avc_has_perm_noaudit
| avc_has_perm
| inode_has_perm
| selinux_inode_permission
| security_inode_permission
| inode_permission
| may_delete
| vfs_unlink
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--10.21%-- __mem_cgroup_try_charge.constprop.27
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--8.34%-- generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--8.15%-- dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.58% tar [kernel.kallsyms] [k] __mark_inode_dirty
|
--- __mark_inode_dirty
|
|--54.06%-- __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--33.43%-- generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--12.51%-- ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.55% tar [kernel.kallsyms] [k] _raw_spin_unlock
|
--- _raw_spin_unlock
|
|--55.58%-- ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--44.42%-- create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.53% tar [kernel.kallsyms] [k] next_zones_zonelist
|
--- next_zones_zonelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--55.47%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--44.53%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.52% tar [kernel.kallsyms] [k] bit_spin_unlock
|
--- bit_spin_unlock
|
|--55.87%-- jbd2_journal_put_journal_head
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| |
| |--71.83%-- ext4_dirty_inode
| | __mark_inode_dirty
| | |
| | |--57.59%-- generic_write_end
| | | ext4_da_write_end
| | | generic_file_buffered_write
| | | __generic_file_aio_write
| | | generic_file_aio_write
| | | ext4_file_write
| | | do_sync_write
| | | vfs_write
| | | sys_write
| | | system_call_fastpath
| | | __GI___libc_write
| | |
| | --42.41%-- file_update_time
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --28.17%-- add_dirent_to_buf
| ext4_add_entry
| ext4_add_nondir
| ext4_create
| vfs_create
| do_last
| path_openat
| do_filp_open
| do_sys_open
| sys_openat
| system_call_fastpath
| __GI___openat
|
--44.13%-- jbd2_journal_add_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.51% tar [kernel.kallsyms] [k] jbd_unlock_bh_state
|
--- jbd_unlock_bh_state
|
|--56.56%-- do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| |
| |--85.42%-- ext4_reserve_inode_write
| | ext4_mark_inode_dirty
| | ext4_dirty_inode
| | __mark_inode_dirty
| | |
| | |--77.84%-- generic_write_end
| | | ext4_da_write_end
| | | generic_file_buffered_write
| | | __generic_file_aio_write
| | | generic_file_aio_write
| | | ext4_file_write
| | | do_sync_write
| | | vfs_write
| | | sys_write
| | | system_call_fastpath
| | | __GI___libc_write
| | |
| | --22.16%-- file_update_time
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --14.58%-- ext4_free_inode
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--43.44%-- jbd2_journal_dirty_metadata
__ext4_handle_dirty_metadata
|
|--53.02%-- ext4_free_inode
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--46.98%-- ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.50% tar [kernel.kallsyms] [k] __block_write_begin
|
--- __block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.49% tar [kernel.kallsyms] [k] __might_sleep
|
--- __might_sleep
|
|--30.54%-- jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--17.09%-- __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--14.20%-- __getblk
| __ext4_get_inode_loc
| ext4_get_inode_loc
| ext4_xattr_get
| ext4_xattr_security_get
| generic_getxattr
| cap_inode_need_killpriv
| security_inode_need_killpriv
| file_remove_suid
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--10.60%-- mutex_lock
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--10.58%-- kmem_cache_alloc
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--9.67%-- lock_buffer
| do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--7.31%-- ext4_map_blocks
ext4_getblk
ext4_bread
dx_probe
ext4_add_entry
ext4_add_nondir
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.49% tar [kernel.kallsyms] [k] jbd2_journal_add_journal_head
|
--- jbd2_journal_add_journal_head
|
|--69.06%-- __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| |
| |--77.62%-- generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --22.38%-- file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--30.94%-- jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.48% tar [kernel.kallsyms] [k] __mem_cgroup_commit_charge.constprop.28
|
--- __mem_cgroup_commit_charge.constprop.28
|
|--83.14%-- __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| |
| |--56.52%-- mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --43.48%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--16.86%-- mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.46% tar [kernel.kallsyms] [k] __rcu_read_lock
|
--- __rcu_read_lock
|
|--30.47%-- avc_has_perm_noaudit
| |
| |--54.86%-- task_has_capability
| | selinux_capable
| | security_capable
| | ns_capable
| | exec_permission
| | link_path_walk
| | path_lookupat
| | do_path_lookup
| | user_path_parent
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| --45.14%-- avc_has_perm
| may_create
| selinux_inode_create
| security_inode_create
| vfs_create
| do_last
| path_openat
| do_filp_open
| do_sys_open
| sys_openat
| system_call_fastpath
| __GI___openat
|
|--20.91%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--18.25%-- find_get_page
| find_lock_page
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--17.06%-- __account_system_time
| irqtime_account_process_tick
| account_process_tick
| update_process_times
| tick_sched_timer
| __run_hrtimer
| hrtimer_interrupt
| smp_apic_timer_interrupt
| apic_timer_interrupt
| rw_verify_area
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--13.31%-- __prop_inc_single
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.44% tar [kernel.kallsyms] [k] jbd2_journal_cancel_revoke
|
--- jbd2_journal_cancel_revoke
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.44% tar [kernel.kallsyms] [k] mark_page_accessed
|
--- mark_page_accessed
|
|--56.95%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--15.53%-- do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--14.54%-- __find_get_block
| __getblk
| __ext4_get_inode_loc
| ext4_get_inode_loc
| ext4_xattr_get
| ext4_xattr_security_get
| generic_getxattr
| cap_inode_need_killpriv
| security_inode_need_killpriv
| file_remove_suid
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--12.98%-- generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.44% tar [kernel.kallsyms] [k] put_mems_allowed
|
--- put_mems_allowed
|
|--55.41%-- alloc_pages_current
| __page_cache_alloc
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--44.59%-- __alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--68.42%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--31.58%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.43% tar [kernel.kallsyms] [k] in_lock_functions
|
--- in_lock_functions
|
|--76.43%-- get_parent_ip
| |
| |--80.00%-- add_preempt_count
| | |
| | |--39.87%-- balance_dirty_pages_ratelimited_nr
| | | generic_file_buffered_write
| | | __generic_file_aio_write
| | | generic_file_aio_write
| | | ext4_file_write
| | | do_sync_write
| | | vfs_write
| | | sys_write
| | | system_call_fastpath
| | | __GI___libc_write
| | |
| | |--30.07%-- radix_tree_preload
| | | add_to_page_cache_locked
| | | add_to_page_cache_lru
| | | grab_cache_page_write_begin
| | | ext4_da_write_begin
| | | generic_file_buffered_write
| | | __generic_file_aio_write
| | | generic_file_aio_write
| | | ext4_file_write
| | | do_sync_write
| | | vfs_write
| | | sys_write
| | | system_call_fastpath
| | | __GI___libc_write
| | |
| | --30.07%-- _raw_read_lock
| | start_this_handle
| | jbd2__journal_start
| | jbd2_journal_start
| | ext4_journal_start_sb
| | ext4_dirty_inode
| | __mark_inode_dirty
| | ext4_setattr
| | notify_change
| | chown_common
| | sys_fchown
| | system_call_fastpath
| | __GI___fchown
| |
| --20.00%-- sub_preempt_count
| avc_has_perm_noaudit
| avc_has_perm
| inode_has_perm
| dentry_has_perm
| selinux_inode_setattr
| security_inode_setattr
| notify_change
| chown_common
| sys_fchown
| system_call_fastpath
| __GI___fchown
|
|--12.38%-- add_preempt_count
| jbd_lock_bh_state
| do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--11.19%-- sub_preempt_count
_raw_spin_unlock_irqrestore
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.43% swapper [kernel.kallsyms] [k] ext4_end_bio
|
--- ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.42% tar [kernel.kallsyms] [k] ext4_get_group_desc
|
--- ext4_get_group_desc
|
|--88.05%-- __ext4_get_inode_loc
| ext4_get_inode_loc
| |
| |--64.96%-- ext4_reserve_inode_write
| | ext4_mark_inode_dirty
| | ext4_dirty_inode
| | __mark_inode_dirty
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --35.04%-- ext4_xattr_get
| ext4_xattr_security_get
| generic_getxattr
| cap_inode_need_killpriv
| security_inode_need_killpriv
| file_remove_suid
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--11.95%-- ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.42% tar [kernel.kallsyms] [k] __percpu_counter_add
|
--- __percpu_counter_add
|
|--39.75%-- ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--25.32%-- ext4_claim_free_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--19.28%-- ext4_free_inode
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--15.64%-- __add_bdi_stat
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.42% tar [kernel.kallsyms] [k] do_get_write_access
|
--- do_get_write_access
|
|--80.57%-- jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--19.43%-- __ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.42% tar [kernel.kallsyms] [k] generic_file_buffered_write
|
--- generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.41% tar [kernel.kallsyms] [k] jbd2_journal_stop
|
--- jbd2_journal_stop
__ext4_journal_stop
|
|--79.71%-- ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--20.29%-- ext4_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.41% tar [kernel.kallsyms] [k] jbd2__journal_start
|
--- jbd2__journal_start
|
|--82.99%-- jbd2_journal_start
| ext4_journal_start_sb
| |
| |--83.87%-- ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --16.13%-- ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--17.01%-- ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.40% tar [kernel.kallsyms] [k] do_mpage_readpage
|
--- do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.39% tar [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
__might_sleep
|
|--55.35%-- __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--61.18%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --38.82%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--28.25%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--16.40%-- __getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.38% tar [kernel.kallsyms] [k] alloc_pages_current
|
--- alloc_pages_current
|
|--82.52%-- __page_cache_alloc
| |
| |--74.42%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --25.58%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--17.48%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.37% tar [kernel.kallsyms] [k] __nr_to_section
|
--- __nr_to_section
lookup_page_cgroup
|
|--70.03%-- __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--17.38%-- mem_cgroup_get_reclaim_stat_from_page
| update_page_reclaim_stat
| ____pagevec_lru_add_fn
| pagevec_lru_move_fn
| ____pagevec_lru_add
| __lru_cache_add
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--12.60%-- mem_cgroup_del_lru_list
activate_page
mark_page_accessed
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.37% tar [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
|
|--52.73%-- _raw_spin_unlock_irqrestore
| |
| |--51.73%-- __wake_up
| | jbd2_journal_stop
| | __ext4_journal_stop
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --48.27%-- pagevec_lru_move_fn
| ____pagevec_lru_add
| __lru_cache_add
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.27%-- __wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.36% tar [kernel.kallsyms] [k] __mem_cgroup_try_charge.constprop.27
|
--- __mem_cgroup_try_charge.constprop.27
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--67.66%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--32.34%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.36% swapper [kernel.kallsyms] [k] unlock_page
|
--- unlock_page
|
|--85.03%-- mpage_end_io
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--14.97%-- bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.35% tar [kernel.kallsyms] [k] ext4_xattr_get
|
--- ext4_xattr_get
|
|--68.95%-- ext4_xattr_security_get
| generic_getxattr
| cap_inode_need_killpriv
| security_inode_need_killpriv
| file_remove_suid
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--31.05%-- generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.35% perf [kernel.kallsyms] [k] seq_puts
|
--- seq_puts
render_sigset_t
proc_pid_status
proc_single_show
seq_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.35% tar [kernel.kallsyms] [k] grab_cache_page_write_begin
|
--- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.34% tar [kernel.kallsyms] [k] alloc_page_buffers
|
--- alloc_page_buffers
|
|--72.20%-- create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--27.80%-- __block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.34% tar [kernel.kallsyms] [k] debug_smp_processor_id
|
--- debug_smp_processor_id
|
|--58.75%-- get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--70.45%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --29.55%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--41.25%-- __lru_cache_add
add_to_page_cache_lru
|
|--61.47%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--38.53%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.34% perf [kernel.kallsyms] [k] clear_page_c
|
--- clear_page_c
__alloc_pages_nodemask
perf_mmap_alloc_page
perf_mmap
mmap_region
do_mmap_pgoff
sys_mmap_pgoff
sys_mmap
system_call_fastpath
__mmap
0x418523
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.34% tar [kernel.kallsyms] [k] iov_iter_copy_from_user_atomic
|
--- iov_iter_copy_from_user_atomic
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.33% tar [kernel.kallsyms] [k] page_cache_get_speculative
|
--- page_cache_get_speculative
find_get_page
|
|--84.05%-- find_lock_page
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--15.95%-- generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.33% top [kernel.kallsyms] [k] follow_managed
|
--- follow_managed
walk_component
link_path_walk
path_openat
do_filp_open
do_sys_open
sys_open
system_call_fastpath
__GI___libc_open
0.33% flush-253:2 [kernel.kallsyms] [k] ext4_num_dirty_pages
|
--- ext4_num_dirty_pages
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.32% swapper [kernel.kallsyms] [k] do_timer
|
--- do_timer
tick_do_update_jiffies64
|
|--61.57%-- tick_sched_timer
| __run_hrtimer
| hrtimer_interrupt
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--38.43%-- tick_check_idle
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.32% tar [kernel.kallsyms] [k] __generic_file_aio_write
|
--- __generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.32% tar [kernel.kallsyms] [k] mem_cgroup_add_lru_list
|
--- mem_cgroup_add_lru_list
add_page_to_lru_list
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
|
|--87.33%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--12.67%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.32% tar [kernel.kallsyms] [k] ext4_da_write_end
|
--- ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.32% flush-253:2 [kernel.kallsyms] [k] ext4_bio_write_page
|
--- ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.32% swapper [kernel.kallsyms] [k] tick_nohz_stop_sched_tick
|
--- tick_nohz_stop_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.31% tar [kernel.kallsyms] [k] down_read
|
--- down_read
|
|--78.35%-- ext4_map_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--21.65%-- ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.31% tar [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
|
|--83.21%-- __wake_up
| jbd2_journal_stop
| __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--16.79%-- force_qs_rnp
force_quiescent_state
__rcu_process_callbacks
rcu_process_callbacks
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.30% tar [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--59.37%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--40.63%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.30% perf [kernel.kallsyms] [k] file_update_time
|
--- file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__write_nocancel
0.30% tar [kernel.kallsyms] [k] mark_buffer_dirty
|
--- mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.29% swapper [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--54.58%-- _raw_spin_lock_irqsave
| |
| |--57.58%-- scsi_device_unbusy
| | scsi_finish_command
| | scsi_softirq_done
| | blk_done_softirq
| | __do_softirq
| | call_softirq
| | do_softirq
| | irq_exit
| | do_IRQ
| | common_interrupt
| | cpuidle_idle_call
| | cpu_idle
| | rest_init
| | start_kernel
| | x86_64_start_reservations
| | x86_64_start_kernel
| |
| --42.42%-- test_clear_page_writeback
| end_page_writeback
| put_io_page
| ext4_end_bio
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--20.90%-- __percpu_counter_add
| __add_bdi_stat
| test_clear_page_writeback
| end_page_writeback
| put_io_page
| ext4_end_bio
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--15.95%-- irq_enter
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--8.56%-- scsi_device_unbusy
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.29% tar [kernel.kallsyms] [k] mnt_want_write_file
|
--- mnt_want_write_file
|
|--76.04%-- __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--23.96%-- system_call_fastpath
__GI___fchown
0.27% tar [kernel.kallsyms] [k] find_busiest_group
|
--- find_busiest_group
load_balance
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.26% perf [kernel.kallsyms] [k] perf_event_mmap_output
|
--- perf_event_mmap_output
perf_event_mmap_ctx
perf_event_mmap
mmap_region
do_mmap_pgoff
sys_mmap_pgoff
sys_mmap
system_call_fastpath
__mmap
0.26% swapper [kernel.kallsyms] [k] kmem_cache_free
|
--- kmem_cache_free
|
|--59.24%-- mempool_free
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--40.76%-- scsi_pool_free_command
__scsi_put_command
scsi_put_command
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.26% perf [kernel.kallsyms] [k] fsnotify
|
--- fsnotify
fsnotify_modify
vfs_write
sys_write
system_call_fastpath
__write_nocancel
0x4293b8
0x429c0a
0x418709
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.26% perf [kernel.kallsyms] [k] put_bh
|
--- put_bh
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__write_nocancel
0x4293b8
0x429c0a
0x418709
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.26% perf [kernel.kallsyms] [k] _raw_spin_lock
|
--- _raw_spin_lock
prepend_path
path_with_deleted
d_path
seq_path
show_map_vma
show_map
seq_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.26% tar [kernel.kallsyms] [k] get_page
|
--- get_page
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.26% tar [kernel.kallsyms] [k] selinux_file_permission
|
--- selinux_file_permission
security_file_permission
rw_verify_area
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.26% tar [kernel.kallsyms] [k] test_ti_thread_flag
|
--- test_ti_thread_flag
need_resched
should_resched
_cond_resched
|
|--64.24%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--35.76%-- generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.26% perf [kernel.kallsyms] [k] number
|
--- number
vsnprintf
seq_printf
show_map_vma
show_map
seq_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.25% tar libc-2.13.90.so [.] __GI___libc_write
|
--- __GI___libc_write
0.25% tar [kernel.kallsyms] [k] test_and_set_bit
|
--- test_and_set_bit
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.25% perf [kernel.kallsyms] [k] vsnprintf
|
--- vsnprintf
seq_printf
show_map_vma
show_map
seq_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.25% tar [kernel.kallsyms] [k] run_timer_softirq
|
--- run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.25% perf [kernel.kallsyms] [k] setup_object
|
--- setup_object
new_slab
__slab_alloc
kmem_cache_alloc
d_alloc
proc_fill_cache
proc_task_readdir
vfs_readdir
sys_getdents
system_call_fastpath
__getdents64
0x429beb
0x418709
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.24% swapper [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
|
|--40.64%-- __bit_spin_unlock.constprop.24
| __slab_free
| kmem_cache_free
| put_io_page
| ext4_end_bio
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--33.53%-- _raw_spin_unlock_irqrestore
| rcu_report_qs_rnp
| __rcu_process_callbacks
| rcu_process_callbacks
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--25.82%-- __slab_free
kmem_cache_free
mempool_free_slab
mempool_free
bio_free
bio_fs_destructor
bio_put
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.24% tar [kernel.kallsyms] [k] add_to_page_cache_locked
|
--- add_to_page_cache_locked
|
|--64.11%-- add_to_page_cache_lru
| |
| |--63.51%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --36.49%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--35.89%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.24% tar [kernel.kallsyms] [k] file_remove_suid
|
--- file_remove_suid
|
|--81.19%-- __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--18.81%-- generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.24% tar [kernel.kallsyms] [k] clear_bit_unlock
|
--- clear_bit_unlock
|
|--71.97%-- unlock_buffer
| do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| |
| |--58.50%-- file_update_time
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --41.50%-- generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--28.03%-- do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.23% tar [kernel.kallsyms] [k] sys_write
|
--- sys_write
system_call_fastpath
__GI___libc_write
0.23% swapper [kernel.kallsyms] [k] SetPageUptodate
|
--- SetPageUptodate
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.23% tar [kernel.kallsyms] [k] __bio_add_page.part.2
|
--- __bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.23% swapper [kernel.kallsyms] [k] rcu_irq_enter
|
--- rcu_irq_enter
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.23% tar [kernel.kallsyms] [k] _raw_spin_unlock_irq
|
--- _raw_spin_unlock_irq
|
|--45.40%-- generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--29.98%-- mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--24.62%-- add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.23% tar [kernel.kallsyms] [k] bit_spin_lock
|
--- bit_spin_lock
|
|--51.02%-- jbd2_journal_put_journal_head
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--48.98%-- jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.22% swapper [kernel.kallsyms] [k] nr_iowait_cpu
|
--- nr_iowait_cpu
tick_nohz_stop_idle
tick_check_idle
irq_enter
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.22% flush-253:2 [kernel.kallsyms] [k] page_waitqueue
|
--- page_waitqueue
|
|--51.36%-- unlock_page
| ext4_bio_write_page
| mpage_da_submit_io
| mpage_da_map_and_submit
| ext4_da_writepages
| do_writepages
| writeback_single_inode
| writeback_sb_inodes
| writeback_inodes_wb
| wb_writeback
| wb_do_writeback
| bdi_writeback_thread
| kthread
| kernel_thread_helper
|
--48.64%-- ext4_num_dirty_pages
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.22% tar [kernel.kallsyms] [k] __jbd2_log_space_left
|
--- __jbd2_log_space_left
start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
|
|--50.77%-- ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--49.23%-- ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.22% tar [kernel.kallsyms] [k] mb_test_bit
|
--- mb_test_bit
mb_free_blocks
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.22% tar [kernel.kallsyms] [k] __blk_recalc_rq_segments
|
--- __blk_recalc_rq_segments
blk_recount_segments
bio_phys_segments
blk_rq_bio_prep
init_request_from_bio
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.22% tar [kernel.kallsyms] [k] generic_write_end
|
--- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.22% tar [kernel.kallsyms] [k] jbd2_journal_dirty_metadata
|
--- jbd2_journal_dirty_metadata
|
|--62.74%-- __ext4_handle_dirty_metadata
| ext4_mark_iloc_dirty
| ext4_mark_inode_dirty
| |
| |--59.79%-- ext4_dirty_inode
| | __mark_inode_dirty
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --40.21%-- ext4_unlink
| vfs_unlink
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--37.26%-- ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.22% tar [kernel.kallsyms] [k] __mem_cgroup_threshold
|
--- __mem_cgroup_threshold
memcg_check_events
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.21% tar [kernel.kallsyms] [k] kfree
|
--- kfree
start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
|
|--79.14%-- ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--20.86%-- ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.21% tar [kernel.kallsyms] [k] ext4_get_inode_flags
|
--- ext4_get_inode_flags
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
|
|--63.71%-- file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--36.29%-- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.21% flush-253:2 [kernel.kallsyms] [k] put_page_testzero
|
--- put_page_testzero
release_pages
__pagevec_release
pagevec_release
|
|--57.93%-- ext4_num_dirty_pages
| ext4_da_writepages
| do_writepages
| writeback_single_inode
| writeback_sb_inodes
| writeback_inodes_wb
| wb_writeback
| wb_do_writeback
| bdi_writeback_thread
| kthread
| kernel_thread_helper
|
--42.07%-- write_cache_pages_da
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.21% flush-253:2 [kernel.kallsyms] [k] __rcu_read_unlock
|
--- __rcu_read_unlock
__find_get_block_slow
unmap_underlying_metadata
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.21% tar [kernel.kallsyms] [k] avtab_search
|
--- avtab_search
security_compute_sid
security_transition_sid
may_create
selinux_inode_create
security_inode_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.21% tar [kernel.kallsyms] [k] search_dirblock
|
--- search_dirblock
ext4_dx_find_entry
ext4_find_entry
ext4_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.21% tar [kernel.kallsyms] [k] __sg_alloc_table
|
--- __sg_alloc_table
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.21% tar [kernel.kallsyms] [k] ext4_journal_start_sb
|
--- ext4_journal_start_sb
ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.21% top libc-2.13.90.so [.] __GI_____strtoll_l_internal
|
--- __GI_____strtoll_l_internal
0.20% swapper [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
_raw_spin_lock
|
|--57.86%-- handle_edge_irq
| handle_irq
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--42.14%-- tick_do_update_jiffies64
tick_check_idle
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.20% tar [kernel.kallsyms] [k] generic_write_sync
|
--- generic_write_sync
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.20% tar [kernel.kallsyms] [k] put_page
|
--- put_page
|
|--64.65%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--35.35%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.20% flush-253:2 [kernel.kallsyms] [k] tag_get
|
--- tag_get
|
|--58.46%-- __lookup_tag
| radix_tree_gang_lookup_tag_slot
| find_get_pages_tag
| pagevec_lookup_tag
| ext4_num_dirty_pages
| ext4_da_writepages
| do_writepages
| writeback_single_inode
| writeback_sb_inodes
| writeback_inodes_wb
| wb_writeback
| wb_do_writeback
| bdi_writeback_thread
| kthread
| kernel_thread_helper
|
--41.54%-- test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.20% tar [kernel.kallsyms] [k] dm_merge_bvec
|
--- dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.19% tar [kernel.kallsyms] [k] __ext4_handle_dirty_metadata
|
--- __ext4_handle_dirty_metadata
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.19% tar [kernel.kallsyms] [k] __block_commit_write
|
--- __block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.19% tar [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore
|
--- _raw_spin_unlock_irqrestore
|
|--54.81%-- __wake_up
| jbd2_journal_stop
| __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--45.19%-- __delayacct_blkio_end
delayacct_blkio_end
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.19% tar [kernel.kallsyms] [k] ext4_writepage_trans_blocks
|
--- ext4_writepage_trans_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.19% tar [kernel.kallsyms] [k] jbd_lock_bh_state
|
--- jbd_lock_bh_state
|
|--62.13%-- do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--37.87%-- jbd2_journal_dirty_metadata
__ext4_handle_dirty_metadata
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.19% tar [kernel.kallsyms] [k] __list_add
|
--- __list_add
|
|--70.74%-- __make_request
| generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--29.26%-- expand
__rmqueue
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.19% tar [kernel.kallsyms] [k] fsnotify_modify
|
--- fsnotify_modify
|
|--74.13%-- vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--25.87%-- sys_write
system_call_fastpath
__GI___libc_write
0.19% flush-253:2 [kernel.kallsyms] [k] lock_page
|
--- lock_page
write_cache_pages_da
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.19% flush-253:2 [kernel.kallsyms] [k] rb_next
|
--- rb_next
set_next_entity
pick_next_task_fair
pick_next_task
schedule
preempt_schedule_irq
retint_kernel
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.19% tar [kernel.kallsyms] [k] ext4_ext_calc_metadata_amount
|
--- ext4_ext_calc_metadata_amount
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.18% swapper [kernel.kallsyms] [k] __wake_up_bit
|
--- __wake_up_bit
|
|--61.24%-- unlock_page
| mpage_end_io
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--38.76%-- end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.18% swapper [kernel.kallsyms] [k] select_nohz_load_balancer
|
--- select_nohz_load_balancer
|
|--75.59%-- tick_nohz_restart_sched_tick
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--24.41%-- tick_nohz_stop_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.18% swapper [kernel.kallsyms] [k] menu_select
|
--- menu_select
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.18% flush-253:2 [kernel.kallsyms] [k] page_cache_get_speculative
|
--- page_cache_get_speculative
find_get_pages
pagevec_lookup
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.18% tar [kernel.kallsyms] [k] generic_file_aio_read
|
--- generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.18% tar [kernel.kallsyms] [k] __prop_inc_single
|
--- __prop_inc_single
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.18% swapper [kernel.kallsyms] [k] page_index
|
--- page_index
test_clear_page_writeback
end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.17% tar [kernel.kallsyms] [k] arch_read_lock
|
--- arch_read_lock
_raw_read_lock
start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.17% tar [kernel.kallsyms] [k] mutex_unlock
|
--- mutex_unlock
|
|--73.48%-- generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--26.52%-- do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.17% tar [kernel.kallsyms] [k] blk_rq_map_sg
|
--- blk_rq_map_sg
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.17% tar [kernel.kallsyms] [k] bit_waitqueue
|
--- bit_waitqueue
|
|--51.20%-- wake_up_bit
| unlock_buffer
| do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--48.80%-- unlock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.17% tar [kernel.kallsyms] [k] SetPageLRU
|
--- SetPageLRU
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
|
|--71.47%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--28.53%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.17% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6
|
--- test_ti_thread_flag.constprop.6
|
|--64.90%-- _raw_spin_unlock_irq
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--35.10%-- inode_add_rsv_space
__dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.17% tar [kernel.kallsyms] [k] lookup_page_cgroup
|
--- lookup_page_cgroup
|
|--58.52%-- __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--41.48%-- mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.17% tar [kernel.kallsyms] [k] generic_make_request
|
--- generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.17% tar [kernel.kallsyms] [k] lock_page_cgroup
|
--- lock_page_cgroup
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.17% swapper [kernel.kallsyms] [k] mempool_free
|
--- mempool_free
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.17% tar [kernel.kallsyms] [k] put_prev_task_fair
|
--- put_prev_task_fair
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.17% tar [kernel.kallsyms] [k] perf_event_task_tick
|
--- perf_event_task_tick
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
|
|--51.07%-- scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--48.93%-- _ext4_get_block
ext4_get_block
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.17% tar [kernel.kallsyms] [k] arch_read_unlock
|
--- arch_read_unlock
|
|--66.07%-- _raw_read_unlock
| start_this_handle
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--33.93%-- _raw_read_unlock_irqrestore
dm_get_live_table
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.16% swapper [kernel.kallsyms] [k] debug_smp_processor_id
|
--- debug_smp_processor_id
|
|--38.89%-- __cycles_2_ns
| native_sched_clock
| sched_clock
| sched_clock_cpu
| account_system_vtime
| irq_exit
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--32.56%-- tick_check_idle
| irq_enter
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--14.27%-- account_idle_time
| irqtime_account_process_tick
| account_idle_ticks
| tick_nohz_restart_sched_tick
| cpu_idle
| start_secondary
|
--14.27%-- irqtime_account_process_tick
account_idle_ticks
tick_nohz_restart_sched_tick
cpu_idle
start_secondary
0.16% tar [kernel.kallsyms] [k] fget_light
|
--- fget_light
|
|--51.07%-- sys_read
| system_call_fastpath
| __GI___libc_read
|
--48.93%-- sys_write
system_call_fastpath
__GI___libc_write
0.16% tar [kernel.kallsyms] [k] SetPageUptodate
|
--- SetPageUptodate
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.16% tar [kernel.kallsyms] [k] __ext4_journal_stop
|
--- __ext4_journal_stop
|
|--50.22%-- ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--49.78%-- ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.16% tar [kernel.kallsyms] [k] ext4_clear_inode_state
|
--- ext4_clear_inode_state
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
|
|--51.89%-- file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--48.11%-- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.16% tar [kernel.kallsyms] [k] mem_cgroup_cache_charge
|
--- mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--65.55%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--34.45%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.16% tar [kernel.kallsyms] [k] ext4_map_blocks
|
--- ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.16% tar [kernel.kallsyms] [k] ata_dev_enabled
|
--- ata_dev_enabled
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.16% tar [kernel.kallsyms] [k] new_slab
|
--- new_slab
__slab_alloc
kmem_cache_alloc
alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.16% swapper [kernel.kallsyms] [k] hrtimer_cancel
|
--- hrtimer_cancel
tick_nohz_restart_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.15% tar [kernel.kallsyms] [k] lock_buffer
|
--- lock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.15% tar [kernel.kallsyms] [k] bit_spin_unlock
|
--- bit_spin_unlock
unlock_page_cgroup
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--72.67%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--27.33%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.15% tar [kernel.kallsyms] [k] up_read
|
--- up_read
|
|--62.56%-- ext4_map_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--37.44%-- ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.15% tar [kernel.kallsyms] [k] preempt_schedule
|
--- preempt_schedule
try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
_raw_read_lock
start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.15% tar [kernel.kallsyms] [k] fsnotify_access
|
--- fsnotify_access
sys_read
system_call_fastpath
__GI___libc_read
0.15% swapper [kernel.kallsyms] [k] native_read_tsc
|
--- native_read_tsc
paravirt_read_tsc
native_sched_clock
sched_clock
sched_clock_cpu
account_system_vtime
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.14% tar [kernel.kallsyms] [k] dequeue_task_fair
|
--- dequeue_task_fair
dequeue_task
deactivate_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.14% tar [kernel.kallsyms] [k] ata_scsi_queuecmd
|
--- ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.14% tar [kernel.kallsyms] [k] file_read_actor
|
--- file_read_actor
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.14% tar [kernel.kallsyms] [k] _raw_spin_lock
|
--- _raw_spin_lock
|
|--52.50%-- ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.50%-- ext4_ext_map_blocks
ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.14% top [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
_raw_spin_lock
nameidata_drop_rcu
link_path_walk
path_openat
do_filp_open
do_sys_open
sys_open
system_call_fastpath
__GI___libc_open
0.14% tar [kernel.kallsyms] [k] generic_file_aio_write
|
--- generic_file_aio_write
|
|--52.11%-- do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.89%-- ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.14% tar [kernel.kallsyms] [k] block_write_end
|
--- block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.14% tar [kernel.kallsyms] [k] hrtimer_interrupt
|
--- hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
|
|--52.67%-- jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.33%-- string_to_context_struct
security_context_to_sid_core
security_context_to_sid_default
inode_doinit_with_dentry
selinux_d_instantiate
security_d_instantiate
d_instantiate
d_splice_alias
ext4_lookup
d_alloc_and_lookup
__lookup_hash.part.3
lookup_hash
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.14% flush-253:2 [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
__percpu_counter_add
__add_bdi_stat
clear_page_dirty_for_io
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.14% swapper [kernel.kallsyms] [k] __rcu_read_lock
|
--- __rcu_read_lock
|
|--50.04%-- __prop_inc_percpu_max
| test_clear_page_writeback
| end_page_writeback
| put_io_page
| ext4_end_bio
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--49.96%-- __atomic_notifier_call_chain
atomic_notifier_call_chain
exit_idle
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.13% tar [kernel.kallsyms] [k] ktime_get_ts
|
--- ktime_get_ts
__delayacct_blkio_end
delayacct_blkio_end
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.13% tar [kernel.kallsyms] [k] page_zone
|
--- page_zone
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] __read_lock_failed
|
--- __read_lock_failed
_raw_read_lock
start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] prop_norm_single
|
--- prop_norm_single
__prop_inc_single
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] put_bh
|
--- put_bh
__brelse
brelse
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] radix_tree_insert
|
--- radix_tree_insert
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] policy_zonelist
|
--- policy_zonelist
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.13% tar [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% swapper [kernel.kallsyms] [k] irq_entries_start
|
--- irq_entries_start
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.13% top libproc-3.2.8.so [.] 0x44c2
|
--- 0x37d94044c2
0.13% flush-253:2 [kernel.kallsyms] [k] attempt_back_merge
|
--- attempt_back_merge
generic_make_request
submit_bio
ext4_io_submit
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.13% tar [kernel.kallsyms] [k] virt_to_head_page
|
--- virt_to_head_page
kmem_cache_free
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] mnt_clone_write
|
--- mnt_clone_write
mnt_want_write_file
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] rb_erase
|
--- rb_erase
deadline_remove_request
deadline_move_request
deadline_dispatch_requests
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.13% tar [kernel.kallsyms] [k] __set_page_dirty
|
--- __set_page_dirty
|
|--51.11%-- __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--48.89%-- mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] do_sync_write
|
--- do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
__find_get_block
__getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] arch_local_irq_disable
|
--- arch_local_irq_disable
_raw_spin_lock_irq
|
|--55.62%-- add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--44.38%-- __set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] kmem_cache_free
|
--- kmem_cache_free
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] ext4_da_write_begin
|
--- ext4_da_write_begin
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% swapper [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
|
|--69.63%-- account_system_vtime
| irq_exit
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--30.37%-- irqtime_account_process_tick
account_idle_ticks
tick_nohz_restart_sched_tick
cpu_idle
start_secondary
0.12% flush-253:2 [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.12% tar [kernel.kallsyms] [k] inode_reserved_space
|
--- inode_reserved_space
__dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% swapper [kernel.kallsyms] [k] update_cfs_shares
|
--- update_cfs_shares
enqueue_task_fair
enqueue_task
activate_task
try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.12% tar [kernel.kallsyms] [k] schedule
|
--- schedule
|
|--59.95%-- io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--40.05%-- __GI___libc_read
0.12% tar [kernel.kallsyms] [k] security_sid_to_context_force
|
--- security_sid_to_context_force
security_inode_init_security
ext4_init_security
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.12% tar [kernel.kallsyms] [k] radix_tree_preload
|
--- radix_tree_preload
add_to_page_cache_locked
add_to_page_cache_lru
|
|--54.07%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--45.93%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.12% swapper [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
_local_bh_enable
irq_enter
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.12% flush-253:2 [kernel.kallsyms] [k] release_pages
|
--- release_pages
__pagevec_release
pagevec_release
write_cache_pages_da
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.12% flush-253:2 [kernel.kallsyms] [k] page_mkclean
|
--- page_mkclean
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.12% tar [kernel.kallsyms] [k] set_bh_page
|
--- set_bh_page
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% tar [kernel.kallsyms] [k] account_entity_dequeue
|
--- account_entity_dequeue
dequeue_task
deactivate_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.12% tar [kernel.kallsyms] [k] ata_qc_issue
|
--- ata_qc_issue
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.12% tar libc-2.13.90.so [.] __cfree
|
--- __cfree
0.12% tar [kernel.kallsyms] [k] radix_tree_lookup_slot
|
--- radix_tree_lookup_slot
|
|--62.63%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--37.37%-- find_lock_page
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% top libc-2.13.90.so [.] _IO_vfscanf_internal
|
--- _IO_vfscanf_internal
_IO_vsscanf
0x7fffef292b20
0.12% tar [kernel.kallsyms] [k] ext4_inode_table
|
--- ext4_inode_table
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% tar [kernel.kallsyms] [k] xattr_resolve_name
|
--- xattr_resolve_name
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% tar [kernel.kallsyms] [k] add_to_page_cache_lru
|
--- add_to_page_cache_lru
|
|--67.10%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--32.90%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.12% migration/0 [kernel.kallsyms] [k] resched_task
|
--- resched_task
check_preempt_curr
__migrate_task
migration_cpu_stop
cpu_stopper_thread
kthread
kernel_thread_helper
0.12% tar [kernel.kallsyms] [k] __count_vm_events
|
--- __count_vm_events
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--50.90%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--49.10%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% tar [kernel.kallsyms] [k] system_call_fastpath
|
--- system_call_fastpath
__GI___libc_write
0.11% flush-253:2 [kernel.kallsyms] [k] __percpu_counter_add
|
--- __percpu_counter_add
__add_bdi_stat
clear_page_dirty_for_io
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.11% swapper [kernel.kallsyms] [k] page_waitqueue
|
--- page_waitqueue
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.11% kworker/0:2 [kernel.kallsyms] [k] worker_thread
|
--- worker_thread
kthread
kernel_thread_helper
0.11% tar [kernel.kallsyms] [k] fsnotify
|
--- fsnotify
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.11% tar [kernel.kallsyms] [k] is_handle_aborted
|
--- is_handle_aborted
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] blk_finish_plug
|
--- blk_finish_plug
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] arch_local_irq_save
|
--- arch_local_irq_save
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] jbd2_journal_get_write_access
|
--- jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] vfs_write
|
--- vfs_write
system_call_fastpath
__GI___libc_write
0.11% swapper [kernel.kallsyms] [k] irqtime_account_process_tick
|
--- irqtime_account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
start_secondary
0.11% tar [kernel.kallsyms] [k] blk_flush_plug_list
|
--- blk_flush_plug_list
blk_finish_plug
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.11% tar [kernel.kallsyms] [k] deadline_remove_request
|
--- deadline_remove_request
deadline_dispatch_requests
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.11% tar [kernel.kallsyms] [k] mb_find_buddy
|
--- mb_find_buddy
mb_free_blocks
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.11% tar [kernel.kallsyms] [k] sys_read
|
--- sys_read
|
|--58.45%-- system_call_fastpath
| __GI___libc_read
|
--41.55%-- __GI___libc_read
0.11% flush-253:2 [kernel.kallsyms] [k] arch_read_unlock
|
--- arch_read_unlock
_raw_read_unlock_irqrestore
dm_get_live_table
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.11% swapper [kernel.kallsyms] [k] __slab_free
|
--- __slab_free
kmem_cache_free
mempool_free_slab
mempool_free
scsi_sg_free
__sg_free_table
__scsi_release_buffers
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.11% tar [kernel.kallsyms] [k] sidtab_search_core
|
--- sidtab_search_core
sidtab_search
security_compute_sid
security_transition_sid
may_create
selinux_inode_create
security_inode_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.11% swapper [kernel.kallsyms] [k] switch_mm
|
--- switch_mm
schedule
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.11% flush-253:2 [kernel.kallsyms] [k] mpage_da_submit_io
|
--- mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.11% tar [kernel.kallsyms] [k] mem_cgroup_charge_statistics
|
--- mem_cgroup_charge_statistics
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--56.66%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--43.34%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.11% tar [kernel.kallsyms] [k] zone_watermark_ok
|
--- zone_watermark_ok
|
|--50.80%-- __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--49.20%-- get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] ext4_free_blocks
|
--- ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.11% tar [kernel.kallsyms] [k] ext4_ext_map_blocks
|
--- ext4_ext_map_blocks
ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] page_waitqueue
|
--- page_waitqueue
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] add_dirent_to_buf
|
--- add_dirent_to_buf
ext4_add_entry
ext4_add_nondir
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.11% tar [kernel.kallsyms] [k] strcmp
|
--- strcmp
symcmp
hashtab_search
string_to_context_struct
security_context_to_sid_core
security_context_to_sid_default
inode_doinit_with_dentry
selinux_d_instantiate
security_d_instantiate
d_instantiate
d_splice_alias
ext4_lookup
d_alloc_and_lookup
__lookup_hash.part.3
lookup_hash
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.11% tar [kernel.kallsyms] [k] select_idle_sibling
|
--- select_idle_sibling
select_task_rq_fair
select_task_rq
try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.11% flush-253:2 [kernel.kallsyms] [k] __wake_up_bit
|
--- __wake_up_bit
unlock_page
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.11% tar [kernel.kallsyms] [k] radix_tree_node_alloc
|
--- radix_tree_node_alloc
radix_tree_insert
add_to_page_cache_locked
add_to_page_cache_lru
|
|--60.14%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--39.86%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% swapper [kernel.kallsyms] [k] rcu_irq_exit
|
--- rcu_irq_exit
|
|--54.97%-- smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--45.03%-- irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.10% tar [kernel.kallsyms] [k] radix_tree_tag_set
|
--- radix_tree_tag_set
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% tar [kernel.kallsyms] [k] check_irqs_on
|
--- check_irqs_on
__getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% flush-253:2 [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
__add_bdi_stat
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% flush-253:2 [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
__might_sleep
unmap_underlying_metadata
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% tar [kernel.kallsyms] [k] irqtime_account_process_tick
|
--- irqtime_account_process_tick
account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% tar [kernel.kallsyms] [k] alloc_buffer_head
|
--- alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% tar [kernel.kallsyms] [k] jiffies_to_timeval
|
--- jiffies_to_timeval
__account_system_time
irqtime_account_process_tick
account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
__find_get_block
__getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.10% tar [kernel.kallsyms] [k] ext4_get_reserved_space
|
--- ext4_get_reserved_space
inode_add_rsv_space
__dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% swapper [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.10% tar [kernel.kallsyms] [k] blk_start_plug
|
--- blk_start_plug
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.10% flush-253:2 [kernel.kallsyms] [k] sg_init_table
|
--- sg_init_table
scsi_alloc_sgtable
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
__make_request
generic_make_request
submit_bio
ext4_io_submit
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% tar [kernel.kallsyms] [k] ata_qc_new_init
|
--- ata_qc_new_init
|
|--51.58%-- ata_scsi_queuecmd
| scsi_dispatch_cmd
| scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--48.42%-- __ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.10% tar [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
_raw_spin_unlock_irq
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.10% flush-253:2 [kernel.kallsyms] [k] inc_zone_page_state
|
--- inc_zone_page_state
account_page_writeback
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% tar [kernel.kallsyms] [k] system_call
|
--- system_call
__GI___libc_write
0.10% flush-253:2 [kernel.kallsyms] [k] __bio_clone
|
--- __bio_clone
clone_bio
__split_and_process_bio
dm_request
generic_make_request
submit_bio
ext4_io_submit
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% flush-253:2 [kernel.kallsyms] [k] mod_state
|
--- mod_state
inc_zone_page_state
account_page_writeback
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% tar [kernel.kallsyms] [k] arch_local_irq_save
|
--- arch_local_irq_save
_raw_spin_lock_irqsave
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% tar [kernel.kallsyms] [k] bio_get_nr_vecs
|
--- bio_get_nr_vecs
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.10% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_set
|
--- radix_tree_tag_set
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% tar [kernel.kallsyms] [k] test_ti_thread_flag
|
--- test_ti_thread_flag
__mem_cgroup_try_charge.constprop.27
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.10% flush-253:2 [kernel.kallsyms] [k] bit_spin_lock
|
--- bit_spin_lock
jbd2_journal_add_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.09% tar [kernel.kallsyms] [k] inode_init_once
|
--- inode_init_once
init_once
setup_object
new_slab
__slab_alloc
kmem_cache_alloc
ext4_alloc_inode
alloc_inode
new_inode
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.09% jbd2/dm-2-8 [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--61.31%-- _raw_spin_lock
| jbd2_journal_file_buffer
| jbd2_journal_write_metadata_buffer
| jbd2_journal_commit_transaction
| kjournald2
| kthread
| kernel_thread_helper
|
--38.69%-- jbd_trylock_bh_state
journal_clean_one_cp_list
__jbd2_journal_clean_checkpoint_list
jbd2_journal_commit_transaction
kjournald2
kthread
kernel_thread_helper
0.09% swapper [kernel.kallsyms] [k] kobject_put
|
--- kobject_put
__scsi_put_command
scsi_put_command
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.09% flush-253:2 [kernel.kallsyms] [k] __elv_add_request
|
--- __elv_add_request
__make_request
generic_make_request
submit_bio
ext4_io_submit
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.09% swapper [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
rcu_process_gp_end
__rcu_process_callbacks
rcu_process_callbacks
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.09% tar [kernel.kallsyms] [k] round_jiffies_up
|
--- round_jiffies_up
blk_start_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.09% tar [kernel.kallsyms] [k] attach_page_buffers
|
--- attach_page_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.09% tar [kernel.kallsyms] [k] hrtick_update
|
--- hrtick_update
dequeue_task
deactivate_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.09% tar [kernel.kallsyms] [k] ext4_clear_inode_flag
|
--- ext4_clear_inode_flag
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.09% jbd2/dm-2-8 [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
_raw_spin_lock_irq
blk_throtl_bio
generic_make_request
submit_bio
submit_bh
jbd2_journal_commit_transaction
kjournald2
kthread
kernel_thread_helper
0.09% top [kernel.kallsyms] [k] __rcu_read_lock
|
--- __rcu_read_lock
__d_lookup
walk_component
link_path_walk
path_openat
do_filp_open
do_sys_open
sys_open
system_call_fastpath
__GI___libc_open
0.09% tar [kernel.kallsyms] [k] __brelse
|
--- __brelse
brelse
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.09% flush-253:2 [kernel.kallsyms] [k] page_zone
|
--- page_zone
inc_zone_page_state
account_page_writeback
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.09% tar [kernel.kallsyms] [k] bio_integrity_enabled
|
--- bio_integrity_enabled
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.09% tar [kernel.kallsyms] [k] dm_table_find_target
|
--- dm_table_find_target
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.09% tar [kernel.kallsyms] [k] __mod_zone_page_state
|
--- __mod_zone_page_state
add_page_to_lru_list
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.09% tar [kernel.kallsyms] [k] ahci_fill_cmd_slot
|
--- ahci_fill_cmd_slot
ahci_qc_prep
ata_qc_issue
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.09% swapper [kernel.kallsyms] [k] blk_unprep_request
|
--- blk_unprep_request
blk_finish_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.09% top libc-2.13.90.so [.] __GI___close
|
--- __GI___close
0.09% tar [kernel.kallsyms] [k] put_bh
|
--- put_bh
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.09% swapper [kernel.kallsyms] [k] finish_task_switch
|
--- finish_task_switch
schedule
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.09% flush-253:2 [kernel.kallsyms] [k] page_mapping
|
--- page_mapping
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.09% tar [kernel.kallsyms] [k] alloc_tio
|
--- alloc_tio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_clear
|
--- radix_tree_tag_clear
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.08% tar [kernel.kallsyms] [k] timespec_trunc
|
--- timespec_trunc
touch_atime
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% tar [kernel.kallsyms] [k] ext4_claim_free_blocks
|
--- ext4_claim_free_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.08% tar libc-2.13.90.so [.] __GI___errno_location
|
--- __GI___errno_location
0.08% tar [kernel.kallsyms] [k] path_to_nameidata
|
--- path_to_nameidata
walk_component
link_path_walk
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.08% tar [kernel.kallsyms] [k] current_sid
|
--- current_sid
selinux_file_permission
security_file_permission
rw_verify_area
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% top [kernel.kallsyms] [k] path_openat
|
--- path_openat
do_filp_open
do_sys_open
sys_open
system_call_fastpath
__GI___libc_open
0.08% tar [kernel.kallsyms] [k] __do_page_cache_readahead
|
--- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% tar [kernel.kallsyms] [k] _cond_resched
|
--- _cond_resched
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% swapper [kernel.kallsyms] [k] run_timer_softirq
|
--- run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.08% swapper [kernel.kallsyms] [k] note_interrupt
|
--- note_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.08% tar [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.08% swapper [kernel.kallsyms] [k] native_apic_mem_write
|
--- native_apic_mem_write
apic_write
ack_APIC_irq
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
start_secondary
0.08% tar [kernel.kallsyms] [k] bio_data
|
--- bio_data
init_request_from_bio
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% tar [kernel.kallsyms] [k] string_to_context_struct
|
--- string_to_context_struct
security_context_to_sid_core
security_context_to_sid_default
inode_doinit_with_dentry
selinux_d_instantiate
security_d_instantiate
d_instantiate
d_splice_alias
ext4_lookup
d_alloc_and_lookup
__lookup_hash.part.3
lookup_hash
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.08% tar [kernel.kallsyms] [k] mb_clear_bit
|
--- mb_clear_bit
mb_clear_bits
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.08% tar [kernel.kallsyms] [k] ext4_mb_load_buddy
|
--- ext4_mb_load_buddy
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.08% top [kernel.kallsyms] [k] __fsnotify_parent
|
--- __fsnotify_parent
fsnotify_perm
security_file_permission
rw_verify_area
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% sleep [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
_raw_spin_lock
|
|--60.83%-- __do_fault
| handle_pte_fault
| handle_mm_fault
| do_page_fault
| page_fault
| 0x37d9082b80
|
--39.17%-- deny_write_access
mmap_region
do_mmap_pgoff
do_mmap
elf_map
load_elf_binary
search_binary_handler
do_execve
sys_execve
stub_execve
0x37d90ae097
0.08% swapper [kernel.kallsyms] [k] ahci_interrupt
|
--- ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.08% tar [kernel.kallsyms] [k] __srcu_read_unlock
|
--- __srcu_read_unlock
fsnotify_modify
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.08% tar [kernel.kallsyms] [k] signal_pending
|
--- signal_pending
fatal_signal_pending
__mem_cgroup_try_charge.constprop.27
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% tar [kernel.kallsyms] [k] __read_seqcount_begin
|
--- __read_seqcount_begin
path_init
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.08% tar [kernel.kallsyms] [k] test_ti_thread_flag
|
--- test_ti_thread_flag
mnt_clone_write
mnt_want_write_file
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.08% tar [kernel.kallsyms] [k] jbd2_journal_start
|
--- jbd2_journal_start
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] __srcu_read_lock
|
--- __srcu_read_lock
fsnotify
fsnotify_modify
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% swapper [kernel.kallsyms] [k] handle_irq
|
--- handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% swapper [kernel.kallsyms] [k] __ata_scsi_queuecmd
|
--- __ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
blk_run_queue
scsi_run_queue
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
drive_stat_acct
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] __switch_to
|
--- __switch_to
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] ext4_file_write
|
--- ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% swapper [kernel.kallsyms] [k] wake_bit_function
|
--- wake_bit_function
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] _raw_spin_lock_irq
|
--- _raw_spin_lock_irq
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] drive_stat_acct
|
--- drive_stat_acct
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.22
|
--- test_ti_thread_flag.constprop.22
zero_user_segments
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] sd_prep_fn
|
--- sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% top libc-2.13.90.so [.] __strchrnul
|
--- __strchrnul
___vsnprintf_chk
0x39335b1b6d5b1b42
0.07% swapper [kernel.kallsyms] [k] sd_prep_fn
|
--- sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
blk_run_queue
scsi_run_queue
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] try_to_wake_up
|
--- try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
_raw_spin_unlock_irqrestore
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] __bio_clone
|
--- __bio_clone
clone_bio
__split_and_process_bio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] __zone_watermark_ok
|
--- __zone_watermark_ok
zone_watermark_ok
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% swapper [kernel.kallsyms] [k] __remove_hrtimer
|
--- __remove_hrtimer
remove_hrtimer.part.4
hrtimer_try_to_cancel
hrtimer_cancel
tick_nohz_restart_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% swapper [kernel.kallsyms] [k] scsi_device_unbusy
|
--- scsi_device_unbusy
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] scsi_prep_return
|
--- scsi_prep_return
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] prepare_to_wait_exclusive
|
--- prepare_to_wait_exclusive
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] set_buffer_uptodate
|
--- set_buffer_uptodate
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] ata_get_xlat_func
|
--- ata_get_xlat_func
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] __find_get_block_slow
|
--- __find_get_block_slow
__find_get_block
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.07% tar [kernel.kallsyms] [k] generic_getxattr
|
--- generic_getxattr
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% swapper [kernel.kallsyms] [k] bit_spin_lock.constprop.22
|
--- bit_spin_lock.constprop.22
__slab_free
kmem_cache_free
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% swapper [kernel.kallsyms] [k] is_swiotlb_buffer
|
--- is_swiotlb_buffer
unmap_single
swiotlb_unmap_sg_attrs
ata_sg_clean
__ata_qc_complete
ata_qc_complete
ata_qc_complete_multiple
ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] native_read_tsc
|
--- native_read_tsc
paravirt_read_tsc
native_sched_clock
sched_clock
blk_rq_init
get_request
get_request_wait
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] ext4_has_free_blocks
|
--- ext4_has_free_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar libc-2.13.90.so [.] __GI___fxstat64
|
--- __GI___fxstat64
0.07% swapper [kernel.kallsyms] [k] perf_pmu_disable
|
--- perf_pmu_disable
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] sidtab_context_to_sid
|
--- sidtab_context_to_sid
security_context_to_sid_core
security_context_to_sid_default
inode_doinit_with_dentry
selinux_d_instantiate
security_d_instantiate
d_instantiate
d_splice_alias
ext4_lookup
d_alloc_and_lookup
__lookup_hash.part.3
lookup_hash
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.07% tar [kernel.kallsyms] [k] scsi_pool_alloc_command
|
--- scsi_pool_alloc_command
scsi_host_alloc_command
__scsi_get_command
scsi_get_command
scsi_get_cmd_from_req
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] list_del
|
--- list_del
activate_page
mark_page_accessed
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] pagefault_enable
|
--- pagefault_enable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] security_file_permission
|
--- security_file_permission
rw_verify_area
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% swapper [kernel.kallsyms] [k] scsi_next_command
|
--- scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] __phys_addr
|
--- __phys_addr
virt_to_head_page
kmem_cache_free
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] page_cache_async_readahead
|
--- page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] truncate_inode_pages_range
|
--- truncate_inode_pages_range
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.07% tar [kernel.kallsyms] [k] mem_cgroup_del_lru_list
|
--- mem_cgroup_del_lru_list
activate_page
mark_page_accessed
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] ext4_xattr_check_names
|
--- ext4_xattr_check_names
ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] arch_local_irq_enable
|
--- arch_local_irq_enable
__find_get_block
__getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] get_request
|
--- get_request
get_request_wait
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] swiotlb_map_sg_attrs
|
--- swiotlb_map_sg_attrs
ata_qc_issue
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] dm_get_live_table
|
--- dm_get_live_table
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] mem_cgroup_get_reclaim_stat_from_page
|
--- mem_cgroup_get_reclaim_stat_from_page
update_page_reclaim_stat
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% swapper [kernel.kallsyms] [k] check_preempt_curr
|
--- check_preempt_curr
ttwu_post_activation
try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] atomic_long_add
|
--- atomic_long_add
zone_page_state_add
__mod_zone_page_state
add_page_to_lru_list
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% swapper [kernel.kallsyms] [k] pm_qos_request
|
--- pm_qos_request
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] __kmalloc
|
--- __kmalloc
kzalloc.constprop.9
ext4_ext_find_extent
ext4_ext_map_blocks
ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% swapper [kernel.kallsyms] [k] rcu_bh_qs
|
--- rcu_bh_qs
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% swapper [kernel.kallsyms] [k] __blk_complete_request
|
--- __blk_complete_request
scsi_done
ata_scsi_qc_complete
__ata_qc_complete
ata_qc_complete
ata_qc_complete_multiple
ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] __d_lookup
|
--- __d_lookup
walk_component
link_path_walk
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.06% tar [kernel.kallsyms] [k] context_struct_to_string
|
--- context_struct_to_string
security_sid_to_context_core
security_sid_to_context_force
selinux_inode_init_security
security_inode_init_security
ext4_init_security
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.06% swapper [kernel.kallsyms] [k] tick_nohz_restart_sched_tick
|
--- tick_nohz_restart_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] max_sane_readahead
|
--- max_sane_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] rw_verify_area
|
--- rw_verify_area
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] ahci_qc_prep
|
--- ahci_qc_prep
ata_qc_issue
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] linear_map
|
--- linear_map
__split_and_process_bio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% swapper [kernel.kallsyms] [k] sched_clock
|
--- sched_clock
account_system_vtime
irq_enter
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] fput_light
|
--- fput_light
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] iov_iter_advance
|
--- iov_iter_advance
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] __queue_work
|
--- __queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
lock_page_cgroup
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% swapper [kernel.kallsyms] [k] scsi_io_completion
|
--- scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] activate_page
|
--- activate_page
mark_page_accessed
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] touch_atime
|
--- touch_atime
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] atomic_sub
|
--- atomic_sub
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] __inc_zone_page_state
|
--- __inc_zone_page_state
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] ext4_test_inode_state
|
--- ext4_test_inode_state
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] ext4_xattr_find_entry
|
--- ext4_xattr_find_entry
ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% swapper [kernel.kallsyms] [k] deadline_dispatch_requests
|
--- deadline_dispatch_requests
blk_peek_request
scsi_request_fn
__blk_run_queue
blk_run_queue
scsi_run_queue
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% swapper [kernel.kallsyms] [k] hrtick_update
|
--- hrtick_update
enqueue_task_fair
enqueue_task
activate_task
try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% swapper [kernel.kallsyms] [k] try_to_wake_up
|
--- try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% swapper [kernel.kallsyms] [k] pick_next_task_fair
|
--- pick_next_task_fair
schedule
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] pick_next_task_idle
|
--- pick_next_task_idle
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% swapper [kernel.kallsyms] [k] ktime_get
|
--- ktime_get
tick_check_idle
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] balance_dirty_pages_ratelimited_nr
|
--- balance_dirty_pages_ratelimited_nr
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% swapper [kernel.kallsyms] [k] mpage_end_io
|
--- mpage_end_io
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] radix_tree_preload_end
|
--- radix_tree_preload_end
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] submit_bio
|
--- submit_bio
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] __ata_scsi_queuecmd
|
--- __ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] dm_table_put
|
--- dm_table_put
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% sleep [kernel.kallsyms] [k] __ext4_journal_get_write_access
|
--- __ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
touch_atime
ext4_file_mmap
mmap_region
do_mmap_pgoff
sys_mmap_pgoff
sys_mmap
system_call_fastpath
__mmap
_nl_find_locale
__GI_setlocale
0x40124c
0.05% tar [kernel.kallsyms] [k] sysret_check
|
--- sysret_check
__GI___libc_read
0.05% swapper [kernel.kallsyms] [k] clockevents_program_event
|
--- clockevents_program_event
tick_dev_program_event
tick_program_event
__hrtimer_start_range_ns
hrtimer_start_range_ns
hrtimer_start_expires.constprop.1
tick_nohz_restart_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.05% tar [kernel.kallsyms] [k] pick_next_task_rt
|
--- pick_next_task_rt
pick_next_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% swapper [kernel.kallsyms] [k] account_entity_enqueue
|
--- account_entity_enqueue
enqueue_task
activate_task
try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.05% tar [kernel.kallsyms] [k] need_resched
|
--- need_resched
_cond_resched
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] find_lock_page
|
--- find_lock_page
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] memcmp
|
--- memcmp
search_dirblock
ext4_dx_find_entry
ext4_find_entry
ext4_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.05% tar [kernel.kallsyms] [k] generic_segment_checks
|
--- generic_segment_checks
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% tar [kernel.kallsyms] [k] update_vsyscall
|
--- update_vsyscall
do_timer
tick_do_update_jiffies64
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
_raw_spin_lock
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% swapper [kernel.kallsyms] [k] hrtimer_start_expires.constprop.1
|
--- hrtimer_start_expires.constprop.1
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.05% tar [kernel.kallsyms] [k] __page_cache_alloc
|
--- __page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] avc_has_perm_noaudit
|
--- avc_has_perm_noaudit
avc_has_perm
inode_has_perm
selinux_inode_permission
security_inode_exec_permission
exec_permission
link_path_walk
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.05% tar [kernel.kallsyms] [k] deactivate_slab
|
--- deactivate_slab
__slab_alloc
kmem_cache_alloc
alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% swapper [kernel.kallsyms] [k] bio_free
|
--- bio_free
bio_fs_destructor
bio_put
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.05% tar libc-2.13.90.so [.] __GI___libc_read
|
--- __GI___libc_read
0.05% tar [kernel.kallsyms] [k] selinux_capable
|
--- selinux_capable
ns_capable
capable
inode_change_ok
ext4_setattr
notify_change
chown_common
sys_fchown
system_call_fastpath
__GI___fchown
0.05% tar [kernel.kallsyms] [k] _raw_read_lock_irqsave
|
--- _raw_read_lock_irqsave
dm_get_live_table
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% swapper [kernel.kallsyms] [k] scsi_handle_queue_ramp_up
|
--- scsi_handle_queue_ramp_up
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.05% tar [kernel.kallsyms] [k] dm_table_any_congested
|
--- dm_table_any_congested
dm_any_congested
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% tar [kernel.kallsyms] [k] __wake_up_common
|
--- __wake_up_common
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] acct_update_integrals
|
--- acct_update_integrals
__account_system_time
irqtime_account_process_tick
account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
blk_finish_plug
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% swapper [kernel.kallsyms] [k] ns_to_timespec
|
--- ns_to_timespec
intel_idle
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.05% tar [kernel.kallsyms] [k] _raw_read_unlock_irqrestore
|
--- _raw_read_unlock_irqrestore
dm_get_live_table
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% tar [kernel.kallsyms] [k] __add_bdi_stat
|
--- __add_bdi_stat
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] sg_init_table
|
--- sg_init_table
scsi_alloc_sgtable
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% swapper [kernel.kallsyms] [k] getnstimeofday
|
--- getnstimeofday
ktime_get_real
intel_idle
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.04% tar [kernel.kallsyms] [k] get_page
|
--- get_page
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% tar [kernel.kallsyms] [k] timekeeping_get_ns
|
--- timekeeping_get_ns
delayacct_end
__delayacct_blkio_end
delayacct_blkio_end
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% tar [kernel.kallsyms] [k] trylock_page
|
--- trylock_page
lock_page
find_lock_page
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.04% swapper [kernel.kallsyms] [k] native_sched_clock
|
--- native_sched_clock
sched_clock_cpu
account_system_vtime
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
start_secondary
0.04% tar [kernel.kallsyms] [k] native_load_tls
|
--- native_load_tls
__GI___libc_read
0.04% tar [kernel.kallsyms] [k] files_lglock_local_unlock_cpu
|
--- files_lglock_local_unlock_cpu
file_sb_list_del
fput
filp_close
sys_close
system_call_fastpath
__GI___close
0.04% tar [kernel.kallsyms] [k] release_pages
|
--- release_pages
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.04% jbd2/dm-2-8 [kernel.kallsyms] [k] bit_spin_lock.constprop.22
|
--- bit_spin_lock.constprop.22
__slab_free
kmem_cache_free
__journal_remove_journal_head
jbd2_journal_remove_journal_head
journal_clean_one_cp_list
__jbd2_journal_clean_checkpoint_list
jbd2_journal_commit_transaction
kjournald2
kthread
kernel_thread_helper
0.04% tar [kernel.kallsyms] [k] pick_next_task_fair
|
--- pick_next_task_fair
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% swapper [kernel.kallsyms] [k] ktime_get_real
|
--- ktime_get_real
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.04% tar [kernel.kallsyms] [k] tick_do_update_jiffies64
|
--- tick_do_update_jiffies64
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.03% swapper [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
arch_local_irq_save
irqtime_account_process_tick
account_idle_ticks
tick_nohz_restart_sched_tick
cpu_idle
start_secondary
0.03% sleep [kernel.kallsyms] [k] mem_cgroup_uncharge_start
|
--- mem_cgroup_uncharge_start
exit_mmap
mmput
flush_old_exec
load_elf_binary
search_binary_handler
do_execve
sys_execve
stub_execve
__execve
0.02% swapper [kernel.kallsyms] [k] irq_enter
|
--- irq_enter
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
start_secondary
#
# (For a higher level overview, try: perf report --sort comm,dso)
#
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 16:50 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 16:50 UTC (permalink / raw)
To: Mel Gorman
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
This is the output of perf record -g -a -f sleep 5
(hopefully the list won't choke)
James
---
# Events: 1K cycles
#
# Overhead Command Shared Object Symbol
# ........ ........... ................. .......................................
#
9.03% tar [kernel.kallsyms] [k] copy_user_generic_string
|
--- copy_user_generic_string
|
|--60.55%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--39.45%-- generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
3.45% swapper [kernel.kallsyms] [k] intel_idle
|
--- intel_idle
cpuidle_idle_call
cpu_idle
|
|--76.47%-- rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--23.53%-- start_secondary
3.39% tar [kernel.kallsyms] [k] zero_user_segments
|
--- zero_user_segments
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
1.78% tar [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--16.72%-- __percpu_counter_add
| |
| |--59.73%-- __prop_inc_single
| | task_dirty_inc
| | account_page_dirtied
| | __set_page_dirty
| | mark_buffer_dirty
| | __block_commit_write
| | block_write_end
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --40.27%-- ext4_claim_free_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--14.83%-- _raw_spin_lock_irq
| |
| |--63.66%-- add_to_page_cache_locked
| | add_to_page_cache_lru
| | |
| | |--51.51%-- mpage_readpages
| | | ext4_readpages
| | | __do_page_cache_readahead
| | | ra_submit
| | | ondemand_readahead
| | | page_cache_async_readahead
| | | generic_file_aio_read
| | | do_sync_read
| | | vfs_read
| | | sys_read
| | | system_call_fastpath
| | | __GI___libc_read
| | |
| | --48.49%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --36.34%-- blk_throtl_bio
| generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--13.85%-- bit_spin_lock
| jbd2_journal_add_journal_head
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--12.19%-- __lru_cache_add
| add_to_page_cache_lru
| |
| |--70.86%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --29.14%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--8.30%-- bit_spin_lock
| lock_page_cgroup
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| |
| |--58.88%-- mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --41.12%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--7.17%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.90%-- __srcu_read_lock
| fsnotify
| fsnotify_modify
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.61%-- __srcu_read_unlock
| fsnotify
| fsnotify_modify
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.14%-- _raw_read_lock_irqsave
| dm_get_live_table
| dm_merge_bvec
| __bio_add_page.part.2
| bio_add_page
| do_mpage_readpage
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--3.81%-- _raw_read_lock
| start_this_handle
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--3.77%-- balance_dirty_pages_ratelimited_nr
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--3.55%-- __add_bdi_stat
| account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--2.18%-- _raw_spin_lock
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
1.70% tar [kernel.kallsyms] [k] __list_del_entry
|
--- __list_del_entry
|
|--72.43%-- list_del
| |
| |--76.01%-- __rmqueue
| | get_page_from_freelist
| | __alloc_pages_nodemask
| | alloc_pages_current
| | |
| | |--92.23%-- __page_cache_alloc
| | | |
| | | |--60.54%-- __do_page_cache_readahead
| | | | ra_submit
| | | | ondemand_readahead
| | | | page_cache_async_readahead
| | | | generic_file_aio_read
| | | | do_sync_read
| | | | vfs_read
| | | | sys_read
| | | | system_call_fastpath
| | | | __GI___libc_read
| | | |
| | | --39.46%-- grab_cache_page_write_begin
| | | ext4_da_write_begin
| | | generic_file_buffered_write
| | | __generic_file_aio_write
| | | generic_file_aio_write
| | | ext4_file_write
| | | do_sync_write
| | | vfs_write
| | | sys_write
| | | system_call_fastpath
| | | __GI___libc_write
| | |
| | --7.77%-- alloc_slab_page
| | new_slab
| | __slab_alloc
| | kmem_cache_alloc
| | alloc_buffer_head
| | alloc_page_buffers
| | create_empty_buffers
| | __block_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| |--18.55%-- get_page_from_freelist
| | __alloc_pages_nodemask
| | alloc_pages_current
| | __page_cache_alloc
| | |
| | |--60.32%-- __do_page_cache_readahead
| | | ra_submit
| | | ondemand_readahead
| | | page_cache_async_readahead
| | | generic_file_aio_read
| | | do_sync_read
| | | vfs_read
| | | sys_read
| | | system_call_fastpath
| | | __GI___libc_read
| | |
| | --39.68%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --5.44%-- activate_page
| mark_page_accessed
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--17.37%-- __rmqueue
| get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--63.38%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --36.62%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--10.20%-- get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--65.70%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--34.30%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
1.53% tar tar [.] 0x75ff
|
--- 0x407546
|
--- 0x42a9c2
|
--- 0x40753f
|
--- 0x40e790
|
--- 0x407586
|
--- 0x40fe1a
|
--- 0x424b50
|
--- 0x412972
|
--- 0x407501
|
--- 0x433336
|
--- 0x4075ff
|
--- 0x403b40
|
--- 0x4055c7
|
--- 0x4332d9
|
--- 0x405efa
|
--- 0x42a9fe
|
--- 0x405a43
|
--- 0x4332d6
1.44% tar [kernel.kallsyms] [k] __ext4_get_inode_loc
|
--- __ext4_get_inode_loc
ext4_get_inode_loc
|
|--62.07%-- ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| |
| |--94.67%-- generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --5.33%-- file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--37.93%-- ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
1.43% tar [kernel.kallsyms] [k] kmem_cache_alloc
|
--- kmem_cache_alloc
|
|--66.77%-- alloc_buffer_head
| alloc_page_buffers
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--22.22%-- jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| |
| |--65.27%-- ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --34.73%-- ext4_dirty_inode
| __mark_inode_dirty
| file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--6.29%-- radix_tree_preload
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--4.72%-- scsi_pool_alloc_command
scsi_host_alloc_command
__scsi_get_command
scsi_get_command
scsi_get_cmd_from_req
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
1.25% tar [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
|
|--24.47%-- _raw_spin_unlock_irqrestore
| |
| |--30.14%-- scsi_get_command
| | scsi_get_cmd_from_req
| | scsi_setup_fs_cmnd
| | sd_prep_fn
| | blk_peek_request
| | scsi_request_fn
| | __blk_run_queue
| | queue_unplugged
| | blk_flush_plug_list
| | io_schedule
| | sleep_on_page_killable
| | __wait_on_bit_lock
| | __lock_page_killable
| | lock_page_killable
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| |--28.74%-- prepare_to_wait_exclusive
| | __wait_on_bit_lock
| | __lock_page_killable
| | lock_page_killable
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| |--23.48%-- __wake_up
| | jbd2_journal_stop
| | __ext4_journal_stop
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --17.65%-- blkiocg_update_dispatch_stats
| throtl_charge_bio
| blk_throtl_bio
| generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--12.68%-- radix_tree_preload_end
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--12.51%-- pagefault_enable
| iov_iter_copy_from_user_atomic
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--10.90%-- _raw_spin_unlock_irq
| |
| |--52.31%-- add_to_page_cache_locked
| | add_to_page_cache_lru
| | mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --47.69%-- __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--10.37%-- mem_cgroup_charge_statistics
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--9.99%-- _raw_spin_unlock
| |
| |--60.73%-- inode_add_rsv_space
| | __dquot_alloc_space
| | ext4_da_get_block_prep
| | __block_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --39.27%-- ext4_ext_map_blocks
| ext4_map_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--5.89%-- __wake_up
| jbd2_journal_stop
| __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.86%-- balance_dirty_pages_ratelimited_nr
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.52%-- add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--3.81%-- avc_has_perm_noaudit
task_has_capability
selinux_capable
security_capable
ns_capable
generic_permission
inode_permission
may_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
1.24% tar [kernel.kallsyms] [k] bit_spin_lock
|
--- bit_spin_lock
|
|--94.43%-- lock_page_cgroup
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| |
| |--56.27%-- mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --43.73%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--5.57%-- __mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
1.13% swapper [kernel.kallsyms] [k] poll_idle
|
--- poll_idle
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
1.00% tar [kernel.kallsyms] [k] ext4_da_get_block_prep
|
--- ext4_da_get_block_prep
|
|--82.10%-- __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--17.90%-- ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.95% tar [kernel.kallsyms] [k] __find_get_block
|
--- __find_get_block
|
|--85.78%-- __getblk
| __ext4_get_inode_loc
| ext4_get_inode_loc
| |
| |--84.94%-- ext4_reserve_inode_write
| | ext4_mark_inode_dirty
| | ext4_dirty_inode
| | __mark_inode_dirty
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --15.06%-- ext4_xattr_get
| ext4_xattr_security_get
| generic_getxattr
| cap_inode_need_killpriv
| security_inode_need_killpriv
| file_remove_suid
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--7.86%-- ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--6.36%-- ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.92% tar [kernel.kallsyms] [k] radix_tree_lookup_element
|
--- radix_tree_lookup_element
|
|--79.19%-- radix_tree_lookup_slot
| find_get_page
| |
| |--39.67%-- __find_get_block_slow
| | unmap_underlying_metadata
| | __block_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| |--37.78%-- generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --22.55%-- find_lock_page
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--13.79%-- radix_tree_lookup
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--7.02%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.86% tar [kernel.kallsyms] [k] ext4_mark_iloc_dirty
|
--- ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.84% tar [kernel.kallsyms] [k] __alloc_pages_nodemask
|
--- __alloc_pages_nodemask
|
|--88.77%-- alloc_pages_current
| __page_cache_alloc
| |
| |--53.12%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --46.88%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--11.23%-- __page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.80% tar [kernel.kallsyms] [k] get_parent_ip
|
--- get_parent_ip
|
|--27.09%-- add_preempt_count
| |
| |--44.58%-- jbd_lock_bh_state
| | do_get_write_access
| | jbd2_journal_get_write_access
| | __ext4_journal_get_write_access
| | ext4_reserve_inode_write
| | ext4_mark_inode_dirty
| | ext4_dirty_inode
| | __mark_inode_dirty
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| |--32.43%-- bit_spin_lock
| | lock_page_cgroup
| | __mem_cgroup_commit_charge.constprop.28
| | __mem_cgroup_commit_charge_lrucare
| | mem_cgroup_cache_charge
| | add_to_page_cache_locked
| | add_to_page_cache_lru
| | grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --22.99%-- _raw_spin_lock_irqsave
| __wake_up
| jbd2_journal_stop
| __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--26.54%-- sub_preempt_count
| |
| |--43.60%-- __percpu_counter_add
| | ext4_claim_free_blocks
| | ext4_da_get_block_prep
| | __block_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| |--29.86%-- bit_spin_unlock
| | unlock_page_cgroup
| | __mem_cgroup_commit_charge.constprop.28
| | __mem_cgroup_commit_charge_lrucare
| | mem_cgroup_cache_charge
| | add_to_page_cache_locked
| | add_to_page_cache_lru
| | mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --26.53%-- __lru_cache_add
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--13.83%-- radix_tree_preload
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--9.64%-- jbd_lock_bh_state
| jbd2_journal_dirty_metadata
| __ext4_handle_dirty_metadata
| ext4_mark_iloc_dirty
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--8.02%-- bit_spin_unlock
| jbd2_journal_add_journal_head
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--7.60%-- _raw_spin_lock
| inode_add_rsv_space
| __dquot_alloc_space
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--7.27%-- balance_dirty_pages_ratelimited_nr
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.74% tar [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
|
|--55.40%-- _raw_spin_lock_irq
| |
| |--86.22%-- __set_page_dirty
| | mark_buffer_dirty
| | __block_commit_write
| | block_write_end
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --13.78%-- blk_throtl_bio
| generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--44.60%-- _raw_spin_lock
|
|--76.80%-- inode_add_rsv_space
| __dquot_alloc_space
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--23.20%-- ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.71% tar [kernel.kallsyms] [k] put_page_testzero
|
--- put_page_testzero
put_page
|
|--49.42%-- generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--26.92%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--23.66%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.69% tar [kernel.kallsyms] [k] get_page_from_freelist
|
--- get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--63.44%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--36.56%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.67% tar [kernel.kallsyms] [k] __wake_up_bit
|
--- __wake_up_bit
|
|--90.71%-- unlock_page
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--9.29%-- wake_up_bit
unlock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.65% tar [kernel.kallsyms] [k] put_mems_allowed
|
--- put_mems_allowed
|
|--92.20%-- alloc_pages_current
| __page_cache_alloc
| |
| |--54.97%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --45.03%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--7.80%-- __page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.65% tar [kernel.kallsyms] [k] __rmqueue
|
--- __rmqueue
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--70.26%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--29.74%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.62% tar [kernel.kallsyms] [k] start_this_handle
|
--- start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
|
|--71.12%-- ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--28.88%-- ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.58% tar [kernel.kallsyms] [k] __rcu_read_unlock
|
--- __rcu_read_unlock
|
|--35.45%-- __prop_inc_single
| task_dirty_inc
| account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--14.77%-- find_get_page
| find_lock_page
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--11.81%-- __find_get_block_slow
| __find_get_block
| ext4_free_blocks
| ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--11.28%-- avc_has_perm_noaudit
| avc_has_perm
| inode_has_perm
| selinux_inode_permission
| security_inode_permission
| inode_permission
| may_delete
| vfs_unlink
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--10.21%-- __mem_cgroup_try_charge.constprop.27
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--8.34%-- generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--8.15%-- dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.58% tar [kernel.kallsyms] [k] __mark_inode_dirty
|
--- __mark_inode_dirty
|
|--54.06%-- __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--33.43%-- generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--12.51%-- ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.55% tar [kernel.kallsyms] [k] _raw_spin_unlock
|
--- _raw_spin_unlock
|
|--55.58%-- ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--44.42%-- create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.53% tar [kernel.kallsyms] [k] next_zones_zonelist
|
--- next_zones_zonelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--55.47%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--44.53%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.52% tar [kernel.kallsyms] [k] bit_spin_unlock
|
--- bit_spin_unlock
|
|--55.87%-- jbd2_journal_put_journal_head
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| |
| |--71.83%-- ext4_dirty_inode
| | __mark_inode_dirty
| | |
| | |--57.59%-- generic_write_end
| | | ext4_da_write_end
| | | generic_file_buffered_write
| | | __generic_file_aio_write
| | | generic_file_aio_write
| | | ext4_file_write
| | | do_sync_write
| | | vfs_write
| | | sys_write
| | | system_call_fastpath
| | | __GI___libc_write
| | |
| | --42.41%-- file_update_time
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --28.17%-- add_dirent_to_buf
| ext4_add_entry
| ext4_add_nondir
| ext4_create
| vfs_create
| do_last
| path_openat
| do_filp_open
| do_sys_open
| sys_openat
| system_call_fastpath
| __GI___openat
|
--44.13%-- jbd2_journal_add_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.51% tar [kernel.kallsyms] [k] jbd_unlock_bh_state
|
--- jbd_unlock_bh_state
|
|--56.56%-- do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| |
| |--85.42%-- ext4_reserve_inode_write
| | ext4_mark_inode_dirty
| | ext4_dirty_inode
| | __mark_inode_dirty
| | |
| | |--77.84%-- generic_write_end
| | | ext4_da_write_end
| | | generic_file_buffered_write
| | | __generic_file_aio_write
| | | generic_file_aio_write
| | | ext4_file_write
| | | do_sync_write
| | | vfs_write
| | | sys_write
| | | system_call_fastpath
| | | __GI___libc_write
| | |
| | --22.16%-- file_update_time
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --14.58%-- ext4_free_inode
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--43.44%-- jbd2_journal_dirty_metadata
__ext4_handle_dirty_metadata
|
|--53.02%-- ext4_free_inode
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--46.98%-- ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.50% tar [kernel.kallsyms] [k] __block_write_begin
|
--- __block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.49% tar [kernel.kallsyms] [k] __might_sleep
|
--- __might_sleep
|
|--30.54%-- jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--17.09%-- __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--14.20%-- __getblk
| __ext4_get_inode_loc
| ext4_get_inode_loc
| ext4_xattr_get
| ext4_xattr_security_get
| generic_getxattr
| cap_inode_need_killpriv
| security_inode_need_killpriv
| file_remove_suid
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--10.60%-- mutex_lock
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--10.58%-- kmem_cache_alloc
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--9.67%-- lock_buffer
| do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--7.31%-- ext4_map_blocks
ext4_getblk
ext4_bread
dx_probe
ext4_add_entry
ext4_add_nondir
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.49% tar [kernel.kallsyms] [k] jbd2_journal_add_journal_head
|
--- jbd2_journal_add_journal_head
|
|--69.06%-- __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| |
| |--77.62%-- generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --22.38%-- file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--30.94%-- jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.48% tar [kernel.kallsyms] [k] __mem_cgroup_commit_charge.constprop.28
|
--- __mem_cgroup_commit_charge.constprop.28
|
|--83.14%-- __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| |
| |--56.52%-- mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --43.48%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--16.86%-- mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.46% tar [kernel.kallsyms] [k] __rcu_read_lock
|
--- __rcu_read_lock
|
|--30.47%-- avc_has_perm_noaudit
| |
| |--54.86%-- task_has_capability
| | selinux_capable
| | security_capable
| | ns_capable
| | exec_permission
| | link_path_walk
| | path_lookupat
| | do_path_lookup
| | user_path_parent
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| --45.14%-- avc_has_perm
| may_create
| selinux_inode_create
| security_inode_create
| vfs_create
| do_last
| path_openat
| do_filp_open
| do_sys_open
| sys_openat
| system_call_fastpath
| __GI___openat
|
|--20.91%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--18.25%-- find_get_page
| find_lock_page
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--17.06%-- __account_system_time
| irqtime_account_process_tick
| account_process_tick
| update_process_times
| tick_sched_timer
| __run_hrtimer
| hrtimer_interrupt
| smp_apic_timer_interrupt
| apic_timer_interrupt
| rw_verify_area
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--13.31%-- __prop_inc_single
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.44% tar [kernel.kallsyms] [k] jbd2_journal_cancel_revoke
|
--- jbd2_journal_cancel_revoke
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.44% tar [kernel.kallsyms] [k] mark_page_accessed
|
--- mark_page_accessed
|
|--56.95%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--15.53%-- do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--14.54%-- __find_get_block
| __getblk
| __ext4_get_inode_loc
| ext4_get_inode_loc
| ext4_xattr_get
| ext4_xattr_security_get
| generic_getxattr
| cap_inode_need_killpriv
| security_inode_need_killpriv
| file_remove_suid
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--12.98%-- generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.44% tar [kernel.kallsyms] [k] put_mems_allowed
|
--- put_mems_allowed
|
|--55.41%-- alloc_pages_current
| __page_cache_alloc
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--44.59%-- __alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--68.42%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--31.58%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.43% tar [kernel.kallsyms] [k] in_lock_functions
|
--- in_lock_functions
|
|--76.43%-- get_parent_ip
| |
| |--80.00%-- add_preempt_count
| | |
| | |--39.87%-- balance_dirty_pages_ratelimited_nr
| | | generic_file_buffered_write
| | | __generic_file_aio_write
| | | generic_file_aio_write
| | | ext4_file_write
| | | do_sync_write
| | | vfs_write
| | | sys_write
| | | system_call_fastpath
| | | __GI___libc_write
| | |
| | |--30.07%-- radix_tree_preload
| | | add_to_page_cache_locked
| | | add_to_page_cache_lru
| | | grab_cache_page_write_begin
| | | ext4_da_write_begin
| | | generic_file_buffered_write
| | | __generic_file_aio_write
| | | generic_file_aio_write
| | | ext4_file_write
| | | do_sync_write
| | | vfs_write
| | | sys_write
| | | system_call_fastpath
| | | __GI___libc_write
| | |
| | --30.07%-- _raw_read_lock
| | start_this_handle
| | jbd2__journal_start
| | jbd2_journal_start
| | ext4_journal_start_sb
| | ext4_dirty_inode
| | __mark_inode_dirty
| | ext4_setattr
| | notify_change
| | chown_common
| | sys_fchown
| | system_call_fastpath
| | __GI___fchown
| |
| --20.00%-- sub_preempt_count
| avc_has_perm_noaudit
| avc_has_perm
| inode_has_perm
| dentry_has_perm
| selinux_inode_setattr
| security_inode_setattr
| notify_change
| chown_common
| sys_fchown
| system_call_fastpath
| __GI___fchown
|
|--12.38%-- add_preempt_count
| jbd_lock_bh_state
| do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--11.19%-- sub_preempt_count
_raw_spin_unlock_irqrestore
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.43% swapper [kernel.kallsyms] [k] ext4_end_bio
|
--- ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.42% tar [kernel.kallsyms] [k] ext4_get_group_desc
|
--- ext4_get_group_desc
|
|--88.05%-- __ext4_get_inode_loc
| ext4_get_inode_loc
| |
| |--64.96%-- ext4_reserve_inode_write
| | ext4_mark_inode_dirty
| | ext4_dirty_inode
| | __mark_inode_dirty
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --35.04%-- ext4_xattr_get
| ext4_xattr_security_get
| generic_getxattr
| cap_inode_need_killpriv
| security_inode_need_killpriv
| file_remove_suid
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--11.95%-- ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.42% tar [kernel.kallsyms] [k] __percpu_counter_add
|
--- __percpu_counter_add
|
|--39.75%-- ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--25.32%-- ext4_claim_free_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--19.28%-- ext4_free_inode
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--15.64%-- __add_bdi_stat
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.42% tar [kernel.kallsyms] [k] do_get_write_access
|
--- do_get_write_access
|
|--80.57%-- jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--19.43%-- __ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.42% tar [kernel.kallsyms] [k] generic_file_buffered_write
|
--- generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.41% tar [kernel.kallsyms] [k] jbd2_journal_stop
|
--- jbd2_journal_stop
__ext4_journal_stop
|
|--79.71%-- ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--20.29%-- ext4_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.41% tar [kernel.kallsyms] [k] jbd2__journal_start
|
--- jbd2__journal_start
|
|--82.99%-- jbd2_journal_start
| ext4_journal_start_sb
| |
| |--83.87%-- ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --16.13%-- ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--17.01%-- ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.40% tar [kernel.kallsyms] [k] do_mpage_readpage
|
--- do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.39% tar [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
__might_sleep
|
|--55.35%-- __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--61.18%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --38.82%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--28.25%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--16.40%-- __getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.38% tar [kernel.kallsyms] [k] alloc_pages_current
|
--- alloc_pages_current
|
|--82.52%-- __page_cache_alloc
| |
| |--74.42%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --25.58%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--17.48%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.37% tar [kernel.kallsyms] [k] __nr_to_section
|
--- __nr_to_section
lookup_page_cgroup
|
|--70.03%-- __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--17.38%-- mem_cgroup_get_reclaim_stat_from_page
| update_page_reclaim_stat
| ____pagevec_lru_add_fn
| pagevec_lru_move_fn
| ____pagevec_lru_add
| __lru_cache_add
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--12.60%-- mem_cgroup_del_lru_list
activate_page
mark_page_accessed
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.37% tar [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
|
|--52.73%-- _raw_spin_unlock_irqrestore
| |
| |--51.73%-- __wake_up
| | jbd2_journal_stop
| | __ext4_journal_stop
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --48.27%-- pagevec_lru_move_fn
| ____pagevec_lru_add
| __lru_cache_add
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.27%-- __wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.36% tar [kernel.kallsyms] [k] __mem_cgroup_try_charge.constprop.27
|
--- __mem_cgroup_try_charge.constprop.27
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--67.66%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--32.34%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.36% swapper [kernel.kallsyms] [k] unlock_page
|
--- unlock_page
|
|--85.03%-- mpage_end_io
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--14.97%-- bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.35% tar [kernel.kallsyms] [k] ext4_xattr_get
|
--- ext4_xattr_get
|
|--68.95%-- ext4_xattr_security_get
| generic_getxattr
| cap_inode_need_killpriv
| security_inode_need_killpriv
| file_remove_suid
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--31.05%-- generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.35% perf [kernel.kallsyms] [k] seq_puts
|
--- seq_puts
render_sigset_t
proc_pid_status
proc_single_show
seq_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.35% tar [kernel.kallsyms] [k] grab_cache_page_write_begin
|
--- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.34% tar [kernel.kallsyms] [k] alloc_page_buffers
|
--- alloc_page_buffers
|
|--72.20%-- create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--27.80%-- __block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.34% tar [kernel.kallsyms] [k] debug_smp_processor_id
|
--- debug_smp_processor_id
|
|--58.75%-- get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--70.45%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --29.55%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--41.25%-- __lru_cache_add
add_to_page_cache_lru
|
|--61.47%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--38.53%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.34% perf [kernel.kallsyms] [k] clear_page_c
|
--- clear_page_c
__alloc_pages_nodemask
perf_mmap_alloc_page
perf_mmap
mmap_region
do_mmap_pgoff
sys_mmap_pgoff
sys_mmap
system_call_fastpath
__mmap
0x418523
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.34% tar [kernel.kallsyms] [k] iov_iter_copy_from_user_atomic
|
--- iov_iter_copy_from_user_atomic
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.33% tar [kernel.kallsyms] [k] page_cache_get_speculative
|
--- page_cache_get_speculative
find_get_page
|
|--84.05%-- find_lock_page
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--15.95%-- generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.33% top [kernel.kallsyms] [k] follow_managed
|
--- follow_managed
walk_component
link_path_walk
path_openat
do_filp_open
do_sys_open
sys_open
system_call_fastpath
__GI___libc_open
0.33% flush-253:2 [kernel.kallsyms] [k] ext4_num_dirty_pages
|
--- ext4_num_dirty_pages
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.32% swapper [kernel.kallsyms] [k] do_timer
|
--- do_timer
tick_do_update_jiffies64
|
|--61.57%-- tick_sched_timer
| __run_hrtimer
| hrtimer_interrupt
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--38.43%-- tick_check_idle
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.32% tar [kernel.kallsyms] [k] __generic_file_aio_write
|
--- __generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.32% tar [kernel.kallsyms] [k] mem_cgroup_add_lru_list
|
--- mem_cgroup_add_lru_list
add_page_to_lru_list
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
|
|--87.33%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--12.67%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.32% tar [kernel.kallsyms] [k] ext4_da_write_end
|
--- ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.32% flush-253:2 [kernel.kallsyms] [k] ext4_bio_write_page
|
--- ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.32% swapper [kernel.kallsyms] [k] tick_nohz_stop_sched_tick
|
--- tick_nohz_stop_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.31% tar [kernel.kallsyms] [k] down_read
|
--- down_read
|
|--78.35%-- ext4_map_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--21.65%-- ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.31% tar [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
|
|--83.21%-- __wake_up
| jbd2_journal_stop
| __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--16.79%-- force_qs_rnp
force_quiescent_state
__rcu_process_callbacks
rcu_process_callbacks
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.30% tar [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--59.37%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--40.63%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.30% perf [kernel.kallsyms] [k] file_update_time
|
--- file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__write_nocancel
0.30% tar [kernel.kallsyms] [k] mark_buffer_dirty
|
--- mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.29% swapper [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--54.58%-- _raw_spin_lock_irqsave
| |
| |--57.58%-- scsi_device_unbusy
| | scsi_finish_command
| | scsi_softirq_done
| | blk_done_softirq
| | __do_softirq
| | call_softirq
| | do_softirq
| | irq_exit
| | do_IRQ
| | common_interrupt
| | cpuidle_idle_call
| | cpu_idle
| | rest_init
| | start_kernel
| | x86_64_start_reservations
| | x86_64_start_kernel
| |
| --42.42%-- test_clear_page_writeback
| end_page_writeback
| put_io_page
| ext4_end_bio
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--20.90%-- __percpu_counter_add
| __add_bdi_stat
| test_clear_page_writeback
| end_page_writeback
| put_io_page
| ext4_end_bio
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--15.95%-- irq_enter
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--8.56%-- scsi_device_unbusy
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.29% tar [kernel.kallsyms] [k] mnt_want_write_file
|
--- mnt_want_write_file
|
|--76.04%-- __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--23.96%-- system_call_fastpath
__GI___fchown
0.27% tar [kernel.kallsyms] [k] find_busiest_group
|
--- find_busiest_group
load_balance
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.26% perf [kernel.kallsyms] [k] perf_event_mmap_output
|
--- perf_event_mmap_output
perf_event_mmap_ctx
perf_event_mmap
mmap_region
do_mmap_pgoff
sys_mmap_pgoff
sys_mmap
system_call_fastpath
__mmap
0.26% swapper [kernel.kallsyms] [k] kmem_cache_free
|
--- kmem_cache_free
|
|--59.24%-- mempool_free
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--40.76%-- scsi_pool_free_command
__scsi_put_command
scsi_put_command
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.26% perf [kernel.kallsyms] [k] fsnotify
|
--- fsnotify
fsnotify_modify
vfs_write
sys_write
system_call_fastpath
__write_nocancel
0x4293b8
0x429c0a
0x418709
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.26% perf [kernel.kallsyms] [k] put_bh
|
--- put_bh
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__write_nocancel
0x4293b8
0x429c0a
0x418709
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.26% perf [kernel.kallsyms] [k] _raw_spin_lock
|
--- _raw_spin_lock
prepend_path
path_with_deleted
d_path
seq_path
show_map_vma
show_map
seq_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.26% tar [kernel.kallsyms] [k] get_page
|
--- get_page
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.26% tar [kernel.kallsyms] [k] selinux_file_permission
|
--- selinux_file_permission
security_file_permission
rw_verify_area
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.26% tar [kernel.kallsyms] [k] test_ti_thread_flag
|
--- test_ti_thread_flag
need_resched
should_resched
_cond_resched
|
|--64.24%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--35.76%-- generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.26% perf [kernel.kallsyms] [k] number
|
--- number
vsnprintf
seq_printf
show_map_vma
show_map
seq_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.25% tar libc-2.13.90.so [.] __GI___libc_write
|
--- __GI___libc_write
0.25% tar [kernel.kallsyms] [k] test_and_set_bit
|
--- test_and_set_bit
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.25% perf [kernel.kallsyms] [k] vsnprintf
|
--- vsnprintf
seq_printf
show_map_vma
show_map
seq_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.25% tar [kernel.kallsyms] [k] run_timer_softirq
|
--- run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.25% perf [kernel.kallsyms] [k] setup_object
|
--- setup_object
new_slab
__slab_alloc
kmem_cache_alloc
d_alloc
proc_fill_cache
proc_task_readdir
vfs_readdir
sys_getdents
system_call_fastpath
__getdents64
0x429beb
0x418709
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.24% swapper [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
|
|--40.64%-- __bit_spin_unlock.constprop.24
| __slab_free
| kmem_cache_free
| put_io_page
| ext4_end_bio
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--33.53%-- _raw_spin_unlock_irqrestore
| rcu_report_qs_rnp
| __rcu_process_callbacks
| rcu_process_callbacks
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--25.82%-- __slab_free
kmem_cache_free
mempool_free_slab
mempool_free
bio_free
bio_fs_destructor
bio_put
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.24% tar [kernel.kallsyms] [k] add_to_page_cache_locked
|
--- add_to_page_cache_locked
|
|--64.11%-- add_to_page_cache_lru
| |
| |--63.51%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --36.49%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--35.89%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.24% tar [kernel.kallsyms] [k] file_remove_suid
|
--- file_remove_suid
|
|--81.19%-- __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--18.81%-- generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.24% tar [kernel.kallsyms] [k] clear_bit_unlock
|
--- clear_bit_unlock
|
|--71.97%-- unlock_buffer
| do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| |
| |--58.50%-- file_update_time
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --41.50%-- generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--28.03%-- do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.23% tar [kernel.kallsyms] [k] sys_write
|
--- sys_write
system_call_fastpath
__GI___libc_write
0.23% swapper [kernel.kallsyms] [k] SetPageUptodate
|
--- SetPageUptodate
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.23% tar [kernel.kallsyms] [k] __bio_add_page.part.2
|
--- __bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.23% swapper [kernel.kallsyms] [k] rcu_irq_enter
|
--- rcu_irq_enter
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.23% tar [kernel.kallsyms] [k] _raw_spin_unlock_irq
|
--- _raw_spin_unlock_irq
|
|--45.40%-- generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--29.98%-- mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--24.62%-- add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.23% tar [kernel.kallsyms] [k] bit_spin_lock
|
--- bit_spin_lock
|
|--51.02%-- jbd2_journal_put_journal_head
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--48.98%-- jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.22% swapper [kernel.kallsyms] [k] nr_iowait_cpu
|
--- nr_iowait_cpu
tick_nohz_stop_idle
tick_check_idle
irq_enter
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.22% flush-253:2 [kernel.kallsyms] [k] page_waitqueue
|
--- page_waitqueue
|
|--51.36%-- unlock_page
| ext4_bio_write_page
| mpage_da_submit_io
| mpage_da_map_and_submit
| ext4_da_writepages
| do_writepages
| writeback_single_inode
| writeback_sb_inodes
| writeback_inodes_wb
| wb_writeback
| wb_do_writeback
| bdi_writeback_thread
| kthread
| kernel_thread_helper
|
--48.64%-- ext4_num_dirty_pages
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.22% tar [kernel.kallsyms] [k] __jbd2_log_space_left
|
--- __jbd2_log_space_left
start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
|
|--50.77%-- ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--49.23%-- ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.22% tar [kernel.kallsyms] [k] mb_test_bit
|
--- mb_test_bit
mb_free_blocks
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.22% tar [kernel.kallsyms] [k] __blk_recalc_rq_segments
|
--- __blk_recalc_rq_segments
blk_recount_segments
bio_phys_segments
blk_rq_bio_prep
init_request_from_bio
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.22% tar [kernel.kallsyms] [k] generic_write_end
|
--- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.22% tar [kernel.kallsyms] [k] jbd2_journal_dirty_metadata
|
--- jbd2_journal_dirty_metadata
|
|--62.74%-- __ext4_handle_dirty_metadata
| ext4_mark_iloc_dirty
| ext4_mark_inode_dirty
| |
| |--59.79%-- ext4_dirty_inode
| | __mark_inode_dirty
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --40.21%-- ext4_unlink
| vfs_unlink
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--37.26%-- ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.22% tar [kernel.kallsyms] [k] __mem_cgroup_threshold
|
--- __mem_cgroup_threshold
memcg_check_events
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.21% tar [kernel.kallsyms] [k] kfree
|
--- kfree
start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
|
|--79.14%-- ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--20.86%-- ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.21% tar [kernel.kallsyms] [k] ext4_get_inode_flags
|
--- ext4_get_inode_flags
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
|
|--63.71%-- file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--36.29%-- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.21% flush-253:2 [kernel.kallsyms] [k] put_page_testzero
|
--- put_page_testzero
release_pages
__pagevec_release
pagevec_release
|
|--57.93%-- ext4_num_dirty_pages
| ext4_da_writepages
| do_writepages
| writeback_single_inode
| writeback_sb_inodes
| writeback_inodes_wb
| wb_writeback
| wb_do_writeback
| bdi_writeback_thread
| kthread
| kernel_thread_helper
|
--42.07%-- write_cache_pages_da
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.21% flush-253:2 [kernel.kallsyms] [k] __rcu_read_unlock
|
--- __rcu_read_unlock
__find_get_block_slow
unmap_underlying_metadata
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.21% tar [kernel.kallsyms] [k] avtab_search
|
--- avtab_search
security_compute_sid
security_transition_sid
may_create
selinux_inode_create
security_inode_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.21% tar [kernel.kallsyms] [k] search_dirblock
|
--- search_dirblock
ext4_dx_find_entry
ext4_find_entry
ext4_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.21% tar [kernel.kallsyms] [k] __sg_alloc_table
|
--- __sg_alloc_table
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.21% tar [kernel.kallsyms] [k] ext4_journal_start_sb
|
--- ext4_journal_start_sb
ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.21% top libc-2.13.90.so [.] __GI_____strtoll_l_internal
|
--- __GI_____strtoll_l_internal
0.20% swapper [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
_raw_spin_lock
|
|--57.86%-- handle_edge_irq
| handle_irq
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--42.14%-- tick_do_update_jiffies64
tick_check_idle
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.20% tar [kernel.kallsyms] [k] generic_write_sync
|
--- generic_write_sync
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.20% tar [kernel.kallsyms] [k] put_page
|
--- put_page
|
|--64.65%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--35.35%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.20% flush-253:2 [kernel.kallsyms] [k] tag_get
|
--- tag_get
|
|--58.46%-- __lookup_tag
| radix_tree_gang_lookup_tag_slot
| find_get_pages_tag
| pagevec_lookup_tag
| ext4_num_dirty_pages
| ext4_da_writepages
| do_writepages
| writeback_single_inode
| writeback_sb_inodes
| writeback_inodes_wb
| wb_writeback
| wb_do_writeback
| bdi_writeback_thread
| kthread
| kernel_thread_helper
|
--41.54%-- test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.20% tar [kernel.kallsyms] [k] dm_merge_bvec
|
--- dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.19% tar [kernel.kallsyms] [k] __ext4_handle_dirty_metadata
|
--- __ext4_handle_dirty_metadata
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.19% tar [kernel.kallsyms] [k] __block_commit_write
|
--- __block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.19% tar [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore
|
--- _raw_spin_unlock_irqrestore
|
|--54.81%-- __wake_up
| jbd2_journal_stop
| __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--45.19%-- __delayacct_blkio_end
delayacct_blkio_end
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.19% tar [kernel.kallsyms] [k] ext4_writepage_trans_blocks
|
--- ext4_writepage_trans_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.19% tar [kernel.kallsyms] [k] jbd_lock_bh_state
|
--- jbd_lock_bh_state
|
|--62.13%-- do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--37.87%-- jbd2_journal_dirty_metadata
__ext4_handle_dirty_metadata
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.19% tar [kernel.kallsyms] [k] __list_add
|
--- __list_add
|
|--70.74%-- __make_request
| generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--29.26%-- expand
__rmqueue
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.19% tar [kernel.kallsyms] [k] fsnotify_modify
|
--- fsnotify_modify
|
|--74.13%-- vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--25.87%-- sys_write
system_call_fastpath
__GI___libc_write
0.19% flush-253:2 [kernel.kallsyms] [k] lock_page
|
--- lock_page
write_cache_pages_da
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.19% flush-253:2 [kernel.kallsyms] [k] rb_next
|
--- rb_next
set_next_entity
pick_next_task_fair
pick_next_task
schedule
preempt_schedule_irq
retint_kernel
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.19% tar [kernel.kallsyms] [k] ext4_ext_calc_metadata_amount
|
--- ext4_ext_calc_metadata_amount
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.18% swapper [kernel.kallsyms] [k] __wake_up_bit
|
--- __wake_up_bit
|
|--61.24%-- unlock_page
| mpage_end_io
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--38.76%-- end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.18% swapper [kernel.kallsyms] [k] select_nohz_load_balancer
|
--- select_nohz_load_balancer
|
|--75.59%-- tick_nohz_restart_sched_tick
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--24.41%-- tick_nohz_stop_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.18% swapper [kernel.kallsyms] [k] menu_select
|
--- menu_select
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.18% flush-253:2 [kernel.kallsyms] [k] page_cache_get_speculative
|
--- page_cache_get_speculative
find_get_pages
pagevec_lookup
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.18% tar [kernel.kallsyms] [k] generic_file_aio_read
|
--- generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.18% tar [kernel.kallsyms] [k] __prop_inc_single
|
--- __prop_inc_single
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.18% swapper [kernel.kallsyms] [k] page_index
|
--- page_index
test_clear_page_writeback
end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.17% tar [kernel.kallsyms] [k] arch_read_lock
|
--- arch_read_lock
_raw_read_lock
start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.17% tar [kernel.kallsyms] [k] mutex_unlock
|
--- mutex_unlock
|
|--73.48%-- generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--26.52%-- do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.17% tar [kernel.kallsyms] [k] blk_rq_map_sg
|
--- blk_rq_map_sg
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.17% tar [kernel.kallsyms] [k] bit_waitqueue
|
--- bit_waitqueue
|
|--51.20%-- wake_up_bit
| unlock_buffer
| do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--48.80%-- unlock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.17% tar [kernel.kallsyms] [k] SetPageLRU
|
--- SetPageLRU
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
|
|--71.47%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--28.53%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.17% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6
|
--- test_ti_thread_flag.constprop.6
|
|--64.90%-- _raw_spin_unlock_irq
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--35.10%-- inode_add_rsv_space
__dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.17% tar [kernel.kallsyms] [k] lookup_page_cgroup
|
--- lookup_page_cgroup
|
|--58.52%-- __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--41.48%-- mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.17% tar [kernel.kallsyms] [k] generic_make_request
|
--- generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.17% tar [kernel.kallsyms] [k] lock_page_cgroup
|
--- lock_page_cgroup
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.17% swapper [kernel.kallsyms] [k] mempool_free
|
--- mempool_free
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.17% tar [kernel.kallsyms] [k] put_prev_task_fair
|
--- put_prev_task_fair
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.17% tar [kernel.kallsyms] [k] perf_event_task_tick
|
--- perf_event_task_tick
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
|
|--51.07%-- scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--48.93%-- _ext4_get_block
ext4_get_block
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.17% tar [kernel.kallsyms] [k] arch_read_unlock
|
--- arch_read_unlock
|
|--66.07%-- _raw_read_unlock
| start_this_handle
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--33.93%-- _raw_read_unlock_irqrestore
dm_get_live_table
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.16% swapper [kernel.kallsyms] [k] debug_smp_processor_id
|
--- debug_smp_processor_id
|
|--38.89%-- __cycles_2_ns
| native_sched_clock
| sched_clock
| sched_clock_cpu
| account_system_vtime
| irq_exit
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--32.56%-- tick_check_idle
| irq_enter
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--14.27%-- account_idle_time
| irqtime_account_process_tick
| account_idle_ticks
| tick_nohz_restart_sched_tick
| cpu_idle
| start_secondary
|
--14.27%-- irqtime_account_process_tick
account_idle_ticks
tick_nohz_restart_sched_tick
cpu_idle
start_secondary
0.16% tar [kernel.kallsyms] [k] fget_light
|
--- fget_light
|
|--51.07%-- sys_read
| system_call_fastpath
| __GI___libc_read
|
--48.93%-- sys_write
system_call_fastpath
__GI___libc_write
0.16% tar [kernel.kallsyms] [k] SetPageUptodate
|
--- SetPageUptodate
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.16% tar [kernel.kallsyms] [k] __ext4_journal_stop
|
--- __ext4_journal_stop
|
|--50.22%-- ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--49.78%-- ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.16% tar [kernel.kallsyms] [k] ext4_clear_inode_state
|
--- ext4_clear_inode_state
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
|
|--51.89%-- file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--48.11%-- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.16% tar [kernel.kallsyms] [k] mem_cgroup_cache_charge
|
--- mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--65.55%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--34.45%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.16% tar [kernel.kallsyms] [k] ext4_map_blocks
|
--- ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.16% tar [kernel.kallsyms] [k] ata_dev_enabled
|
--- ata_dev_enabled
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.16% tar [kernel.kallsyms] [k] new_slab
|
--- new_slab
__slab_alloc
kmem_cache_alloc
alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.16% swapper [kernel.kallsyms] [k] hrtimer_cancel
|
--- hrtimer_cancel
tick_nohz_restart_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.15% tar [kernel.kallsyms] [k] lock_buffer
|
--- lock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.15% tar [kernel.kallsyms] [k] bit_spin_unlock
|
--- bit_spin_unlock
unlock_page_cgroup
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--72.67%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--27.33%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.15% tar [kernel.kallsyms] [k] up_read
|
--- up_read
|
|--62.56%-- ext4_map_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--37.44%-- ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.15% tar [kernel.kallsyms] [k] preempt_schedule
|
--- preempt_schedule
try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
_raw_read_lock
start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.15% tar [kernel.kallsyms] [k] fsnotify_access
|
--- fsnotify_access
sys_read
system_call_fastpath
__GI___libc_read
0.15% swapper [kernel.kallsyms] [k] native_read_tsc
|
--- native_read_tsc
paravirt_read_tsc
native_sched_clock
sched_clock
sched_clock_cpu
account_system_vtime
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.14% tar [kernel.kallsyms] [k] dequeue_task_fair
|
--- dequeue_task_fair
dequeue_task
deactivate_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.14% tar [kernel.kallsyms] [k] ata_scsi_queuecmd
|
--- ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.14% tar [kernel.kallsyms] [k] file_read_actor
|
--- file_read_actor
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.14% tar [kernel.kallsyms] [k] _raw_spin_lock
|
--- _raw_spin_lock
|
|--52.50%-- ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.50%-- ext4_ext_map_blocks
ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.14% top [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
_raw_spin_lock
nameidata_drop_rcu
link_path_walk
path_openat
do_filp_open
do_sys_open
sys_open
system_call_fastpath
__GI___libc_open
0.14% tar [kernel.kallsyms] [k] generic_file_aio_write
|
--- generic_file_aio_write
|
|--52.11%-- do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.89%-- ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.14% tar [kernel.kallsyms] [k] block_write_end
|
--- block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.14% tar [kernel.kallsyms] [k] hrtimer_interrupt
|
--- hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
|
|--52.67%-- jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.33%-- string_to_context_struct
security_context_to_sid_core
security_context_to_sid_default
inode_doinit_with_dentry
selinux_d_instantiate
security_d_instantiate
d_instantiate
d_splice_alias
ext4_lookup
d_alloc_and_lookup
__lookup_hash.part.3
lookup_hash
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.14% flush-253:2 [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
__percpu_counter_add
__add_bdi_stat
clear_page_dirty_for_io
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.14% swapper [kernel.kallsyms] [k] __rcu_read_lock
|
--- __rcu_read_lock
|
|--50.04%-- __prop_inc_percpu_max
| test_clear_page_writeback
| end_page_writeback
| put_io_page
| ext4_end_bio
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--49.96%-- __atomic_notifier_call_chain
atomic_notifier_call_chain
exit_idle
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.13% tar [kernel.kallsyms] [k] ktime_get_ts
|
--- ktime_get_ts
__delayacct_blkio_end
delayacct_blkio_end
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.13% tar [kernel.kallsyms] [k] page_zone
|
--- page_zone
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] __read_lock_failed
|
--- __read_lock_failed
_raw_read_lock
start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] prop_norm_single
|
--- prop_norm_single
__prop_inc_single
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] put_bh
|
--- put_bh
__brelse
brelse
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] radix_tree_insert
|
--- radix_tree_insert
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] policy_zonelist
|
--- policy_zonelist
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.13% tar [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% swapper [kernel.kallsyms] [k] irq_entries_start
|
--- irq_entries_start
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.13% top libproc-3.2.8.so [.] 0x44c2
|
--- 0x37d94044c2
0.13% flush-253:2 [kernel.kallsyms] [k] attempt_back_merge
|
--- attempt_back_merge
generic_make_request
submit_bio
ext4_io_submit
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.13% tar [kernel.kallsyms] [k] virt_to_head_page
|
--- virt_to_head_page
kmem_cache_free
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] mnt_clone_write
|
--- mnt_clone_write
mnt_want_write_file
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] rb_erase
|
--- rb_erase
deadline_remove_request
deadline_move_request
deadline_dispatch_requests
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.13% tar [kernel.kallsyms] [k] __set_page_dirty
|
--- __set_page_dirty
|
|--51.11%-- __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--48.89%-- mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] do_sync_write
|
--- do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
__find_get_block
__getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] arch_local_irq_disable
|
--- arch_local_irq_disable
_raw_spin_lock_irq
|
|--55.62%-- add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--44.38%-- __set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] kmem_cache_free
|
--- kmem_cache_free
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.13% tar [kernel.kallsyms] [k] ext4_da_write_begin
|
--- ext4_da_write_begin
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% swapper [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
|
|--69.63%-- account_system_vtime
| irq_exit
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--30.37%-- irqtime_account_process_tick
account_idle_ticks
tick_nohz_restart_sched_tick
cpu_idle
start_secondary
0.12% flush-253:2 [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.12% tar [kernel.kallsyms] [k] inode_reserved_space
|
--- inode_reserved_space
__dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% swapper [kernel.kallsyms] [k] update_cfs_shares
|
--- update_cfs_shares
enqueue_task_fair
enqueue_task
activate_task
try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.12% tar [kernel.kallsyms] [k] schedule
|
--- schedule
|
|--59.95%-- io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--40.05%-- __GI___libc_read
0.12% tar [kernel.kallsyms] [k] security_sid_to_context_force
|
--- security_sid_to_context_force
security_inode_init_security
ext4_init_security
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.12% tar [kernel.kallsyms] [k] radix_tree_preload
|
--- radix_tree_preload
add_to_page_cache_locked
add_to_page_cache_lru
|
|--54.07%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--45.93%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.12% swapper [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
_local_bh_enable
irq_enter
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.12% flush-253:2 [kernel.kallsyms] [k] release_pages
|
--- release_pages
__pagevec_release
pagevec_release
write_cache_pages_da
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.12% flush-253:2 [kernel.kallsyms] [k] page_mkclean
|
--- page_mkclean
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.12% tar [kernel.kallsyms] [k] set_bh_page
|
--- set_bh_page
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% tar [kernel.kallsyms] [k] account_entity_dequeue
|
--- account_entity_dequeue
dequeue_task
deactivate_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.12% tar [kernel.kallsyms] [k] ata_qc_issue
|
--- ata_qc_issue
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.12% tar libc-2.13.90.so [.] __cfree
|
--- __cfree
0.12% tar [kernel.kallsyms] [k] radix_tree_lookup_slot
|
--- radix_tree_lookup_slot
|
|--62.63%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--37.37%-- find_lock_page
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% top libc-2.13.90.so [.] _IO_vfscanf_internal
|
--- _IO_vfscanf_internal
_IO_vsscanf
0x7fffef292b20
0.12% tar [kernel.kallsyms] [k] ext4_inode_table
|
--- ext4_inode_table
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% tar [kernel.kallsyms] [k] xattr_resolve_name
|
--- xattr_resolve_name
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% tar [kernel.kallsyms] [k] add_to_page_cache_lru
|
--- add_to_page_cache_lru
|
|--67.10%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--32.90%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.12% migration/0 [kernel.kallsyms] [k] resched_task
|
--- resched_task
check_preempt_curr
__migrate_task
migration_cpu_stop
cpu_stopper_thread
kthread
kernel_thread_helper
0.12% tar [kernel.kallsyms] [k] __count_vm_events
|
--- __count_vm_events
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--50.90%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--49.10%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.12% tar [kernel.kallsyms] [k] system_call_fastpath
|
--- system_call_fastpath
__GI___libc_write
0.11% flush-253:2 [kernel.kallsyms] [k] __percpu_counter_add
|
--- __percpu_counter_add
__add_bdi_stat
clear_page_dirty_for_io
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.11% swapper [kernel.kallsyms] [k] page_waitqueue
|
--- page_waitqueue
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.11% kworker/0:2 [kernel.kallsyms] [k] worker_thread
|
--- worker_thread
kthread
kernel_thread_helper
0.11% tar [kernel.kallsyms] [k] fsnotify
|
--- fsnotify
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.11% tar [kernel.kallsyms] [k] is_handle_aborted
|
--- is_handle_aborted
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] blk_finish_plug
|
--- blk_finish_plug
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] arch_local_irq_save
|
--- arch_local_irq_save
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] jbd2_journal_get_write_access
|
--- jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] vfs_write
|
--- vfs_write
system_call_fastpath
__GI___libc_write
0.11% swapper [kernel.kallsyms] [k] irqtime_account_process_tick
|
--- irqtime_account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
start_secondary
0.11% tar [kernel.kallsyms] [k] blk_flush_plug_list
|
--- blk_flush_plug_list
blk_finish_plug
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.11% tar [kernel.kallsyms] [k] deadline_remove_request
|
--- deadline_remove_request
deadline_dispatch_requests
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.11% tar [kernel.kallsyms] [k] mb_find_buddy
|
--- mb_find_buddy
mb_free_blocks
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.11% tar [kernel.kallsyms] [k] sys_read
|
--- sys_read
|
|--58.45%-- system_call_fastpath
| __GI___libc_read
|
--41.55%-- __GI___libc_read
0.11% flush-253:2 [kernel.kallsyms] [k] arch_read_unlock
|
--- arch_read_unlock
_raw_read_unlock_irqrestore
dm_get_live_table
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.11% swapper [kernel.kallsyms] [k] __slab_free
|
--- __slab_free
kmem_cache_free
mempool_free_slab
mempool_free
scsi_sg_free
__sg_free_table
__scsi_release_buffers
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.11% tar [kernel.kallsyms] [k] sidtab_search_core
|
--- sidtab_search_core
sidtab_search
security_compute_sid
security_transition_sid
may_create
selinux_inode_create
security_inode_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.11% swapper [kernel.kallsyms] [k] switch_mm
|
--- switch_mm
schedule
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.11% flush-253:2 [kernel.kallsyms] [k] mpage_da_submit_io
|
--- mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.11% tar [kernel.kallsyms] [k] mem_cgroup_charge_statistics
|
--- mem_cgroup_charge_statistics
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--56.66%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--43.34%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.11% tar [kernel.kallsyms] [k] zone_watermark_ok
|
--- zone_watermark_ok
|
|--50.80%-- __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--49.20%-- get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] ext4_free_blocks
|
--- ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.11% tar [kernel.kallsyms] [k] ext4_ext_map_blocks
|
--- ext4_ext_map_blocks
ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] page_waitqueue
|
--- page_waitqueue
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.11% tar [kernel.kallsyms] [k] add_dirent_to_buf
|
--- add_dirent_to_buf
ext4_add_entry
ext4_add_nondir
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.11% tar [kernel.kallsyms] [k] strcmp
|
--- strcmp
symcmp
hashtab_search
string_to_context_struct
security_context_to_sid_core
security_context_to_sid_default
inode_doinit_with_dentry
selinux_d_instantiate
security_d_instantiate
d_instantiate
d_splice_alias
ext4_lookup
d_alloc_and_lookup
__lookup_hash.part.3
lookup_hash
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.11% tar [kernel.kallsyms] [k] select_idle_sibling
|
--- select_idle_sibling
select_task_rq_fair
select_task_rq
try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.11% flush-253:2 [kernel.kallsyms] [k] __wake_up_bit
|
--- __wake_up_bit
unlock_page
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.11% tar [kernel.kallsyms] [k] radix_tree_node_alloc
|
--- radix_tree_node_alloc
radix_tree_insert
add_to_page_cache_locked
add_to_page_cache_lru
|
|--60.14%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--39.86%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% swapper [kernel.kallsyms] [k] rcu_irq_exit
|
--- rcu_irq_exit
|
|--54.97%-- smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--45.03%-- irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.10% tar [kernel.kallsyms] [k] radix_tree_tag_set
|
--- radix_tree_tag_set
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% tar [kernel.kallsyms] [k] check_irqs_on
|
--- check_irqs_on
__getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% flush-253:2 [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
__add_bdi_stat
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% flush-253:2 [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
__might_sleep
unmap_underlying_metadata
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% tar [kernel.kallsyms] [k] irqtime_account_process_tick
|
--- irqtime_account_process_tick
account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% tar [kernel.kallsyms] [k] alloc_buffer_head
|
--- alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% tar [kernel.kallsyms] [k] jiffies_to_timeval
|
--- jiffies_to_timeval
__account_system_time
irqtime_account_process_tick
account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
__find_get_block
__getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.10% tar [kernel.kallsyms] [k] ext4_get_reserved_space
|
--- ext4_get_reserved_space
inode_add_rsv_space
__dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% swapper [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.10% tar [kernel.kallsyms] [k] blk_start_plug
|
--- blk_start_plug
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.10% flush-253:2 [kernel.kallsyms] [k] sg_init_table
|
--- sg_init_table
scsi_alloc_sgtable
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
__make_request
generic_make_request
submit_bio
ext4_io_submit
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% tar [kernel.kallsyms] [k] ata_qc_new_init
|
--- ata_qc_new_init
|
|--51.58%-- ata_scsi_queuecmd
| scsi_dispatch_cmd
| scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--48.42%-- __ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.10% tar [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
_raw_spin_unlock_irq
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.10% flush-253:2 [kernel.kallsyms] [k] inc_zone_page_state
|
--- inc_zone_page_state
account_page_writeback
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% tar [kernel.kallsyms] [k] system_call
|
--- system_call
__GI___libc_write
0.10% flush-253:2 [kernel.kallsyms] [k] __bio_clone
|
--- __bio_clone
clone_bio
__split_and_process_bio
dm_request
generic_make_request
submit_bio
ext4_io_submit
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% flush-253:2 [kernel.kallsyms] [k] mod_state
|
--- mod_state
inc_zone_page_state
account_page_writeback
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% tar [kernel.kallsyms] [k] arch_local_irq_save
|
--- arch_local_irq_save
_raw_spin_lock_irqsave
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.10% tar [kernel.kallsyms] [k] bio_get_nr_vecs
|
--- bio_get_nr_vecs
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.10% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_set
|
--- radix_tree_tag_set
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.10% tar [kernel.kallsyms] [k] test_ti_thread_flag
|
--- test_ti_thread_flag
__mem_cgroup_try_charge.constprop.27
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.10% flush-253:2 [kernel.kallsyms] [k] bit_spin_lock
|
--- bit_spin_lock
jbd2_journal_add_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.09% tar [kernel.kallsyms] [k] inode_init_once
|
--- inode_init_once
init_once
setup_object
new_slab
__slab_alloc
kmem_cache_alloc
ext4_alloc_inode
alloc_inode
new_inode
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.09% jbd2/dm-2-8 [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--61.31%-- _raw_spin_lock
| jbd2_journal_file_buffer
| jbd2_journal_write_metadata_buffer
| jbd2_journal_commit_transaction
| kjournald2
| kthread
| kernel_thread_helper
|
--38.69%-- jbd_trylock_bh_state
journal_clean_one_cp_list
__jbd2_journal_clean_checkpoint_list
jbd2_journal_commit_transaction
kjournald2
kthread
kernel_thread_helper
0.09% swapper [kernel.kallsyms] [k] kobject_put
|
--- kobject_put
__scsi_put_command
scsi_put_command
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.09% flush-253:2 [kernel.kallsyms] [k] __elv_add_request
|
--- __elv_add_request
__make_request
generic_make_request
submit_bio
ext4_io_submit
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.09% swapper [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
rcu_process_gp_end
__rcu_process_callbacks
rcu_process_callbacks
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.09% tar [kernel.kallsyms] [k] round_jiffies_up
|
--- round_jiffies_up
blk_start_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.09% tar [kernel.kallsyms] [k] attach_page_buffers
|
--- attach_page_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.09% tar [kernel.kallsyms] [k] hrtick_update
|
--- hrtick_update
dequeue_task
deactivate_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.09% tar [kernel.kallsyms] [k] ext4_clear_inode_flag
|
--- ext4_clear_inode_flag
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.09% jbd2/dm-2-8 [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
_raw_spin_lock_irq
blk_throtl_bio
generic_make_request
submit_bio
submit_bh
jbd2_journal_commit_transaction
kjournald2
kthread
kernel_thread_helper
0.09% top [kernel.kallsyms] [k] __rcu_read_lock
|
--- __rcu_read_lock
__d_lookup
walk_component
link_path_walk
path_openat
do_filp_open
do_sys_open
sys_open
system_call_fastpath
__GI___libc_open
0.09% tar [kernel.kallsyms] [k] __brelse
|
--- __brelse
brelse
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.09% flush-253:2 [kernel.kallsyms] [k] page_zone
|
--- page_zone
inc_zone_page_state
account_page_writeback
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.09% tar [kernel.kallsyms] [k] bio_integrity_enabled
|
--- bio_integrity_enabled
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.09% tar [kernel.kallsyms] [k] dm_table_find_target
|
--- dm_table_find_target
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.09% tar [kernel.kallsyms] [k] __mod_zone_page_state
|
--- __mod_zone_page_state
add_page_to_lru_list
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.09% tar [kernel.kallsyms] [k] ahci_fill_cmd_slot
|
--- ahci_fill_cmd_slot
ahci_qc_prep
ata_qc_issue
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.09% swapper [kernel.kallsyms] [k] blk_unprep_request
|
--- blk_unprep_request
blk_finish_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.09% top libc-2.13.90.so [.] __GI___close
|
--- __GI___close
0.09% tar [kernel.kallsyms] [k] put_bh
|
--- put_bh
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.09% swapper [kernel.kallsyms] [k] finish_task_switch
|
--- finish_task_switch
schedule
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.09% flush-253:2 [kernel.kallsyms] [k] page_mapping
|
--- page_mapping
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.09% tar [kernel.kallsyms] [k] alloc_tio
|
--- alloc_tio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_clear
|
--- radix_tree_tag_clear
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.08% tar [kernel.kallsyms] [k] timespec_trunc
|
--- timespec_trunc
touch_atime
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% tar [kernel.kallsyms] [k] ext4_claim_free_blocks
|
--- ext4_claim_free_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.08% tar libc-2.13.90.so [.] __GI___errno_location
|
--- __GI___errno_location
0.08% tar [kernel.kallsyms] [k] path_to_nameidata
|
--- path_to_nameidata
walk_component
link_path_walk
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.08% tar [kernel.kallsyms] [k] current_sid
|
--- current_sid
selinux_file_permission
security_file_permission
rw_verify_area
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% top [kernel.kallsyms] [k] path_openat
|
--- path_openat
do_filp_open
do_sys_open
sys_open
system_call_fastpath
__GI___libc_open
0.08% tar [kernel.kallsyms] [k] __do_page_cache_readahead
|
--- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% tar [kernel.kallsyms] [k] _cond_resched
|
--- _cond_resched
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% swapper [kernel.kallsyms] [k] run_timer_softirq
|
--- run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.08% swapper [kernel.kallsyms] [k] note_interrupt
|
--- note_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.08% tar [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.08% swapper [kernel.kallsyms] [k] native_apic_mem_write
|
--- native_apic_mem_write
apic_write
ack_APIC_irq
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
start_secondary
0.08% tar [kernel.kallsyms] [k] bio_data
|
--- bio_data
init_request_from_bio
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% tar [kernel.kallsyms] [k] string_to_context_struct
|
--- string_to_context_struct
security_context_to_sid_core
security_context_to_sid_default
inode_doinit_with_dentry
selinux_d_instantiate
security_d_instantiate
d_instantiate
d_splice_alias
ext4_lookup
d_alloc_and_lookup
__lookup_hash.part.3
lookup_hash
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.08% tar [kernel.kallsyms] [k] mb_clear_bit
|
--- mb_clear_bit
mb_clear_bits
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.08% tar [kernel.kallsyms] [k] ext4_mb_load_buddy
|
--- ext4_mb_load_buddy
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.08% top [kernel.kallsyms] [k] __fsnotify_parent
|
--- __fsnotify_parent
fsnotify_perm
security_file_permission
rw_verify_area
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% sleep [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
_raw_spin_lock
|
|--60.83%-- __do_fault
| handle_pte_fault
| handle_mm_fault
| do_page_fault
| page_fault
| 0x37d9082b80
|
--39.17%-- deny_write_access
mmap_region
do_mmap_pgoff
do_mmap
elf_map
load_elf_binary
search_binary_handler
do_execve
sys_execve
stub_execve
0x37d90ae097
0.08% swapper [kernel.kallsyms] [k] ahci_interrupt
|
--- ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.08% tar [kernel.kallsyms] [k] __srcu_read_unlock
|
--- __srcu_read_unlock
fsnotify_modify
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.08% tar [kernel.kallsyms] [k] signal_pending
|
--- signal_pending
fatal_signal_pending
__mem_cgroup_try_charge.constprop.27
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.08% tar [kernel.kallsyms] [k] __read_seqcount_begin
|
--- __read_seqcount_begin
path_init
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.08% tar [kernel.kallsyms] [k] test_ti_thread_flag
|
--- test_ti_thread_flag
mnt_clone_write
mnt_want_write_file
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.08% tar [kernel.kallsyms] [k] jbd2_journal_start
|
--- jbd2_journal_start
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] __srcu_read_lock
|
--- __srcu_read_lock
fsnotify
fsnotify_modify
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% swapper [kernel.kallsyms] [k] handle_irq
|
--- handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% swapper [kernel.kallsyms] [k] __ata_scsi_queuecmd
|
--- __ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
blk_run_queue
scsi_run_queue
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
drive_stat_acct
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] __switch_to
|
--- __switch_to
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] ext4_file_write
|
--- ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% swapper [kernel.kallsyms] [k] wake_bit_function
|
--- wake_bit_function
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] _raw_spin_lock_irq
|
--- _raw_spin_lock_irq
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] drive_stat_acct
|
--- drive_stat_acct
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.22
|
--- test_ti_thread_flag.constprop.22
zero_user_segments
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] sd_prep_fn
|
--- sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% top libc-2.13.90.so [.] __strchrnul
|
--- __strchrnul
___vsnprintf_chk
0x39335b1b6d5b1b42
0.07% swapper [kernel.kallsyms] [k] sd_prep_fn
|
--- sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
blk_run_queue
scsi_run_queue
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] try_to_wake_up
|
--- try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
_raw_spin_unlock_irqrestore
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] __bio_clone
|
--- __bio_clone
clone_bio
__split_and_process_bio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] __zone_watermark_ok
|
--- __zone_watermark_ok
zone_watermark_ok
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% swapper [kernel.kallsyms] [k] __remove_hrtimer
|
--- __remove_hrtimer
remove_hrtimer.part.4
hrtimer_try_to_cancel
hrtimer_cancel
tick_nohz_restart_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% swapper [kernel.kallsyms] [k] scsi_device_unbusy
|
--- scsi_device_unbusy
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] scsi_prep_return
|
--- scsi_prep_return
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] prepare_to_wait_exclusive
|
--- prepare_to_wait_exclusive
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] set_buffer_uptodate
|
--- set_buffer_uptodate
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] ata_get_xlat_func
|
--- ata_get_xlat_func
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] __find_get_block_slow
|
--- __find_get_block_slow
__find_get_block
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.07% tar [kernel.kallsyms] [k] generic_getxattr
|
--- generic_getxattr
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% swapper [kernel.kallsyms] [k] bit_spin_lock.constprop.22
|
--- bit_spin_lock.constprop.22
__slab_free
kmem_cache_free
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% swapper [kernel.kallsyms] [k] is_swiotlb_buffer
|
--- is_swiotlb_buffer
unmap_single
swiotlb_unmap_sg_attrs
ata_sg_clean
__ata_qc_complete
ata_qc_complete
ata_qc_complete_multiple
ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] native_read_tsc
|
--- native_read_tsc
paravirt_read_tsc
native_sched_clock
sched_clock
blk_rq_init
get_request
get_request_wait
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] ext4_has_free_blocks
|
--- ext4_has_free_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar libc-2.13.90.so [.] __GI___fxstat64
|
--- __GI___fxstat64
0.07% swapper [kernel.kallsyms] [k] perf_pmu_disable
|
--- perf_pmu_disable
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] sidtab_context_to_sid
|
--- sidtab_context_to_sid
security_context_to_sid_core
security_context_to_sid_default
inode_doinit_with_dentry
selinux_d_instantiate
security_d_instantiate
d_instantiate
d_splice_alias
ext4_lookup
d_alloc_and_lookup
__lookup_hash.part.3
lookup_hash
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.07% tar [kernel.kallsyms] [k] scsi_pool_alloc_command
|
--- scsi_pool_alloc_command
scsi_host_alloc_command
__scsi_get_command
scsi_get_command
scsi_get_cmd_from_req
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] list_del
|
--- list_del
activate_page
mark_page_accessed
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] pagefault_enable
|
--- pagefault_enable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] security_file_permission
|
--- security_file_permission
rw_verify_area
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% swapper [kernel.kallsyms] [k] scsi_next_command
|
--- scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.07% tar [kernel.kallsyms] [k] __phys_addr
|
--- __phys_addr
virt_to_head_page
kmem_cache_free
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] page_cache_async_readahead
|
--- page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.07% tar [kernel.kallsyms] [k] truncate_inode_pages_range
|
--- truncate_inode_pages_range
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.07% tar [kernel.kallsyms] [k] mem_cgroup_del_lru_list
|
--- mem_cgroup_del_lru_list
activate_page
mark_page_accessed
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] ext4_xattr_check_names
|
--- ext4_xattr_check_names
ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] arch_local_irq_enable
|
--- arch_local_irq_enable
__find_get_block
__getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.07% tar [kernel.kallsyms] [k] get_request
|
--- get_request
get_request_wait
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] swiotlb_map_sg_attrs
|
--- swiotlb_map_sg_attrs
ata_qc_issue
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] dm_get_live_table
|
--- dm_get_live_table
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] mem_cgroup_get_reclaim_stat_from_page
|
--- mem_cgroup_get_reclaim_stat_from_page
update_page_reclaim_stat
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% swapper [kernel.kallsyms] [k] check_preempt_curr
|
--- check_preempt_curr
ttwu_post_activation
try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] atomic_long_add
|
--- atomic_long_add
zone_page_state_add
__mod_zone_page_state
add_page_to_lru_list
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% swapper [kernel.kallsyms] [k] pm_qos_request
|
--- pm_qos_request
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] __kmalloc
|
--- __kmalloc
kzalloc.constprop.9
ext4_ext_find_extent
ext4_ext_map_blocks
ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% swapper [kernel.kallsyms] [k] rcu_bh_qs
|
--- rcu_bh_qs
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% swapper [kernel.kallsyms] [k] __blk_complete_request
|
--- __blk_complete_request
scsi_done
ata_scsi_qc_complete
__ata_qc_complete
ata_qc_complete
ata_qc_complete_multiple
ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] __d_lookup
|
--- __d_lookup
walk_component
link_path_walk
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.06% tar [kernel.kallsyms] [k] context_struct_to_string
|
--- context_struct_to_string
security_sid_to_context_core
security_sid_to_context_force
selinux_inode_init_security
security_inode_init_security
ext4_init_security
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.06% swapper [kernel.kallsyms] [k] tick_nohz_restart_sched_tick
|
--- tick_nohz_restart_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] max_sane_readahead
|
--- max_sane_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] rw_verify_area
|
--- rw_verify_area
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] ahci_qc_prep
|
--- ahci_qc_prep
ata_qc_issue
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] linear_map
|
--- linear_map
__split_and_process_bio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% swapper [kernel.kallsyms] [k] sched_clock
|
--- sched_clock
account_system_vtime
irq_enter
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] fput_light
|
--- fput_light
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] iov_iter_advance
|
--- iov_iter_advance
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] __queue_work
|
--- __queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
lock_page_cgroup
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% swapper [kernel.kallsyms] [k] scsi_io_completion
|
--- scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] activate_page
|
--- activate_page
mark_page_accessed
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] touch_atime
|
--- touch_atime
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] atomic_sub
|
--- atomic_sub
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] __inc_zone_page_state
|
--- __inc_zone_page_state
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] ext4_test_inode_state
|
--- ext4_test_inode_state
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] ext4_xattr_find_entry
|
--- ext4_xattr_find_entry
ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% swapper [kernel.kallsyms] [k] deadline_dispatch_requests
|
--- deadline_dispatch_requests
blk_peek_request
scsi_request_fn
__blk_run_queue
blk_run_queue
scsi_run_queue
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% swapper [kernel.kallsyms] [k] hrtick_update
|
--- hrtick_update
enqueue_task_fair
enqueue_task
activate_task
try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% swapper [kernel.kallsyms] [k] try_to_wake_up
|
--- try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% swapper [kernel.kallsyms] [k] pick_next_task_fair
|
--- pick_next_task_fair
schedule
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] pick_next_task_idle
|
--- pick_next_task_idle
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% swapper [kernel.kallsyms] [k] ktime_get
|
--- ktime_get
tick_check_idle
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] balance_dirty_pages_ratelimited_nr
|
--- balance_dirty_pages_ratelimited_nr
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% swapper [kernel.kallsyms] [k] mpage_end_io
|
--- mpage_end_io
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] radix_tree_preload_end
|
--- radix_tree_preload_end
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] submit_bio
|
--- submit_bio
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] __ata_scsi_queuecmd
|
--- __ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% tar [kernel.kallsyms] [k] dm_table_put
|
--- dm_table_put
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.06% sleep [kernel.kallsyms] [k] __ext4_journal_get_write_access
|
--- __ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
touch_atime
ext4_file_mmap
mmap_region
do_mmap_pgoff
sys_mmap_pgoff
sys_mmap
system_call_fastpath
__mmap
_nl_find_locale
__GI_setlocale
0x40124c
0.05% tar [kernel.kallsyms] [k] sysret_check
|
--- sysret_check
__GI___libc_read
0.05% swapper [kernel.kallsyms] [k] clockevents_program_event
|
--- clockevents_program_event
tick_dev_program_event
tick_program_event
__hrtimer_start_range_ns
hrtimer_start_range_ns
hrtimer_start_expires.constprop.1
tick_nohz_restart_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.05% tar [kernel.kallsyms] [k] pick_next_task_rt
|
--- pick_next_task_rt
pick_next_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% swapper [kernel.kallsyms] [k] account_entity_enqueue
|
--- account_entity_enqueue
enqueue_task
activate_task
try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.05% tar [kernel.kallsyms] [k] need_resched
|
--- need_resched
_cond_resched
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] find_lock_page
|
--- find_lock_page
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] memcmp
|
--- memcmp
search_dirblock
ext4_dx_find_entry
ext4_find_entry
ext4_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.05% tar [kernel.kallsyms] [k] generic_segment_checks
|
--- generic_segment_checks
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% tar [kernel.kallsyms] [k] update_vsyscall
|
--- update_vsyscall
do_timer
tick_do_update_jiffies64
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
_raw_spin_lock
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% swapper [kernel.kallsyms] [k] hrtimer_start_expires.constprop.1
|
--- hrtimer_start_expires.constprop.1
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.05% tar [kernel.kallsyms] [k] __page_cache_alloc
|
--- __page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] avc_has_perm_noaudit
|
--- avc_has_perm_noaudit
avc_has_perm
inode_has_perm
selinux_inode_permission
security_inode_exec_permission
exec_permission
link_path_walk
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.05% tar [kernel.kallsyms] [k] deactivate_slab
|
--- deactivate_slab
__slab_alloc
kmem_cache_alloc
alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% swapper [kernel.kallsyms] [k] bio_free
|
--- bio_free
bio_fs_destructor
bio_put
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.05% tar libc-2.13.90.so [.] __GI___libc_read
|
--- __GI___libc_read
0.05% tar [kernel.kallsyms] [k] selinux_capable
|
--- selinux_capable
ns_capable
capable
inode_change_ok
ext4_setattr
notify_change
chown_common
sys_fchown
system_call_fastpath
__GI___fchown
0.05% tar [kernel.kallsyms] [k] _raw_read_lock_irqsave
|
--- _raw_read_lock_irqsave
dm_get_live_table
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% swapper [kernel.kallsyms] [k] scsi_handle_queue_ramp_up
|
--- scsi_handle_queue_ramp_up
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.05% tar [kernel.kallsyms] [k] dm_table_any_congested
|
--- dm_table_any_congested
dm_any_congested
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% tar [kernel.kallsyms] [k] __wake_up_common
|
--- __wake_up_common
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] acct_update_integrals
|
--- acct_update_integrals
__account_system_time
irqtime_account_process_tick
account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
blk_finish_plug
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% swapper [kernel.kallsyms] [k] ns_to_timespec
|
--- ns_to_timespec
intel_idle
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.05% tar [kernel.kallsyms] [k] _raw_read_unlock_irqrestore
|
--- _raw_read_unlock_irqrestore
dm_get_live_table
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% tar [kernel.kallsyms] [k] __add_bdi_stat
|
--- __add_bdi_stat
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] sg_init_table
|
--- sg_init_table
scsi_alloc_sgtable
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.05% swapper [kernel.kallsyms] [k] getnstimeofday
|
--- getnstimeofday
ktime_get_real
intel_idle
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.04% tar [kernel.kallsyms] [k] get_page
|
--- get_page
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% tar [kernel.kallsyms] [k] timekeeping_get_ns
|
--- timekeeping_get_ns
delayacct_end
__delayacct_blkio_end
delayacct_blkio_end
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% tar [kernel.kallsyms] [k] trylock_page
|
--- trylock_page
lock_page
find_lock_page
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.04% swapper [kernel.kallsyms] [k] native_sched_clock
|
--- native_sched_clock
sched_clock_cpu
account_system_vtime
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
start_secondary
0.04% tar [kernel.kallsyms] [k] native_load_tls
|
--- native_load_tls
__GI___libc_read
0.04% tar [kernel.kallsyms] [k] files_lglock_local_unlock_cpu
|
--- files_lglock_local_unlock_cpu
file_sb_list_del
fput
filp_close
sys_close
system_call_fastpath
__GI___close
0.04% tar [kernel.kallsyms] [k] release_pages
|
--- release_pages
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.04% jbd2/dm-2-8 [kernel.kallsyms] [k] bit_spin_lock.constprop.22
|
--- bit_spin_lock.constprop.22
__slab_free
kmem_cache_free
__journal_remove_journal_head
jbd2_journal_remove_journal_head
journal_clean_one_cp_list
__jbd2_journal_clean_checkpoint_list
jbd2_journal_commit_transaction
kjournald2
kthread
kernel_thread_helper
0.04% tar [kernel.kallsyms] [k] pick_next_task_fair
|
--- pick_next_task_fair
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% swapper [kernel.kallsyms] [k] ktime_get_real
|
--- ktime_get_real
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.04% tar [kernel.kallsyms] [k] tick_do_update_jiffies64
|
--- tick_do_update_jiffies64
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.03% swapper [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
arch_local_irq_save
irqtime_account_process_tick
account_idle_ticks
tick_nohz_restart_sched_tick
cpu_idle
start_secondary
0.03% sleep [kernel.kallsyms] [k] mem_cgroup_uncharge_start
|
--- mem_cgroup_uncharge_start
exit_mmap
mmput
flush_old_exec
load_elf_binary
search_binary_handler
do_execve
sys_execve
stub_execve
__execve
0.02% swapper [kernel.kallsyms] [k] irq_enter
|
--- irq_enter
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
start_secondary
#
# (For a higher level overview, try: perf report --sort comm,dso)
#
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 16:50 ` James Bottomley
@ 2011-04-28 16:56 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 16:56 UTC (permalink / raw)
To: Mel Gorman
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 11:50 -0500, James Bottomley wrote:
> This is the output of perf record -g -a -f sleep 5
>
> (hopefully the list won't choke)
Um, this one actually shows kswapd
James
---
# Events: 6K cycles
#
# Overhead Command Shared Object Symbol
# ........ ........... ................... .......................................
#
20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab
|
--- shrink_slab
|
|--99.91%-- kswapd
| kthread
| kernel_thread_helper
--0.09%-- [...]
9.98% kswapd0 [kernel.kallsyms] [k] shrink_zone
|
--- shrink_zone
|
|--99.46%-- kswapd
| kthread
| kernel_thread_helper
|
--0.54%-- kthread
kernel_thread_helper
7.70% kswapd0 [kernel.kallsyms] [k] kswapd
|
--- kswapd
kthread
kernel_thread_helper
5.40% kswapd0 [kernel.kallsyms] [k] zone_watermark_ok_safe
|
--- zone_watermark_ok_safe
|
|--72.66%-- kswapd
| kthread
| kernel_thread_helper
|
|--20.88%-- sleeping_prematurely.part.12
| kswapd
| kthread
| kernel_thread_helper
|
--6.46%-- kthread
kernel_thread_helper
4.25% kswapd0 [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
|
|--77.49%-- _raw_spin_lock
| |
| |--51.85%-- mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --48.15%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--19.47%-- _raw_spin_lock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--1.73%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--0.88%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
--0.44%-- [...]
3.34% kswapd0 [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
|
|--48.58%-- _raw_spin_unlock
| |
| |--80.77%-- mem_cgroup_soft_limit_reclaim
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --19.23%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
|--17.63%-- _raw_spin_unlock_irqrestore
| |
| |--52.96%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --47.04%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--13.83%-- _raw_spin_unlock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--6.67%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--5.54%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--2.76%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--2.24%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--1.68%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
|--0.55%-- irq_exit
| smp_apic_timer_interrupt
| apic_timer_interrupt
| mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
--0.54%-- mem_cgroup_charge_statistics
__mem_cgroup_uncharge_common
mem_cgroup_uncharge_cache_page
__remove_mapping
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
2.96% kswapd0 [kernel.kallsyms] [k] __zone_watermark_ok
|
--- __zone_watermark_ok
|
|--85.77%-- zone_watermark_ok_safe
| |
| |--74.67%-- kswapd
| | kthread
| | kernel_thread_helper
| |
| --25.33%-- sleeping_prematurely.part.12
| kswapd
| kthread
| kernel_thread_helper
|
|--11.78%-- kswapd
| kthread
| kernel_thread_helper
|
--2.45%-- sleeping_prematurely.part.12
kswapd
kthread
kernel_thread_helper
2.67% kswapd0 [kernel.kallsyms] [k] global_dirty_limits
|
--- global_dirty_limits
|
|--97.28%-- throttle_vm_writeout
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--2.72%-- shrink_zone
kswapd
kthread
kernel_thread_helper
2.30% kswapd0 [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--30.33%-- _raw_spin_lock
| |
| |--52.29%-- mem_cgroup_soft_limit_reclaim
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --47.71%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
|--28.79%-- _raw_spin_lock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--24.01%-- _raw_spin_lock_irqsave
| |
| |--63.28%-- finish_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --36.72%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--6.31%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--4.82%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--3.28%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--1.64%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--0.82%-- finish_wait
kswapd
kthread
kernel_thread_helper
2.02% kswapd0 [kernel.kallsyms] [k] up_read
|
--- up_read
|
|--95.48%-- shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--4.52%-- kswapd
kthread
kernel_thread_helper
2.01% kswapd0 [kernel.kallsyms] [k] get_parent_ip
|
--- get_parent_ip
|
|--37.59%-- sub_preempt_count
| |
| |--66.19%-- _raw_spin_unlock
| | |
| | |--55.77%-- mem_cgroup_soft_limit_reclaim
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | --44.23%-- mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| |--17.13%-- _raw_spin_unlock_irqrestore
| | |
| | |--57.38%-- finish_wait
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | --42.62%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --16.68%-- _raw_spin_unlock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--32.16%-- add_preempt_count
| |
| |--71.32%-- _raw_spin_lock
| | |
| | |--56.02%-- mem_cgroup_soft_limit_reclaim
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | --43.98%-- mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| |--17.25%-- _raw_spin_lock_irqsave
| | |
| | |--83.09%-- finish_wait
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | --16.91%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --11.43%-- _raw_spin_lock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--8.20%-- _raw_spin_unlock
| |
| |--66.09%-- mem_cgroup_soft_limit_reclaim
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --33.91%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
|--7.42%-- _raw_spin_lock
| |
| |--62.14%-- mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --37.86%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--5.55%-- _raw_spin_lock_irqsave
| |
| |--66.67%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --33.33%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--4.53%-- _raw_spin_lock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--3.62%-- _raw_spin_unlock_irqrestore
| |
| |--51.20%-- finish_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --48.80%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--0.94%-- _raw_spin_unlock_irq
shrink_zone
kswapd
kthread
kernel_thread_helper
1.88% kswapd0 [kernel.kallsyms] [k] zone_nr_lru_pages
|
--- zone_nr_lru_pages
|
|--85.21%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--14.79%-- kswapd
kthread
kernel_thread_helper
1.79% kswapd0 [kernel.kallsyms] [k] down_read_trylock
|
--- down_read_trylock
|
|--92.80%-- shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--7.20%-- kswapd
kthread
kernel_thread_helper
1.69% kswapd0 [kernel.kallsyms] [k] mutex_trylock
|
--- mutex_trylock
i915_gem_inactive_shrink
shrink_slab
kswapd
kthread
kernel_thread_helper
1.66% kswapd0 [kernel.kallsyms] [k] sleeping_prematurely.part.12
|
--- sleeping_prematurely.part.12
|
|--93.41%-- kswapd
| kthread
| kernel_thread_helper
|
--6.59%-- kthread
kernel_thread_helper
1.54% kswapd0 [kernel.kallsyms] [k] find_next_bit
|
--- find_next_bit
|
|--84.45%-- cpumask_next
| zone_watermark_ok_safe
| kswapd
| kthread
| kernel_thread_helper
|
--15.55%-- zone_watermark_ok_safe
kswapd
kthread
kernel_thread_helper
1.51% kswapd0 [kernel.kallsyms] [k] throttle_vm_writeout
|
--- throttle_vm_writeout
shrink_zone
kswapd
kthread
kernel_thread_helper
1.48% kswapd0 [kernel.kallsyms] [k] mb_cache_shrink_fn
|
--- mb_cache_shrink_fn
|
|--96.31%-- shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--3.69%-- kswapd
kthread
kernel_thread_helper
1.31% kswapd0 [kernel.kallsyms] [k] mem_cgroup_soft_limit_reclaim
|
--- mem_cgroup_soft_limit_reclaim
|
|--97.19%-- kswapd
| kthread
| kernel_thread_helper
|
--2.81%-- kthread
kernel_thread_helper
1.26% kswapd0 [kernel.kallsyms] [k] mutex_unlock
|
--- mutex_unlock
|
|--98.57%-- i915_gem_inactive_shrink
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--1.43%-- shrink_slab
kswapd
kthread
kernel_thread_helper
1.19% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
|
|--47.70%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--47.65%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--4.65%-- kswapd
kthread
kernel_thread_helper
1.14% kswapd0 [kernel.kallsyms] [k] _raw_spin_unlock
|
--- _raw_spin_unlock
|
|--59.71%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--38.63%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--1.66%-- kswapd
kthread
kernel_thread_helper
0.96% kswapd0 [kernel.kallsyms] [k] __mem_cgroup_largest_soft_limit_node
|
--- __mem_cgroup_largest_soft_limit_node
|
|--59.47%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
--40.53%-- kswapd
kthread
kernel_thread_helper
0.88% kswapd0 [kernel.kallsyms] [k] in_lock_functions
|
--- in_lock_functions
|
|--43.80%-- get_parent_ip
| |
| |--52.47%-- add_preempt_count
| | |
| | |--72.99%-- _raw_spin_lock
| | | |
| | | |--62.71%-- mem_cgroup_soft_limit_reclaim
| | | | kswapd
| | | | kthread
| | | | kernel_thread_helper
| | | |
| | | --37.29%-- mb_cache_shrink_fn
| | | shrink_slab
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | |--18.30%-- _raw_spin_lock_irq
| | | shrink_zone
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | --8.71%-- _raw_spin_lock_irqsave
| | prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --47.53%-- sub_preempt_count
| |
| |--49.81%-- _raw_spin_unlock
| | mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| |--30.32%-- _raw_spin_unlock_irqrestore
| | finish_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --19.86%-- _raw_spin_unlock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--29.12%-- add_preempt_count
| |
| |--49.50%-- _raw_spin_lock
| | |
| | |--56.31%-- mem_cgroup_soft_limit_reclaim
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | --43.69%-- mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| |--29.19%-- _raw_spin_lock_irq
| | shrink_zone
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --21.32%-- _raw_spin_lock_irqsave
| |
| |--65.66%-- finish_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --34.34%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--27.07%-- sub_preempt_count
|
|--62.26%-- _raw_spin_unlock
| |
| |--50.27%-- mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --49.73%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--22.85%-- _raw_spin_unlock_irqrestore
| |
| |--67.34%-- finish_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --32.66%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--14.89%-- _raw_spin_unlock_irq
shrink_zone
kswapd
kthread
kernel_thread_helper
0.81% kswapd0 [kernel.kallsyms] [k] cpumask_next
|
--- cpumask_next
|
|--79.81%-- zone_watermark_ok_safe
| kswapd
| kthread
| kernel_thread_helper
|
--20.19%-- kswapd
kthread
kernel_thread_helper
0.79% kswapd0 [kernel.kallsyms] [k] zone_reclaimable_pages
|
--- zone_reclaimable_pages
|
|--93.08%-- kswapd
| kthread
| kernel_thread_helper
|
--6.92%-- kthread
kernel_thread_helper
0.77% kswapd0 [i915] [k] i915_gem_inactive_shrink
|
--- i915_gem_inactive_shrink
|
|--97.55%-- shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--2.45%-- kswapd
kthread
kernel_thread_helper
0.76% kswapd0 [kernel.kallsyms] [k] prepare_to_wait
|
--- prepare_to_wait
|
|--97.54%-- kswapd
| kthread
| kernel_thread_helper
|
--2.46%-- kthread
kernel_thread_helper
0.74% kswapd0 [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
|
|--79.88%-- _raw_spin_unlock_irqrestore
| |
| |--62.49%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --37.51%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--15.16%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--4.96%-- finish_wait
kswapd
kthread
kernel_thread_helper
0.65% kswapd0 [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6
|
--- test_ti_thread_flag.constprop.6
|
|--20.12%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
|--20.09%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--14.28%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--14.16%-- _raw_spin_unlock
| |
| |--80.68%-- mem_cgroup_soft_limit_reclaim
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --19.32%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
|--11.40%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--8.63%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--8.58%-- _raw_spin_unlock_irqrestore
| |
| |--66.13%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --33.87%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
--2.74%-- _raw_spin_unlock_irq
shrink_zone
kswapd
kthread
kernel_thread_helper
0.63% tar [kernel.kallsyms] [k] copy_user_generic_string
|
--- copy_user_generic_string
|
|--56.11%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--43.89%-- generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.59% kswapd0 [kernel.kallsyms] [k] test_tsk_thread_flag
|
--- test_tsk_thread_flag
|
|--50.16%-- kswapd
| kthread
| kernel_thread_helper
|
|--46.73%-- try_to_freeze
| kswapd
| kthread
| kernel_thread_helper
|
--3.11%-- kthread
kernel_thread_helper
0.59% kswapd0 [kernel.kallsyms] [k] shrink_icache_memory
|
--- shrink_icache_memory
|
|--81.19%-- shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--18.81%-- kswapd
kthread
kernel_thread_helper
0.52% kswapd0 [kernel.kallsyms] [k] shrink_dqcache_memory
|
--- shrink_dqcache_memory
|
|--89.35%-- shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--10.65%-- kswapd
kthread
kernel_thread_helper
0.50% kswapd0 [kernel.kallsyms] [k] zone_clear_flag
|
--- zone_clear_flag
|
|--77.78%-- kswapd
| kthread
| kernel_thread_helper
|
--22.22%-- kthread
kernel_thread_helper
0.45% kswapd0 [kernel.kallsyms] [k] kthread_should_stop
|
--- kthread_should_stop
|
|--54.28%-- kswapd
| kthread
| kernel_thread_helper
|
--45.72%-- kthread
kernel_thread_helper
0.44% kswapd0 [kernel.kallsyms] [k] global_page_state
|
--- global_page_state
|
|--33.60%-- throttle_vm_writeout
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--29.47%-- determine_dirtyable_memory
| global_dirty_limits
| throttle_vm_writeout
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--20.52%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--16.41%-- global_dirty_limits
throttle_vm_writeout
shrink_zone
kswapd
kthread
kernel_thread_helper
0.44% kswapd0 [kernel.kallsyms] [k] shrink_dcache_memory
|
--- shrink_dcache_memory
shrink_slab
kswapd
kthread
kernel_thread_helper
0.41% kswapd0 [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore
|
--- _raw_spin_unlock_irqrestore
|
|--59.01%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--40.99%-- finish_wait
kswapd
kthread
kernel_thread_helper
0.39% kswapd0 [kernel.kallsyms] [k] global_reclaimable_pages
|
--- global_reclaimable_pages
determine_dirtyable_memory
global_dirty_limits
throttle_vm_writeout
shrink_zone
kswapd
kthread
kernel_thread_helper
0.37% kswapd0 [kernel.kallsyms] [k] finish_wait
|
--- finish_wait
|
|--95.03%-- kswapd
| kthread
| kernel_thread_helper
|
--4.97%-- kthread
kernel_thread_helper
0.33% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock
|
--- _raw_spin_lock
|
|--60.70%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
--39.30%-- mb_cache_shrink_fn
shrink_slab
kswapd
kthread
kernel_thread_helper
0.31% kswapd0 [kernel.kallsyms] [k] arch_local_irq_enable
|
--- arch_local_irq_enable
|
|--76.94%-- _raw_spin_unlock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--23.06%-- shrink_zone
kswapd
kthread
kernel_thread_helper
0.28% kswapd0 [kernel.kallsyms] [k] arch_local_irq_save
|
--- arch_local_irq_save
|
|--93.18%-- _raw_spin_lock_irqsave
| |
| |--64.21%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --35.79%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
--6.82%-- finish_wait
kswapd
kthread
kernel_thread_helper
0.22% kswapd0 [kernel.kallsyms] [k] _raw_spin_unlock_irq
|
--- _raw_spin_unlock_irq
|
|--92.00%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--8.00%-- __remove_mapping
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.22% kswapd0 [kernel.kallsyms] [k] arch_local_irq_disable
|
--- arch_local_irq_disable
|
|--58.81%-- arch_local_irq_save
| _raw_spin_lock_irqsave
| |
| |--70.97%-- finish_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --29.03%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--33.06%-- _raw_spin_lock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--8.13%-- shrink_zone
kswapd
kthread
kernel_thread_helper
0.22% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock_irq
|
--- _raw_spin_lock_irq
|
|--58.62%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--33.19%-- kswapd
| kthread
| kernel_thread_helper
|
--8.19%-- __remove_mapping
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.22% swapper [kernel.kallsyms] [k] poll_idle
|
--- poll_idle
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.20% kswapd0 [kernel.kallsyms] [k] __list_add
|
--- __list_add
|
|--90.97%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--9.03%-- free_hot_cold_page
__pagevec_free
free_page_list
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.15% tar [kernel.kallsyms] [k] zero_user_segments
|
--- zero_user_segments
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.15% kswapd0 [kernel.kallsyms] [k] rb_last
|
--- rb_last
mem_cgroup_soft_limit_reclaim
kswapd
kthread
kernel_thread_helper
0.14% swapper [kernel.kallsyms] [k] intel_idle
|
--- intel_idle
cpuidle_idle_call
cpu_idle
|
|--65.05%-- rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--34.95%-- start_secondary
0.13% kswapd0 [kernel.kallsyms] [k] get_reclaim_stat
|
--- get_reclaim_stat
|
|--57.55%-- kswapd
| kthread
| kernel_thread_helper
|
--42.45%-- shrink_zone
kswapd
kthread
kernel_thread_helper
0.12% perf [kernel.kallsyms] [k] selinux_file_alloc_security
|
--- selinux_file_alloc_security
security_file_alloc
get_empty_filp
path_openat
do_filp_open
do_sys_open
sys_open
system_call_fastpath
__GI___libc_open
0x429beb
0x418709
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.12% tar [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--18.90%-- _raw_spin_lock
| |
| |--63.01%-- ext4_da_get_block_prep
| | __block_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| |--20.46%-- get_page_from_freelist
| | __alloc_pages_nodemask
| | alloc_pages_current
| | __page_cache_alloc
| | grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --16.53%-- ext4_ext_map_blocks
| ext4_map_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--13.68%-- __lru_cache_add
| add_to_page_cache_lru
| |
| |--51.23%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --48.77%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--13.32%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--12.87%-- alloc_buffer_head
| alloc_page_buffers
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--9.79%-- _raw_spin_lock_irq
| |
| |--62.37%-- add_to_page_cache_locked
| | add_to_page_cache_lru
| | grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --37.63%-- __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--7.52%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.32%-- lock_page_cgroup
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--3.69%-- __percpu_counter_add
| __add_bdi_stat
| account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--3.54%-- bit_spin_lock
| lock_page_cgroup
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--3.41%-- create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--3.32%-- _raw_read_lock_irqsave
| dm_get_live_table
| __split_and_process_bio
| dm_request
| generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--3.29%-- _raw_read_lock
| start_this_handle
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--2.33%-- dput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.11% kswapd0 [kernel.kallsyms] [k] try_to_freeze
|
--- try_to_freeze
|
|--83.76%-- kswapd
| kthread
| kernel_thread_helper
|
--16.24%-- kthread
kernel_thread_helper
0.10% swapper [kernel.kallsyms] [k] ahci_interrupt
|
--- ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.09% kswapd0 [kernel.kallsyms] [k] determine_dirtyable_memory
|
--- determine_dirtyable_memory
|
|--80.46%-- global_dirty_limits
| throttle_vm_writeout
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--19.54%-- throttle_vm_writeout
shrink_zone
kswapd
kthread
kernel_thread_helper
0.08% tar [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
|
|--27.37%-- _raw_spin_unlock
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--12.90%-- blk_dequeue_request
| blk_start_request
| scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--9.76%-- bit_spin_unlock
| unlock_page_cgroup
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--6.85%-- _raw_spin_unlock_irq
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--6.81%-- alloc_buffer_head
| alloc_page_buffers
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--5.86%-- _raw_spin_unlock_irqrestore
| pagevec_lru_move_fn
| ____pagevec_lru_add
| __lru_cache_add
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--5.51%-- _raw_read_unlock
| start_this_handle
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--5.37%-- jbd_unlock_bh_state
| do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_free_inode
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--4.68%-- deactivate_slab
| __slab_alloc
| kmem_cache_alloc
| alloc_buffer_head
| alloc_page_buffers
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.15%-- __srcu_read_unlock
| fsnotify
| fsnotify_modify
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--3.72%-- radix_tree_preload_end
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--3.62%-- bit_spin_unlock
| jbd2_journal_add_journal_head
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--3.41%-- pagefault_enable
iov_iter_copy_from_user_atomic
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.08% swapper [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--52.45%-- _raw_spin_lock
| get_next_timer_interrupt
| tick_nohz_stop_sched_tick
| irq_exit
| smp_call_function_single_interrupt
| call_function_single_interrupt
| cpuidle_idle_call
| cpu_idle
| start_secondary
|
|--26.09%-- bit_spin_lock.constprop.22
| __slab_free
| kmem_cache_free
| ext4_free_io_end
| ext4_end_bio
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--21.46%-- __percpu_counter_add
__prop_inc_percpu_max
test_clear_page_writeback
end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.08% tar [kernel.kallsyms] [k] kmem_cache_alloc
|
--- kmem_cache_alloc
|
|--74.26%-- alloc_buffer_head
| alloc_page_buffers
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--12.98%-- jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--6.88%-- mempool_alloc_slab
| mempool_alloc
| scsi_sg_alloc
| __sg_alloc_table
| scsi_alloc_sgtable
| scsi_init_sgtable
| scsi_init_io
| scsi_setup_fs_cmnd
| sd_prep_fn
| blk_peek_request
| scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--5.87%-- selinux_inode_alloc_security
security_inode_alloc
inode_init_always
alloc_inode
new_inode
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.07% kswapd0 [kernel.kallsyms] [k] __free_one_page
|
--- __free_one_page
free_pcppages_bulk
free_hot_cold_page
__pagevec_free
free_page_list
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.07% swapper [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
|
|--56.43%-- __percpu_counter_add
| __add_bdi_stat
| test_clear_page_writeback
| end_page_writeback
| put_io_page
| ext4_end_bio
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--17.99%-- _raw_spin_unlock_irqrestore
| __hrtimer_start_range_ns
| hrtimer_start_range_ns
| hrtimer_start_expires.constprop.1
| tick_nohz_restart_sched_tick
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--15.61%-- _raw_spin_unlock
| |
| |--55.21%-- handle_irq_event
| | handle_edge_irq
| | handle_irq
| | do_IRQ
| | common_interrupt
| | cpuidle_idle_call
| | cpu_idle
| | rest_init
| | start_kernel
| | x86_64_start_reservations
| | x86_64_start_kernel
| |
| --44.79%-- scheduler_tick
| update_process_times
| tick_sched_timer
| __run_hrtimer
| hrtimer_interrupt
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--5.91%-- try_to_wake_up
| default_wake_function
| autoremove_wake_function
| wake_bit_function
| __wake_up_common
| __wake_up
| __wake_up_bit
| unlock_page
| mpage_end_io
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--4.06%-- default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] mark_page_accessed
|
--- mark_page_accessed
|
|--87.24%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--6.42%-- __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--6.35%-- __find_get_block
__getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] get_page_from_freelist
|
--- get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--54.79%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--45.21%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] __ext4_get_inode_loc
|
--- __ext4_get_inode_loc
ext4_get_inode_loc
|
|--55.97%-- ext4_reserve_inode_write
| ext4_mark_inode_dirty
| |
| |--76.52%-- ext4_dirty_inode
| | __mark_inode_dirty
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --23.48%-- ext4_ext_dirty
| ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--44.03%-- ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar tar [.] 0x2aa04
|
--- 0x40753f
|
--- 0x4055c7
|
--- 0x42aa04
|
--- 0x4166d8
0x417081
|
--- 0x40fdfb
|
--- 0x405f26
|
--- 0x4332d6
|
--- 0x40fe1a
|
--- 0x408ab1
|
--- 0x40fdb8
|
--- 0x408a87
|
--- 0x4100c5
|
--- 0x407564
0.06% perf [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__write_nocancel
0x4293b8
0x429c0a
0x418709
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.05% tar [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
|
|--79.42%-- _raw_spin_lock_irq
| |
| |--57.06%-- __set_page_dirty
| | mark_buffer_dirty
| | __block_commit_write
| | block_write_end
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --42.94%-- add_to_page_cache_locked
| add_to_page_cache_lru
| |
| |--52.51%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --47.49%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--15.99%-- _raw_spin_lock
| |
| |--60.63%-- ext4_ext_map_blocks
| | ext4_map_blocks
| | ext4_da_get_block_prep
| | __block_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --39.37%-- inode_add_rsv_space
| __dquot_alloc_space
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--4.59%-- inode_add_rsv_space
__dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] __mem_cgroup_commit_charge.constprop.28
|
--- __mem_cgroup_commit_charge.constprop.28
|
|--91.58%-- __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| |
| |--51.67%-- mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --48.33%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--8.42%-- mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] radix_tree_lookup_element
|
--- radix_tree_lookup_element
|
|--69.74%-- radix_tree_lookup_slot
| find_get_page
| |
| |--58.70%-- generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --41.30%-- find_lock_page
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--17.45%-- radix_tree_lookup
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--12.81%-- find_get_page
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% tar [kernel.kallsyms] [k] __alloc_pages_nodemask
|
--- __alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--71.39%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--28.61%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% tar [kernel.kallsyms] [k] put_mems_allowed
|
--- put_mems_allowed
alloc_pages_current
__page_cache_alloc
|
|--70.79%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--29.21%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% swapper [kernel.kallsyms] [k] load_balance
|
--- load_balance
rebalance_domains
run_rebalance_domains
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
start_secondary
0.04% flush-253:2 [kernel.kallsyms] [k] __bio_clone
|
--- __bio_clone
clone_bio
__split_and_process_bio
dm_request
generic_make_request
submit_bio
ext4_io_submit
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.04% tar [kernel.kallsyms] [k] __wake_up_bit
|
--- __wake_up_bit
|
|--89.40%-- unlock_page
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--10.60%-- wake_up_bit
unlock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.04% tar [kernel.kallsyms] [k] get_parent_ip
|
--- get_parent_ip
|
|--29.28%-- sub_preempt_count
| |
| |--59.65%-- pagefault_enable
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --40.35%-- bit_spin_unlock
| jbd2_journal_put_journal_head
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--19.16%-- avc_has_perm_noaudit
| avc_has_perm
| inode_has_perm
| selinux_inode_permission
| security_inode_exec_permission
| exec_permission
| link_path_walk
| path_openat
| do_filp_open
| do_sys_open
| sys_openat
| system_call_fastpath
| __GI___openat
|
|--19.01%-- _raw_spin_lock
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--17.20%-- bit_spin_lock
| lock_page_cgroup
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--15.36%-- add_preempt_count
file_read_actor
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% tar [kernel.kallsyms] [k] ext4_mark_iloc_dirty
|
--- ext4_mark_iloc_dirty
ext4_mark_inode_dirty
|
|--77.33%-- ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--13.35%-- ext4_new_inode
| ext4_create
| vfs_create
| do_last
| path_openat
| do_filp_open
| do_sys_open
| sys_openat
| system_call_fastpath
| __GI___openat
|
--9.32%-- ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.04% kswapd0 [kernel.kallsyms] [k] native_write_msr_safe
|
--- native_write_msr_safe
paravirt_write_msr
intel_pmu_disable_all
x86_pmu_disable
perf_pmu_disable
perf_event_task_tick
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
|
|--50.07%-- zone_watermark_ok_safe
| sleeping_prematurely.part.12
| kswapd
| kthread
| kernel_thread_helper
|
--49.93%-- kswapd
kthread
kernel_thread_helper
0.04% swapper [kernel.kallsyms] [k] ext4_end_bio
|
--- ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.04% kswapd0 [kernel.kallsyms] [k] __list_del_entry
|
--- __list_del_entry
finish_wait
kswapd
kthread
kernel_thread_helper
0.04% kswapd0 [kernel.kallsyms] [k] mcount
|
--- mcount
_raw_spin_lock_irq
shrink_zone
kswapd
kthread
kernel_thread_helper
0.04% tar [kernel.kallsyms] [k] __percpu_counter_add
|
--- __percpu_counter_add
|
|--87.00%-- ext4_claim_free_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--13.00%-- __prop_inc_single
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.04% tar [kernel.kallsyms] [k] arch_read_unlock
|
--- arch_read_unlock
|
|--57.64%-- _raw_read_unlock_irqrestore
| dm_get_live_table
| dm_merge_bvec
| __bio_add_page.part.2
| bio_add_page
| do_mpage_readpage
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--31.54%-- _raw_read_unlock
| start_this_handle
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--10.82%-- start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.04% kswapd0 [kernel.kallsyms] [k] mem_cgroup_del_lru_list
|
--- mem_cgroup_del_lru_list
mem_cgroup_del_lru
isolate_lru_pages
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.04% kswapd0 [kernel.kallsyms] [k] test_and_set_bit
|
--- test_and_set_bit
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.04% tar [kernel.kallsyms] [k] __might_sleep
|
--- __might_sleep
|
|--22.45%-- kmem_cache_alloc
| |
| |--59.27%-- getname_flags
| | getname
| | do_sys_open
| | sys_openat
| | system_call_fastpath
| | __GI___openat
| |
| --40.73%-- jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--22.30%-- __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--54.37%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --45.63%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--15.00%-- lock_buffer
| do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--11.33%-- __getblk
| __ext4_get_inode_loc
| ext4_get_inode_loc
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--11.31%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--10.36%-- unmap_underlying_metadata
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--7.25%-- __generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.04% kswapd0 [kernel.kallsyms] [k] debug_smp_processor_id
|
--- debug_smp_processor_id
|
|--50.69%-- __pagevec_free
| free_page_list
| shrink_page_list
| shrink_inactive_list
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--49.31%-- free_hot_cold_page
__pagevec_free
free_page_list
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.04% kswapd0 [kernel.kallsyms] [k] __mem_cgroup_uncharge_common
|
--- __mem_cgroup_uncharge_common
mem_cgroup_uncharge_cache_page
__remove_mapping
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.04% kswapd0 [kernel.kallsyms] [k] tag_get
|
--- tag_get
|
|--50.59%-- radix_tree_delete
| __delete_from_page_cache
| __remove_mapping
| shrink_page_list
| shrink_inactive_list
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--49.41%-- __delete_from_page_cache
__remove_mapping
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.03% flush-253:2 [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--51.77%-- _raw_read_lock_irqsave
| dm_get_live_table
| dm_merge_bvec
| __bio_add_page.part.2
| bio_add_page
| ext4_bio_write_page
| mpage_da_submit_io
| mpage_da_map_and_submit
| ext4_da_writepages
| do_writepages
| writeback_single_inode
| writeback_sb_inodes
| writeback_inodes_wb
| wb_writeback
| wb_do_writeback
| bdi_writeback_thread
| kthread
| kernel_thread_helper
|
--48.23%-- blk_throtl_bio
generic_make_request
submit_bio
ext4_io_submit
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% tar [kernel.kallsyms] [k] __inc_zone_state
|
--- __inc_zone_state
|
|--49.98%-- __inc_zone_page_state
| add_to_page_cache_locked
| add_to_page_cache_lru
| |
| |--79.48%-- mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --20.52%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--32.04%-- zone_statistics
| get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--17.98%-- account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.03% tar [kernel.kallsyms] [k] __list_del_entry
|
--- __list_del_entry
|
|--56.86%-- list_del
| __rmqueue
| get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--78.10%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --21.90%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--43.14%-- __rmqueue
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.03% flush-253:2 [kernel.kallsyms] [k] dm_table_find_target
|
--- dm_table_find_target
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% tar [kernel.kallsyms] [k] jbd_lock_bh_state
|
--- jbd_lock_bh_state
|
|--61.84%-- do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| |
| |--72.93%-- generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --27.07%-- ext4_free_blocks
| ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--38.16%-- jbd2_journal_dirty_metadata
__ext4_handle_dirty_metadata
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.03% flush-253:2 [kernel.kallsyms] [k] page_waitqueue
|
--- page_waitqueue
ext4_num_dirty_pages
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% flush-253:2 [kernel.kallsyms] [k] radix_tree_lookup_element
|
--- radix_tree_lookup_element
radix_tree_lookup_slot
find_get_page
__find_get_block_slow
unmap_underlying_metadata
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% tar [kernel.kallsyms] [k] fcheck_files
|
--- fcheck_files
fget_light
|
|--71.87%-- sys_read
| system_call_fastpath
| __GI___libc_read
|
--28.13%-- sys_write
system_call_fastpath
__GI___libc_write
0.03% flush-253:2 [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% flush-253:2 [kernel.kallsyms] [k] lock_page
|
--- lock_page
ext4_num_dirty_pages
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% tar [kernel.kallsyms] [k] put_page_testzero
|
--- put_page_testzero
|
|--70.70%-- release_pages
| pagevec_lru_move_fn
| ____pagevec_lru_add
| __lru_cache_add
| add_to_page_cache_lru
| |
| |--77.62%-- mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --22.38%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--29.30%-- put_page
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.03% tar [kernel.kallsyms] [k] zone_watermark_ok
|
--- zone_watermark_ok
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--84.29%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--15.71%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.03% tar [kernel.kallsyms] [k] start_this_handle
|
--- start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
|
|--86.30%-- ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--13.70%-- ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.03% flush-253:2 [kernel.kallsyms] [k] ext4_num_dirty_pages
|
--- ext4_num_dirty_pages
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% tar [kernel.kallsyms] [k] put_mems_allowed
|
--- put_mems_allowed
|
|--74.79%-- __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--65.17%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --34.83%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--25.21%-- alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.03% tar [kernel.kallsyms] [k] debug_smp_processor_id
|
--- debug_smp_processor_id
|
|--57.28%-- load_balance
| schedule
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--25.57%-- radix_tree_preload
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--17.15%-- get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.03% swapper [kernel.kallsyms] [k] __wake_up_bit
|
--- __wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.03% tar [kernel.kallsyms] [k] avc_has_perm_noaudit
|
--- avc_has_perm_noaudit
avc_has_perm
|
|--82.82%-- inode_has_perm
| |
| |--71.72%-- dentry_has_perm
| | selinux_inode_getattr
| | security_inode_getattr
| | vfs_getattr
| | vfs_fstat
| | sys_newfstat
| | system_call_fastpath
| | __GI___fxstat64
| |
| --28.28%-- selinux_inode_permission
| security_inode_exec_permission
| exec_permission
| link_path_walk
| path_openat
| do_filp_open
| do_sys_open
| sys_openat
| system_call_fastpath
| __GI___openat
|
--17.18%-- may_create
selinux_inode_create
security_inode_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.03% tar [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.03% tar [kernel.kallsyms] [k] do_mpage_readpage
|
--- do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.03% tar [kernel.kallsyms] [k] list_del
|
--- list_del
|
|--55.44%-- get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--67.14%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --32.86%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--28.53%-- ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--16.02%-- __alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] __rmqueue
|
--- __rmqueue
|
|--77.01%-- get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--22.99%-- __alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] __mem_cgroup_commit_charge_lrucare
|
--- __mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--83.13%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--16.87%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] update_page_reclaim_stat
|
--- update_page_reclaim_stat
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
|
|--75.26%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--24.74%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.02% tar [kernel.kallsyms] [k] radix_tree_insert
|
--- radix_tree_insert
add_to_page_cache_locked
add_to_page_cache_lru
|
|--81.25%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--18.75%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] add_to_page_cache_locked
|
--- add_to_page_cache_locked
add_to_page_cache_lru
|
|--62.06%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--37.94%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] page_waitqueue
|
--- page_waitqueue
|
|--58.60%-- generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--41.40%-- unlock_page
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] up_read
|
--- up_read
|
|--52.38%-- ext4_map_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.62%-- ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% swapper [kernel.kallsyms] [k] elv_completed_request
|
--- elv_completed_request
blk_finish_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% tar [kernel.kallsyms] [k] arch_local_irq_save
|
--- arch_local_irq_save
|
|--84.09%-- task_dirty_inc
| account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--15.91%-- account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] ext4_map_blocks
|
--- ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% flush-253:2 [kernel.kallsyms] [k] mcount
|
--- mcount
_raw_spin_unlock_irqrestore
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] clear_buffer_dirty
|
--- clear_buffer_dirty
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] mem_cgroup_add_lru_list
|
--- mem_cgroup_add_lru_list
add_page_to_lru_list
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.02% swapper [kernel.kallsyms] [k] blk_complete_request
|
--- blk_complete_request
scsi_done
ata_scsi_qc_complete
__ata_qc_complete
ata_qc_complete
ata_qc_complete_multiple
ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% swapper [kernel.kallsyms] [k] sg_init_table
|
--- sg_init_table
scsi_alloc_sgtable
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
blk_run_queue
scsi_run_queue
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% tar [kernel.kallsyms] [k] _raw_spin_lock
|
--- _raw_spin_lock
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] touch_atime
|
--- touch_atime
|
|--90.35%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--9.65%-- do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.02% perf ld-2.13.90.so [.] _dl_fixup
|
--- _dl_fixup
0.02% tar [kernel.kallsyms] [k] __block_write_begin
|
--- __block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% kswapd0 [kernel.kallsyms] [k] irqtime_account_process_tick
|
--- irqtime_account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
i915_gem_inactive_shrink
shrink_slab
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] apic_timer_interrupt
|
--- apic_timer_interrupt
kswapd
kthread
kernel_thread_helper
0.02% kworker/2:1 [kernel.kallsyms] [k] ns_to_timeval
|
--- ns_to_timeval
get_cpu_iowait_time_us
do_dbs_timer
process_one_work
worker_thread
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] task_rq_lock
|
--- task_rq_lock
try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
zone_watermark_ok_safe
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] hrtimer_run_queues
|
--- hrtimer_run_queues
run_local_timers
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] next_zones_zonelist
|
--- next_zones_zonelist
|
|--71.41%-- get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--28.59%-- __alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% kswapd0 [kernel.kallsyms] [k] arch_local_irq_disable
|
--- arch_local_irq_disable
run_local_timers
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] __rcu_pending
|
--- __rcu_pending
rcu_check_callbacks
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
kswapd
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] write_cache_pages_da
|
--- write_cache_pages_da
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] intel_pmu_disable_all
|
--- intel_pmu_disable_all
x86_pmu_disable
perf_pmu_disable
perf_event_task_tick
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
sleeping_prematurely.part.12
kswapd
kthread
kernel_thread_helper
0.02% swapper [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
arch_local_irq_save
irqtime_account_process_tick
account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% kswapd0 [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
call_rcu
d_free
dentry_kill
shrink_dentry_list
__shrink_dcache_sb
shrink_dcache_memory
shrink_slab
kswapd
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] SetPageLRU
|
--- SetPageLRU
|
|--66.95%-- ____pagevec_lru_add_fn
| pagevec_lru_move_fn
| ____pagevec_lru_add
| __lru_cache_add
| add_to_page_cache_lru
| |
| |--56.88%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --43.12%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--33.05%-- pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.02% kswapd0 [kernel.kallsyms] [k] run_timer_softirq
|
--- run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
i915_gem_inactive_shrink
shrink_slab
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] __isolate_lru_page
|
--- __isolate_lru_page
isolate_lru_pages
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
__might_sleep
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] radix_tree_delete
|
--- radix_tree_delete
__delete_from_page_cache
__remove_mapping
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_clear
|
--- radix_tree_tag_clear
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] ext4_map_blocks
|
--- ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
__might_sleep
|
|--51.15%-- __getblk
| __ext4_get_inode_loc
| ext4_get_inode_loc
| |
| |--50.51%-- ext4_reserve_inode_write
| | ext4_mark_inode_dirty
| | ext4_ext_truncate
| | ext4_truncate
| | ext4_evict_inode
| | evict
| | iput
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| --49.49%-- ext4_xattr_get
| ext4_xattr_security_get
| generic_getxattr
| cap_inode_need_killpriv
| security_inode_need_killpriv
| file_remove_suid
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--48.85%-- kmem_cache_alloc
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% kswapd0 [kernel.kallsyms] [k] mem_cgroup_uncharge_cache_page
|
--- mem_cgroup_uncharge_cache_page
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] update_curr
|
--- update_curr
task_tick_fair
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] timerqueue_del
|
--- timerqueue_del
__remove_hrtimer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
_raw_spin_unlock_irqrestore
prepare_to_wait
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] ack_APIC_irq
|
--- ack_APIC_irq
smp_apic_timer_interrupt
apic_timer_interrupt
kswapd
kthread
kernel_thread_helper
0.02% kworker/2:1 [kernel.kallsyms] [k] too_many_workers
|
--- too_many_workers
worker_enter_idle
worker_thread
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] update_isolated_counts
|
--- update_isolated_counts
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6
|
--- test_ti_thread_flag.constprop.6
_raw_spin_unlock_irqrestore
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] jbd2_journal_get_write_access
|
--- jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_ext_dirty
ext4_ext_insert_extent
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_set
|
--- radix_tree_tag_set
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] __lookup
|
--- __lookup
radix_tree_gang_lookup_slot
find_get_pages
pagevec_lookup
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] page_cache_get_speculative
|
--- page_cache_get_speculative
find_get_pages_tag
pagevec_lookup_tag
ext4_num_dirty_pages
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
_raw_spin_unlock_irqrestore
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] ext4_ext_map_blocks
|
--- ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] jbd_lock_bh_state
|
--- jbd_lock_bh_state
jbd2_journal_dirty_metadata
__ext4_handle_dirty_metadata
ext4_mb_mark_diskspace_used
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
jbd_unlock_bh_state
jbd2_journal_dirty_metadata
__ext4_handle_dirty_metadata
ext4_mb_mark_diskspace_used
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] alloc_pages_current
|
--- alloc_pages_current
__page_cache_alloc
|
|--72.21%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--27.79%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.02% flush-253:2 [kernel.kallsyms] [k] ext4_bio_write_page
|
--- ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] ext4_mb_complex_scan_group
|
--- ext4_mb_complex_scan_group
ext4_mb_regular_allocator
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% top [kernel.kallsyms] [k] __d_lookup
|
--- __d_lookup
d_lookup
proc_fill_cache
proc_pid_readdir
proc_root_readdir
vfs_readdir
sys_getdents
system_call_fastpath
__getdents64
0.02% flush-253:2 [kernel.kallsyms] [k] kmem_cache_alloc
|
--- kmem_cache_alloc
ext4_init_io_end
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] page_cache_get_speculative
|
--- page_cache_get_speculative
find_get_page
|
|--51.46%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--48.54%-- find_lock_page
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% flush-253:2 [kernel.kallsyms] [k] mb_find_extent.constprop.12
|
--- mb_find_extent.constprop.12
ext4_mb_complex_scan_group
ext4_mb_regular_allocator
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] generic_file_buffered_write
|
--- generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% top libc-2.13.90.so [.] __GI_vfprintf
|
--- __GI_vfprintf
___vsprintf_chk
0.02% tar [kernel.kallsyms] [k] ext4_da_get_block_prep
|
--- ext4_da_get_block_prep
|
|--75.27%-- __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--24.73%-- ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% top [kernel.kallsyms] [k] getname_flags
|
--- getname_flags
getname
do_sys_open
sys_open
system_call_fastpath
__GI___libc_open
0.02% swapper [kernel.kallsyms] [k] __count_vm_events
|
--- __count_vm_events
__free_pages
__free_slab
discard_slab
__slab_free
kmem_cache_free
ext4_free_io_end
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% swapper [kernel.kallsyms] [k] tick_check_idle
|
--- tick_check_idle
irq_enter
|
|--73.47%-- smp_call_function_single_interrupt
| call_function_single_interrupt
| cpuidle_idle_call
| cpu_idle
| start_secondary
|
--26.53%-- smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% flush-253:2 [kernel.kallsyms] [k] find_next_zero_bit
|
--- find_next_zero_bit
mb_find_next_zero_bit
ext4_mb_complex_scan_group
ext4_mb_regular_allocator
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] arch_read_lock
|
--- arch_read_lock
_raw_read_lock
start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% flush-253:2 [kernel.kallsyms] [k] mb_find_buddy
|
--- mb_find_buddy
mb_mark_used
ext4_mb_use_best_found
ext4_mb_check_limits
ext4_mb_complex_scan_group
ext4_mb_regular_allocator
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% swapper [kernel.kallsyms] [k] __rcu_read_unlock
|
--- __rcu_read_unlock
__prop_inc_percpu_max
test_clear_page_writeback
end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% tar [kernel.kallsyms] [k] mark_buffer_dirty
|
--- mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] new_slab
|
--- new_slab
__slab_alloc
kmem_cache_alloc
alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% swapper [kernel.kallsyms] [k] __do_softirq
|
--- __do_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% swapper [kernel.kallsyms] [k] select_task_rq_fair
|
--- select_task_rq_fair
select_task_rq
try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% tar [kernel.kallsyms] [k] do_get_write_access
|
--- do_get_write_access
|
|--70.88%-- jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| |
| |--54.37%-- ext4_new_inode
| | ext4_create
| | vfs_create
| | do_last
| | path_openat
| | do_filp_open
| | do_sys_open
| | sys_openat
| | system_call_fastpath
| | __GI___openat
| |
| --45.63%-- ext4_reserve_inode_write
| ext4_orphan_add
| ext4_unlink
| vfs_unlink
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--29.12%-- __ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] in_lock_functions
|
--- in_lock_functions
|
|--58.27%-- get_parent_ip
| add_preempt_count
| |
| |--56.37%-- jbd_lock_bh_state
| | jbd2_journal_dirty_metadata
| | __ext4_handle_dirty_metadata
| | ext4_mark_iloc_dirty
| | ext4_mark_inode_dirty
| | ext4_dirty_inode
| | __mark_inode_dirty
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --43.63%-- radix_tree_preload
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--41.73%-- sub_preempt_count
bit_spin_unlock
jbd2_journal_add_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] jbd2_journal_dirty_metadata
|
--- jbd2_journal_dirty_metadata
__ext4_handle_dirty_metadata
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] ext4_ext_map_blocks
|
--- ext4_ext_map_blocks
ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] block_write_end
|
--- block_write_end
|
|--67.22%-- ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--32.78%-- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% flush-253:2 [kernel.kallsyms] [k] mb_test_bit
|
--- mb_test_bit
mb_find_order_for_block
mb_find_extent.constprop.12
ext4_mb_complex_scan_group
ext4_mb_regular_allocator
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] attach_page_buffers
|
--- attach_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] __phys_addr
|
--- __phys_addr
virt_to_head_page
kmem_cache_free
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] do_sync_read
|
--- do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] swiotlb_map_sg_attrs
|
--- swiotlb_map_sg_attrs
ata_qc_issue
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] jbd2__journal_start
|
--- jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
|
|--73.97%-- ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--26.03%-- ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] iov_iter_advance
|
--- iov_iter_advance
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] generic_getxattr
|
--- generic_getxattr
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% jbd2/dm-2-8 [kernel.kallsyms] [k] elv_rq_merge_ok
|
--- elv_rq_merge_ok
elv_try_merge
__make_request
generic_make_request
submit_bio
submit_bh
jbd2_journal_commit_transaction
kjournald2
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] clear_bit_unlock
|
--- clear_bit_unlock
unlock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] page_waitqueue
|
--- page_waitqueue
|
|--52.26%-- unlock_page
| mpage_end_io
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--47.74%-- mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] ext4_xattr_find_entry
|
--- ext4_xattr_find_entry
ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] bit_spin_lock
|
--- bit_spin_lock
lock_page_cgroup
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--76.11%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--23.89%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% flush-253:2 [kernel.kallsyms] [k] writeback_single_inode
|
--- writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] find_get_page
|
--- find_get_page
|
|--70.62%-- __find_get_block_slow
| __find_get_block
| ext4_free_blocks
| ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--29.38%-- generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% flush-253:2 [kernel.kallsyms] [k] __ext4_get_inode_loc
|
--- __ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] fsnotify_modify
|
--- fsnotify_modify
|
|--73.71%-- vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--26.29%-- sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] blk_rq_map_sg
|
--- blk_rq_map_sg
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
blk_run_queue
scsi_run_queue
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] get_page
|
--- get_page
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% swapper [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
|
|--44.23%-- blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--32.84%-- blk_run_queue
| scsi_run_queue
| scsi_next_command
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--22.93%-- pm_qos_request
menu_select
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] __bio_clone
|
--- __bio_clone
clone_bio
__split_and_process_bio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] sg_next
|
--- sg_next
|
|--36.30%-- ata_qc_issue
| __ata_scsi_queuecmd
| ata_scsi_queuecmd
| scsi_dispatch_cmd
| scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--32.06%-- swiotlb_map_sg_attrs
| ata_qc_issue
| __ata_scsi_queuecmd
| ata_scsi_queuecmd
| scsi_dispatch_cmd
| scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--31.64%-- blk_rq_map_sg
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] __rcu_read_unlock
|
--- __rcu_read_unlock
|
|--70.72%-- __prop_inc_single
| task_dirty_inc
| account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--29.28%-- avc_has_perm_noaudit
task_has_capability
selinux_capable
security_capable
ns_capable
capable
setattr_copy
ext4_setattr
notify_change
sys_fchmod
system_call_fastpath
__GI___fchmod
0.01% tar [kernel.kallsyms] [k] _raw_spin_unlock
|
--- _raw_spin_unlock
|
|--36.11%-- smp_apic_timer_interrupt
| apic_timer_interrupt
| __find_get_block_slow
| unmap_underlying_metadata
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--34.75%-- scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--29.14%-- __block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] generic_file_aio_read
|
--- generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] map_bh
|
--- map_bh
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] memcmp
|
--- memcmp
|
|--67.73%-- ext4_dx_find_entry
| ext4_find_entry
| ext4_lookup
| d_alloc_and_lookup
| __lookup_hash.part.3
| lookup_hash
| do_last
| path_openat
| do_filp_open
| do_sys_open
| sys_openat
| system_call_fastpath
| __GI___openat
|
--32.27%-- search_dirblock
ext4_dx_find_entry
ext4_find_entry
ext4_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% swapper [kernel.kallsyms] [k] test_ti_thread_flag
|
--- test_ti_thread_flag
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% swapper [kernel.kallsyms] [k] clone_endio
|
--- clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% swapper [kernel.kallsyms] [k] __slab_free
|
--- __slab_free
kmem_cache_free
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% swapper [kernel.kallsyms] [k] nr_iowait_cpu
|
--- nr_iowait_cpu
tick_nohz_stop_idle
tick_check_idle
irq_enter
smp_call_function_single_interrupt
call_function_single_interrupt
cpuidle_idle_call
cpu_idle
start_secondary
0.01% swapper [kernel.kallsyms] [k] timekeeping_get_ns
|
--- timekeeping_get_ns
getnstimeofday
ktime_get_real
intel_idle
cpuidle_idle_call
cpu_idle
start_secondary
0.01% flush-253:2 [kernel.kallsyms] [k] crc16
|
--- crc16
ext4_group_desc_csum
ext4_mb_mark_diskspace_used
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] ata_scsi_queuecmd
|
--- ata_scsi_queuecmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] jbd2_journal_put_journal_head
|
--- jbd2_journal_put_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
|
|--60.89%-- generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--39.11%-- ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] ext4_da_write_end
|
--- ext4_da_write_end
|
|--68.79%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--31.21%-- __generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] acct_update_integrals
|
--- acct_update_integrals
__account_system_time
irqtime_account_process_tick
account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
|
|--51.45%-- get_parent_ip
| add_preempt_count
| __lru_cache_add
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--48.55%-- find_get_page
__find_get_block_slow
__find_get_block
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] __inc_zone_page_state
|
--- __inc_zone_page_state
|
|--61.16%-- account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--38.84%-- add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] account_idle_time
|
--- account_idle_time
account_idle_ticks
tick_nohz_restart_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] recalc_bh_state
|
--- recalc_bh_state
|
|--62.12%-- alloc_buffer_head
| alloc_page_buffers
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--37.88%-- alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] put_page_testzero
|
--- put_page_testzero
put_page
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] generic_segment_checks
|
--- generic_segment_checks
|
|--56.32%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--43.68%-- __generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] arch_local_irq_save
|
--- arch_local_irq_save
__free_pages
__free_slab
discard_slab
__slab_free
kmem_cache_free
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] rw_verify_area
|
--- rw_verify_area
|
|--65.20%-- vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--34.80%-- vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] file_update_time
|
--- file_update_time
|
|--70.97%-- __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--29.03%-- generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% flush-253:2 [kernel.kallsyms] [k] alloc_tio
|
--- alloc_tio
dm_request
generic_make_request
submit_bio
ext4_io_submit
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] __mark_inode_dirty
|
--- __mark_inode_dirty
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore
|
--- _raw_spin_unlock_irqrestore
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] jbd2_journal_cancel_revoke
|
--- jbd2_journal_cancel_revoke
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] lookup_page_cgroup
|
--- lookup_page_cgroup
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--53.32%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--46.68%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] _raw_spin_unlock_irq
|
--- _raw_spin_unlock_irq
add_to_page_cache_locked
add_to_page_cache_lru
|
|--69.09%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--30.91%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] __do_page_cache_readahead
|
--- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] ext4_acl_chmod
|
--- ext4_acl_chmod
notify_change
sys_fchmod
system_call_fastpath
__GI___fchmod
0.01% flush-253:2 [kernel.kallsyms] [k] start_this_handle
|
--- start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] _raw_read_lock
|
--- _raw_read_lock
|
|--56.63%-- start_this_handle
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--43.37%-- jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] mix_pool_bytes_extract
|
--- mix_pool_bytes_extract
add_timer_randomness
add_disk_randomness
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] system_call_after_swapgs
|
--- system_call_after_swapgs
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] ext4_claim_free_blocks
|
--- ext4_claim_free_blocks
|
|--53.08%-- __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--46.92%-- ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] sys_write
|
--- sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] mcount
|
--- mcount
|
|--52.17%-- radix_tree_preload_end
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.83%-- _raw_spin_unlock
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] pagevec_lru_move_fn
|
--- pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] jbd2_journal_add_journal_head
|
--- jbd2_journal_add_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] __block_commit_write
|
--- __block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] virt_to_head_page
|
--- virt_to_head_page
|
|--57.98%-- kmem_cache_free
| jbd2_journal_stop
| __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--42.02%-- jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] mb_test_bit
|
--- mb_test_bit
|
|--53.79%-- ext4_free_blocks
| ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--46.21%-- mb_free_blocks
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
|
|--57.00%-- ext4_mark_iloc_dirty
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--43.00%-- ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ahci_qc_prep
|
--- ahci_qc_prep
|
|--56.69%-- __ata_scsi_queuecmd
| ata_scsi_queuecmd
| scsi_dispatch_cmd
| scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--43.31%-- ata_qc_issue
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] generic_write_end
|
--- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] iov_iter_fault_in_readable
|
--- iov_iter_fault_in_readable
|
|--51.45%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--48.55%-- __generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] inode_reserved_space
|
--- inode_reserved_space
|
|--54.74%-- inode_add_rsv_space
| __dquot_alloc_space
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--45.26%-- __dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] update_curr
|
--- update_curr
dequeue_task_fair
dequeue_task
deactivate_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] atomic_sub
|
--- atomic_sub
|
|--56.22%-- __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--43.78%-- jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_get_inode_flags
|
--- ext4_get_inode_flags
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] prop_norm_single
|
--- prop_norm_single
|
|--55.85%-- task_dirty_inc
| account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--44.15%-- __prop_inc_single
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] jbd_unlock_bh_state
|
--- jbd_unlock_bh_state
|
|--50.48%-- jbd2_journal_dirty_metadata
| __ext4_handle_dirty_metadata
| ext4_mark_iloc_dirty
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--49.52%-- do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] arch_local_irq_disable
|
--- arch_local_irq_disable
arch_local_irq_save
_raw_spin_lock_irqsave
|
|--65.53%-- __wake_up
| jbd2_journal_stop
| __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--34.47%-- pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] __generic_file_aio_write
|
--- __generic_file_aio_write
|
|--53.33%-- generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--46.67%-- ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] test_and_set_bit
|
--- test_and_set_bit
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% flush-253:2 [kernel.kallsyms] [k] _raw_read_lock_irqsave
|
--- _raw_read_lock_irqsave
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] __find_get_block
|
--- __find_get_block
__getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
|
|--64.27%-- file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--35.73%-- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] __split_and_process_bio
|
--- __split_and_process_bio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] memcg_check_events
|
--- memcg_check_events
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--52.26%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.74%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] xattr_resolve_name
|
--- xattr_resolve_name
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% kworker/0:2 [kernel.kallsyms] [k] bit_cursor
|
--- bit_cursor
fb_flashcursor
process_one_work
worker_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] __set_page_dirty
|
--- __set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_file_write
|
--- ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] bit_spin_unlock
|
--- bit_spin_unlock
jbd2_journal_put_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] fget_light
|
--- fget_light
sys_read
system_call_fastpath
__GI___libc_read
0.01% flush-253:2 [kernel.kallsyms] [k] ext4_clear_inode_state
|
--- ext4_clear_inode_state
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% swapper [kernel.kallsyms] [k] ahci_scr_read
|
--- ahci_scr_read
sata_scr_read
sata_async_notification
ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] add_to_page_cache_lru
|
--- add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
_local_bh_enable
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] current_fs_time
|
--- current_fs_time
|
|--50.34%-- touch_atime
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--49.66%-- file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] need_resched
|
--- need_resched
_cond_resched
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] mb_clear_bit
|
--- mb_clear_bit
|
|--57.54%-- ext4_free_blocks
| ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--42.46%-- mb_free_blocks
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] __list_add
|
--- __list_add
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] task_dirty_inc
|
--- task_dirty_inc
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] vfs_read
|
--- vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] bit_spin_lock
|
--- bit_spin_lock
|
|--55.69%-- jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--44.31%-- jbd2_journal_add_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] paravirt_read_tsc
|
--- paravirt_read_tsc
read_tsc
timekeeping_get_ns
ktime_get
tick_nohz_stop_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% swapper [kernel.kallsyms] [k] irq_entries_start
|
--- irq_entries_start
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] iov_iter_copy_from_user_atomic
|
--- iov_iter_copy_from_user_atomic
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] clear_buffer_new
|
--- clear_buffer_new
|
|--57.80%-- __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--42.20%-- block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] _raw_read_lock_irqsave
|
--- _raw_read_lock_irqsave
dm_get_live_table
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] __prop_inc_single
|
--- __prop_inc_single
|
|--52.30%-- account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.70%-- task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] dm_merge_bvec
|
--- dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% swapper [kernel.kallsyms] [k] menu_select
|
--- menu_select
|
|--59.21%-- cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--40.79%-- cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% swapper [kernel.kallsyms] [k] scsi_decide_disposition
|
--- scsi_decide_disposition
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] bit_spin_trylock.constprop.23
|
--- bit_spin_trylock.constprop.23
get_partial_node
__slab_alloc
kmem_cache_alloc
alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] exit_idle
|
--- exit_idle
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% flush-253:2 [kernel.kallsyms] [k] mpage_da_submit_io
|
--- mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% swapper [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
_raw_spin_lock_irq
schedule
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] get_cwq
|
--- get_cwq
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
_raw_spin_lock
inode_add_rsv_space
__dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ____pagevec_lru_add_fn
|
--- ____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
|
|--57.19%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--42.81%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] alloc_buffer_head
|
--- alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] generic_make_request
|
--- generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% swapper [kernel.kallsyms] [k] rcu_irq_enter
|
--- rcu_irq_enter
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% swapper [kernel.kallsyms] [k] leave_mm
|
--- leave_mm
intel_idle
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] __lru_cache_add
|
--- __lru_cache_add
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar libc-2.13.90.so [.] __GI___libc_write
|
--- __GI___libc_write
0.01% swapper [kernel.kallsyms] [k] task_rq_lock
|
--- task_rq_lock
try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] __elv_add_request
|
--- __elv_add_request
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] may_link
|
--- may_link
selinux_inode_unlink
security_inode_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] __find_get_block_slow
|
--- __find_get_block_slow
unmap_underlying_metadata
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] radix_tree_tag_set
|
--- radix_tree_tag_set
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% flush-253:2 [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
_raw_spin_lock
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] delayacct_blkio_start
|
--- delayacct_blkio_start
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] current_umask
|
--- current_umask
ext4_init_acl
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.01% tar [kernel.kallsyms] [k] linear_merge
|
--- linear_merge
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% flush-253:2 [kernel.kallsyms] [k] get_parent_ip
|
--- get_parent_ip
_raw_spin_lock
ext4_discard_preallocations
ext4_da_update_reserve_space
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] __blk_run_queue
|
--- __blk_run_queue
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] __page_cache_alloc
|
--- __page_cache_alloc
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
account_system_vtime
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] ext4_new_inode
|
--- ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.01% tar [kernel.kallsyms] [k] __slab_alloc
|
--- __slab_alloc
kmem_cache_alloc
alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_ext_calc_metadata_amount
|
--- ext4_ext_calc_metadata_amount
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] task_of
|
--- task_of
dequeue_task_fair
dequeue_task
deactivate_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% swapper [kernel.kallsyms] [k] tick_program_event
|
--- tick_program_event
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% perf [kernel.kallsyms] [k] __wake_up_bit
|
--- __wake_up_bit
unlock_page
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__write_nocancel
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.01% perf [kernel.kallsyms] [k] generic_write_end
|
--- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__write_nocancel
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.01% kworker/0:2 [kernel.kallsyms] [k] native_read_msr_safe
|
--- native_read_msr_safe
acpi_cpufreq_init
do_drv_write
acpi_cpufreq_target
__cpufreq_driver_target
do_dbs_timer
process_one_work
worker_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] kmem_cache_free
|
--- kmem_cache_free
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_journal_start_sb
|
--- ext4_journal_start_sb
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_get_inode_loc
|
--- ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] __mem_cgroup_try_charge.constprop.27
|
--- __mem_cgroup_try_charge.constprop.27
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] hrtick_update
|
--- hrtick_update
enqueue_task
activate_task
try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] mem_cgroup_get_reclaim_stat_from_page
|
--- mem_cgroup_get_reclaim_stat_from_page
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% swapper [kernel.kallsyms] [k] bio_endio
|
--- bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] kfree
|
--- kfree
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_free_blocks
|
--- ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] ext4_test_inode_flag
|
--- ext4_test_inode_flag
ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] scsi_run_queue
|
--- scsi_run_queue
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% kworker/0:2 [cpufreq_ondemand] [k] do_dbs_timer
|
--- do_dbs_timer
worker_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] alloc_page_buffers
|
--- alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] linear_map
|
--- linear_map
__split_and_process_bio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% swapper [kernel.kallsyms] [k] test_ti_thread_flag.constprop.7
|
--- test_ti_thread_flag.constprop.7
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] delayacct_blkio_end
|
--- delayacct_blkio_end
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] inode_permission
|
--- inode_permission
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.01% kworker/0:2 [kernel.kallsyms] [k] kobject_get
|
--- kobject_get
__cpufreq_driver_getavg
do_dbs_timer
process_one_work
worker_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] __wake_up_common
|
--- __wake_up_common
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_read_inode_bitmap
|
--- ext4_read_inode_bitmap
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] mnt_add_count
|
--- mnt_add_count
mntget
path_get
nameidata_to_filp
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.01% tar [kernel.kallsyms] [k] schedule
|
--- schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] try_to_wake_up
|
--- try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] inode_doinit_with_dentry
|
--- inode_doinit_with_dentry
selinux_d_instantiate
security_d_instantiate
d_instantiate
ext4_add_nondir
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.00% tar [kernel.kallsyms] [k] radix_tree_preload
|
--- radix_tree_preload
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] __mod_zone_page_state
|
--- __mod_zone_page_state
add_page_to_lru_list
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] unmap_single
|
--- unmap_single
swiotlb_unmap_sg_attrs
ata_sg_clean
__ata_qc_complete
ata_qc_complete
ata_qc_complete_multiple
ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar libc-2.13.90.so [.] _int_malloc
|
--- _int_malloc
0.00% tar [kernel.kallsyms] [k] ext4_test_inode_state
|
--- ext4_test_inode_state
ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6
|
--- test_ti_thread_flag.constprop.6
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] __ext4_handle_dirty_metadata
|
--- __ext4_handle_dirty_metadata
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] queue_unplugged
|
--- queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] mutex_lock
|
--- mutex_lock
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] crc16
|
--- crc16
ext4_group_desc_csum
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.00% tar [kernel.kallsyms] [k] lock_page_killable
|
--- lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] lock_page
|
--- lock_page
find_lock_page
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] balance_dirty_pages_ratelimited_nr
|
--- balance_dirty_pages_ratelimited_nr
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] setup_object
|
--- setup_object
new_slab
__slab_alloc
kmem_cache_alloc
alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] sd_prep_fn
|
--- sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] system_call_fastpath
|
--- system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] unlock_page
|
--- unlock_page
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] blk_throtl_bio
|
--- blk_throtl_bio
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] ihold
|
--- ihold
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.00% tar [kernel.kallsyms] [k] mb_find_buddy
|
--- mb_find_buddy
mb_free_blocks
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.00% tar [kernel.kallsyms] [k] jbd2_journal_get_write_access
|
--- jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] _raw_spin_unlock
|
--- _raw_spin_unlock
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] bit_spin_unlock
|
--- bit_spin_unlock
unlock_page_cgroup
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] fput_light
|
--- fput_light
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] audit_inode.constprop.6
|
--- audit_inode.constprop.6
system_call_fastpath
__GI___fchown
0.00% tar [kernel.kallsyms] [k] ext4_test_inode_state
|
--- ext4_test_inode_state
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] system_call
|
--- system_call
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] ktime_get
|
--- ktime_get
tick_check_idle
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] radix_tree_lookup_slot
|
--- radix_tree_lookup_slot
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] apic_timer_interrupt
|
--- apic_timer_interrupt
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] __rcu_read_lock
|
--- __rcu_read_lock
__prop_inc_single
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] arch_local_irq_enable
|
--- arch_local_irq_enable
_raw_spin_unlock_irq
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] file_read_actor
|
--- file_read_actor
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] dquot_active
|
--- dquot_active
__dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% sleep [kernel.kallsyms] [k] tlb_gather_mmu
|
--- tlb_gather_mmu
mmput
exit_mm
do_exit
do_group_exit
__wake_up_parent
system_call_fastpath
0.00% tar [kernel.kallsyms] [k] test_ti_thread_flag
|
--- test_ti_thread_flag
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] bvec_free_bs
|
--- bvec_free_bs
dm_bio_destructor
bio_put
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% swapper [kernel.kallsyms] [k] ktime_get_real
|
--- ktime_get_real
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] generic_write_sync
|
--- generic_write_sync
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] _raw_spin_lock_irq
|
--- _raw_spin_lock_irq
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] posix_acl_release
|
--- posix_acl_release
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.00% tar [kernel.kallsyms] [k] ext4_dirty_inode
|
--- ext4_dirty_inode
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] test_clear_page_writeback
|
--- test_clear_page_writeback
end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] deadline_dispatch_requests
|
--- deadline_dispatch_requests
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] deadline_move_request
|
--- deadline_move_request
deadline_dispatch_requests
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% kworker/0:2 [kernel.kallsyms] [k] schedule
|
--- schedule
worker_thread
kthread
kernel_thread_helper
0.00% tar [kernel.kallsyms] [k] blk_start_plug
|
--- blk_start_plug
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] set_buffer_uptodate
|
--- set_buffer_uptodate
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] strncpy_from_user
|
--- strncpy_from_user
getname
user_path_parent
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.00% tar [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
drive_stat_acct
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] update_curr
|
--- update_curr
enqueue_task_fair
enqueue_task
activate_task
try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% swapper [kernel.kallsyms] [k] debug_smp_processor_id
|
--- debug_smp_processor_id
rcu_process_callbacks
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] page_cache_async_readahead
|
--- page_cache_async_readahead
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] bvec_alloc_bs
|
--- bvec_alloc_bs
clone_bio
__split_and_process_bio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] page_mapping
|
--- page_mapping
test_clear_page_writeback
end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% swapper [kernel.kallsyms] [k] kmem_cache_free
|
--- kmem_cache_free
mempool_free_slab
mempool_free
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% swapper [kernel.kallsyms] [k] prop_norm_percpu
|
--- prop_norm_percpu
__prop_inc_percpu_max
test_clear_page_writeback
end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] sg_init_table
|
--- sg_init_table
scsi_alloc_sgtable
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] bit_waitqueue
|
--- bit_waitqueue
wake_up_bit
unlock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] apic_write
|
--- apic_write
ack_APIC_irq
ack_apic_edge
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% swapper [kernel.kallsyms] [k] native_write_msr_safe
|
--- native_write_msr_safe
0.00% swapper [kernel.kallsyms] [k] req_bio_endio
|
--- req_bio_endio
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] __wake_up
|
--- __wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] mem_cgroup_disabled
|
--- mem_cgroup_disabled
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] __make_request
|
--- __make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] tick_check_oneshot_broadcast
|
--- tick_check_oneshot_broadcast
tick_check_idle
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] raise_softirq
|
--- raise_softirq
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
radix_tree_preload_end
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] ext4_setattr
|
--- ext4_setattr
notify_change
utimes_common
do_utimes
sys_utimensat
system_call_fastpath
futimens
0.00% swapper [kernel.kallsyms] [k] native_sched_clock
|
--- native_sched_clock
sched_clock_cpu
account_system_vtime
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% swapper [kernel.kallsyms] [k] read_tsc
|
--- read_tsc
timekeeping_get_ns
ktime_get
tick_check_idle
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] cap_inode_need_killpriv
|
--- cap_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] __cycles_2_ns
|
--- __cycles_2_ns
native_sched_clock
sched_clock
blk_rq_init
get_request
get_request_wait
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] load_balance
|
--- load_balance
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] read_tsc
|
--- read_tsc
ktime_get_ts
delayacct_end
__delayacct_blkio_end
delayacct_blkio_end
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] scsi_sg_alloc
|
--- scsi_sg_alloc
__sg_alloc_table
scsi_alloc_sgtable
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] cpuacct_charge
|
--- cpuacct_charge
update_curr
dequeue_task_fair
dequeue_task
deactivate_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] elv_rqhash_add
|
--- elv_rqhash_add
__elv_add_request
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.22
|
--- test_ti_thread_flag.constprop.22
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] ata_find_dev.part.3
|
--- ata_find_dev.part.3
__ata_scsi_find_dev
ata_scsi_find_dev
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] fsnotify
|
--- fsnotify
fsnotify_modify
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% kworker/0:2 [kernel.kallsyms] [k] ns_to_timeval
|
--- ns_to_timeval
get_cpu_iowait_time_us
do_dbs_timer
process_one_work
worker_thread
kthread
kernel_thread_helper
0.00% tar [kernel.kallsyms] [k] expand
|
--- expand
__rmqueue
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] preempt_schedule
|
--- preempt_schedule
blk_finish_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] max_io_len
|
--- max_io_len
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] release_pages
|
--- release_pages
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] irqtime_account_process_tick
|
--- irqtime_account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] atomic_long_add
|
--- atomic_long_add
zone_page_state_add
__inc_zone_state
__inc_zone_page_state
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] zone_statistics
|
--- zone_statistics
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] str2hashbuf_signed
|
--- str2hashbuf_signed
ext4fs_dirhash
dx_probe
ext4_dx_find_entry
ext4_find_entry
ext4_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.00% swapper [kernel.kallsyms] [k] ata_scsi_qc_complete
|
--- ata_scsi_qc_complete
__ata_qc_complete
ata_qc_complete
ata_qc_complete_multiple
ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] ext4_has_free_blocks
|
--- ext4_has_free_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] deadline_add_request
|
--- deadline_add_request
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] pagefault_enable
|
--- pagefault_enable
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] deadline_merge
|
--- deadline_merge
elv_merge
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] rb_next
|
--- rb_next
timerqueue_del
__remove_hrtimer
remove_hrtimer.part.4
hrtimer_try_to_cancel
hrtimer_cancel
tick_nohz_restart_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] mem_cgroup_cache_charge
|
--- mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] sched_clock_tick
|
--- sched_clock_tick
sched_clock_idle_wakeup_event
tick_nohz_stop_idle
tick_check_idle
irq_enter
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] put_bh
|
--- put_bh
__brelse
brelse
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] get_page
|
--- get_page
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] hashtab_search
|
--- hashtab_search
string_to_context_struct
security_context_to_sid_core
security_context_to_sid_default
inode_doinit_with_dentry
selinux_d_instantiate
security_d_instantiate
d_instantiate
d_splice_alias
ext4_lookup
d_alloc_and_lookup
__lookup_hash.part.3
lookup_hash
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.00% tar [kernel.kallsyms] [k] ata_qc_new_init
|
--- ata_qc_new_init
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] sd_done
|
--- sd_done
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% sleep [kernel.kallsyms] [k] get_parent_ip
|
--- get_parent_ip
add_preempt_count
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
pte_alloc_one
__pte_alloc
handle_mm_fault
do_page_fault
page_fault
_dl_vdso_vsym
0x7fff16d617f5
0x535345535f474458
0.00% sleep [kernel.kallsyms] [k] policy_zonelist
|
--- policy_zonelist
pte_alloc_one
__pte_alloc
handle_mm_fault
do_page_fault
page_fault
_dl_init_paths
_dl_sysdep_start
0.00% tar [kernel.kallsyms] [k] find_lock_page
|
--- find_lock_page
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] lock_buffer
|
--- lock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] _cond_resched
|
--- _cond_resched
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] ext4_da_write_begin
|
--- ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] dquot_free_inode
|
--- dquot_free_inode
ext4_free_inode
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.00% tar [kernel.kallsyms] [k] get_request_wait
|
--- get_request_wait
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] __local_bh_enable
|
--- __local_bh_enable
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] mutex_unlock
|
--- mutex_unlock
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] hrtimer_interrupt
|
--- hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] create_empty_buffers
|
--- create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] radix_tree_preload_end
|
--- radix_tree_preload_end
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] ext4_xattr_set_entry
|
--- ext4_xattr_set_entry
ext4_xattr_ibody_set
ext4_xattr_set_handle
ext4_init_security
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.00% swapper [kernel.kallsyms] [k] SetPageUptodate
|
--- SetPageUptodate
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] mem_cgroup_charge_statistics
|
--- mem_cgroup_charge_statistics
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar libc-2.13.90.so [.] _int_free
|
--- _int_free
0.00% swapper [kernel.kallsyms] [k] schedule
|
--- schedule
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] sysret_check
|
--- sysret_check
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] blk_peek_request
|
--- blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] ata_scsi_rw_xlat
|
--- ata_scsi_rw_xlat
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] blk_flush_plug_list
|
--- blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] jbd2_journal_stop
|
--- jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] __srcu_read_lock
|
--- __srcu_read_lock
fsnotify
fsnotify_modify
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
#
# (For a higher level overview, try: perf report --sort comm,dso)
#
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 16:56 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 16:56 UTC (permalink / raw)
To: Mel Gorman
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 11:50 -0500, James Bottomley wrote:
> This is the output of perf record -g -a -f sleep 5
>
> (hopefully the list won't choke)
Um, this one actually shows kswapd
James
---
# Events: 6K cycles
#
# Overhead Command Shared Object Symbol
# ........ ........... ................... .......................................
#
20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab
|
--- shrink_slab
|
|--99.91%-- kswapd
| kthread
| kernel_thread_helper
--0.09%-- [...]
9.98% kswapd0 [kernel.kallsyms] [k] shrink_zone
|
--- shrink_zone
|
|--99.46%-- kswapd
| kthread
| kernel_thread_helper
|
--0.54%-- kthread
kernel_thread_helper
7.70% kswapd0 [kernel.kallsyms] [k] kswapd
|
--- kswapd
kthread
kernel_thread_helper
5.40% kswapd0 [kernel.kallsyms] [k] zone_watermark_ok_safe
|
--- zone_watermark_ok_safe
|
|--72.66%-- kswapd
| kthread
| kernel_thread_helper
|
|--20.88%-- sleeping_prematurely.part.12
| kswapd
| kthread
| kernel_thread_helper
|
--6.46%-- kthread
kernel_thread_helper
4.25% kswapd0 [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
|
|--77.49%-- _raw_spin_lock
| |
| |--51.85%-- mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --48.15%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--19.47%-- _raw_spin_lock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--1.73%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--0.88%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
--0.44%-- [...]
3.34% kswapd0 [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
|
|--48.58%-- _raw_spin_unlock
| |
| |--80.77%-- mem_cgroup_soft_limit_reclaim
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --19.23%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
|--17.63%-- _raw_spin_unlock_irqrestore
| |
| |--52.96%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --47.04%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--13.83%-- _raw_spin_unlock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--6.67%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--5.54%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--2.76%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--2.24%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--1.68%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
|--0.55%-- irq_exit
| smp_apic_timer_interrupt
| apic_timer_interrupt
| mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
--0.54%-- mem_cgroup_charge_statistics
__mem_cgroup_uncharge_common
mem_cgroup_uncharge_cache_page
__remove_mapping
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
2.96% kswapd0 [kernel.kallsyms] [k] __zone_watermark_ok
|
--- __zone_watermark_ok
|
|--85.77%-- zone_watermark_ok_safe
| |
| |--74.67%-- kswapd
| | kthread
| | kernel_thread_helper
| |
| --25.33%-- sleeping_prematurely.part.12
| kswapd
| kthread
| kernel_thread_helper
|
|--11.78%-- kswapd
| kthread
| kernel_thread_helper
|
--2.45%-- sleeping_prematurely.part.12
kswapd
kthread
kernel_thread_helper
2.67% kswapd0 [kernel.kallsyms] [k] global_dirty_limits
|
--- global_dirty_limits
|
|--97.28%-- throttle_vm_writeout
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--2.72%-- shrink_zone
kswapd
kthread
kernel_thread_helper
2.30% kswapd0 [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--30.33%-- _raw_spin_lock
| |
| |--52.29%-- mem_cgroup_soft_limit_reclaim
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --47.71%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
|--28.79%-- _raw_spin_lock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--24.01%-- _raw_spin_lock_irqsave
| |
| |--63.28%-- finish_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --36.72%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--6.31%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--4.82%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--3.28%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--1.64%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--0.82%-- finish_wait
kswapd
kthread
kernel_thread_helper
2.02% kswapd0 [kernel.kallsyms] [k] up_read
|
--- up_read
|
|--95.48%-- shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--4.52%-- kswapd
kthread
kernel_thread_helper
2.01% kswapd0 [kernel.kallsyms] [k] get_parent_ip
|
--- get_parent_ip
|
|--37.59%-- sub_preempt_count
| |
| |--66.19%-- _raw_spin_unlock
| | |
| | |--55.77%-- mem_cgroup_soft_limit_reclaim
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | --44.23%-- mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| |--17.13%-- _raw_spin_unlock_irqrestore
| | |
| | |--57.38%-- finish_wait
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | --42.62%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --16.68%-- _raw_spin_unlock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--32.16%-- add_preempt_count
| |
| |--71.32%-- _raw_spin_lock
| | |
| | |--56.02%-- mem_cgroup_soft_limit_reclaim
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | --43.98%-- mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| |--17.25%-- _raw_spin_lock_irqsave
| | |
| | |--83.09%-- finish_wait
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | --16.91%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --11.43%-- _raw_spin_lock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--8.20%-- _raw_spin_unlock
| |
| |--66.09%-- mem_cgroup_soft_limit_reclaim
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --33.91%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
|--7.42%-- _raw_spin_lock
| |
| |--62.14%-- mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --37.86%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--5.55%-- _raw_spin_lock_irqsave
| |
| |--66.67%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --33.33%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--4.53%-- _raw_spin_lock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--3.62%-- _raw_spin_unlock_irqrestore
| |
| |--51.20%-- finish_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --48.80%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--0.94%-- _raw_spin_unlock_irq
shrink_zone
kswapd
kthread
kernel_thread_helper
1.88% kswapd0 [kernel.kallsyms] [k] zone_nr_lru_pages
|
--- zone_nr_lru_pages
|
|--85.21%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--14.79%-- kswapd
kthread
kernel_thread_helper
1.79% kswapd0 [kernel.kallsyms] [k] down_read_trylock
|
--- down_read_trylock
|
|--92.80%-- shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--7.20%-- kswapd
kthread
kernel_thread_helper
1.69% kswapd0 [kernel.kallsyms] [k] mutex_trylock
|
--- mutex_trylock
i915_gem_inactive_shrink
shrink_slab
kswapd
kthread
kernel_thread_helper
1.66% kswapd0 [kernel.kallsyms] [k] sleeping_prematurely.part.12
|
--- sleeping_prematurely.part.12
|
|--93.41%-- kswapd
| kthread
| kernel_thread_helper
|
--6.59%-- kthread
kernel_thread_helper
1.54% kswapd0 [kernel.kallsyms] [k] find_next_bit
|
--- find_next_bit
|
|--84.45%-- cpumask_next
| zone_watermark_ok_safe
| kswapd
| kthread
| kernel_thread_helper
|
--15.55%-- zone_watermark_ok_safe
kswapd
kthread
kernel_thread_helper
1.51% kswapd0 [kernel.kallsyms] [k] throttle_vm_writeout
|
--- throttle_vm_writeout
shrink_zone
kswapd
kthread
kernel_thread_helper
1.48% kswapd0 [kernel.kallsyms] [k] mb_cache_shrink_fn
|
--- mb_cache_shrink_fn
|
|--96.31%-- shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--3.69%-- kswapd
kthread
kernel_thread_helper
1.31% kswapd0 [kernel.kallsyms] [k] mem_cgroup_soft_limit_reclaim
|
--- mem_cgroup_soft_limit_reclaim
|
|--97.19%-- kswapd
| kthread
| kernel_thread_helper
|
--2.81%-- kthread
kernel_thread_helper
1.26% kswapd0 [kernel.kallsyms] [k] mutex_unlock
|
--- mutex_unlock
|
|--98.57%-- i915_gem_inactive_shrink
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--1.43%-- shrink_slab
kswapd
kthread
kernel_thread_helper
1.19% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
|
|--47.70%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--47.65%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--4.65%-- kswapd
kthread
kernel_thread_helper
1.14% kswapd0 [kernel.kallsyms] [k] _raw_spin_unlock
|
--- _raw_spin_unlock
|
|--59.71%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--38.63%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--1.66%-- kswapd
kthread
kernel_thread_helper
0.96% kswapd0 [kernel.kallsyms] [k] __mem_cgroup_largest_soft_limit_node
|
--- __mem_cgroup_largest_soft_limit_node
|
|--59.47%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
--40.53%-- kswapd
kthread
kernel_thread_helper
0.88% kswapd0 [kernel.kallsyms] [k] in_lock_functions
|
--- in_lock_functions
|
|--43.80%-- get_parent_ip
| |
| |--52.47%-- add_preempt_count
| | |
| | |--72.99%-- _raw_spin_lock
| | | |
| | | |--62.71%-- mem_cgroup_soft_limit_reclaim
| | | | kswapd
| | | | kthread
| | | | kernel_thread_helper
| | | |
| | | --37.29%-- mb_cache_shrink_fn
| | | shrink_slab
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | |--18.30%-- _raw_spin_lock_irq
| | | shrink_zone
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | --8.71%-- _raw_spin_lock_irqsave
| | prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --47.53%-- sub_preempt_count
| |
| |--49.81%-- _raw_spin_unlock
| | mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| |--30.32%-- _raw_spin_unlock_irqrestore
| | finish_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --19.86%-- _raw_spin_unlock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--29.12%-- add_preempt_count
| |
| |--49.50%-- _raw_spin_lock
| | |
| | |--56.31%-- mem_cgroup_soft_limit_reclaim
| | | kswapd
| | | kthread
| | | kernel_thread_helper
| | |
| | --43.69%-- mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| |--29.19%-- _raw_spin_lock_irq
| | shrink_zone
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --21.32%-- _raw_spin_lock_irqsave
| |
| |--65.66%-- finish_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --34.34%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--27.07%-- sub_preempt_count
|
|--62.26%-- _raw_spin_unlock
| |
| |--50.27%-- mb_cache_shrink_fn
| | shrink_slab
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --49.73%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--22.85%-- _raw_spin_unlock_irqrestore
| |
| |--67.34%-- finish_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --32.66%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--14.89%-- _raw_spin_unlock_irq
shrink_zone
kswapd
kthread
kernel_thread_helper
0.81% kswapd0 [kernel.kallsyms] [k] cpumask_next
|
--- cpumask_next
|
|--79.81%-- zone_watermark_ok_safe
| kswapd
| kthread
| kernel_thread_helper
|
--20.19%-- kswapd
kthread
kernel_thread_helper
0.79% kswapd0 [kernel.kallsyms] [k] zone_reclaimable_pages
|
--- zone_reclaimable_pages
|
|--93.08%-- kswapd
| kthread
| kernel_thread_helper
|
--6.92%-- kthread
kernel_thread_helper
0.77% kswapd0 [i915] [k] i915_gem_inactive_shrink
|
--- i915_gem_inactive_shrink
|
|--97.55%-- shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--2.45%-- kswapd
kthread
kernel_thread_helper
0.76% kswapd0 [kernel.kallsyms] [k] prepare_to_wait
|
--- prepare_to_wait
|
|--97.54%-- kswapd
| kthread
| kernel_thread_helper
|
--2.46%-- kthread
kernel_thread_helper
0.74% kswapd0 [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
|
|--79.88%-- _raw_spin_unlock_irqrestore
| |
| |--62.49%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --37.51%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--15.16%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--4.96%-- finish_wait
kswapd
kthread
kernel_thread_helper
0.65% kswapd0 [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6
|
--- test_ti_thread_flag.constprop.6
|
|--20.12%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
|--20.09%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
|--14.28%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--14.16%-- _raw_spin_unlock
| |
| |--80.68%-- mem_cgroup_soft_limit_reclaim
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --19.32%-- mb_cache_shrink_fn
| shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
|--11.40%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--8.63%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--8.58%-- _raw_spin_unlock_irqrestore
| |
| |--66.13%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --33.87%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
--2.74%-- _raw_spin_unlock_irq
shrink_zone
kswapd
kthread
kernel_thread_helper
0.63% tar [kernel.kallsyms] [k] copy_user_generic_string
|
--- copy_user_generic_string
|
|--56.11%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--43.89%-- generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.59% kswapd0 [kernel.kallsyms] [k] test_tsk_thread_flag
|
--- test_tsk_thread_flag
|
|--50.16%-- kswapd
| kthread
| kernel_thread_helper
|
|--46.73%-- try_to_freeze
| kswapd
| kthread
| kernel_thread_helper
|
--3.11%-- kthread
kernel_thread_helper
0.59% kswapd0 [kernel.kallsyms] [k] shrink_icache_memory
|
--- shrink_icache_memory
|
|--81.19%-- shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--18.81%-- kswapd
kthread
kernel_thread_helper
0.52% kswapd0 [kernel.kallsyms] [k] shrink_dqcache_memory
|
--- shrink_dqcache_memory
|
|--89.35%-- shrink_slab
| kswapd
| kthread
| kernel_thread_helper
|
--10.65%-- kswapd
kthread
kernel_thread_helper
0.50% kswapd0 [kernel.kallsyms] [k] zone_clear_flag
|
--- zone_clear_flag
|
|--77.78%-- kswapd
| kthread
| kernel_thread_helper
|
--22.22%-- kthread
kernel_thread_helper
0.45% kswapd0 [kernel.kallsyms] [k] kthread_should_stop
|
--- kthread_should_stop
|
|--54.28%-- kswapd
| kthread
| kernel_thread_helper
|
--45.72%-- kthread
kernel_thread_helper
0.44% kswapd0 [kernel.kallsyms] [k] global_page_state
|
--- global_page_state
|
|--33.60%-- throttle_vm_writeout
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--29.47%-- determine_dirtyable_memory
| global_dirty_limits
| throttle_vm_writeout
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--20.52%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--16.41%-- global_dirty_limits
throttle_vm_writeout
shrink_zone
kswapd
kthread
kernel_thread_helper
0.44% kswapd0 [kernel.kallsyms] [k] shrink_dcache_memory
|
--- shrink_dcache_memory
shrink_slab
kswapd
kthread
kernel_thread_helper
0.41% kswapd0 [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore
|
--- _raw_spin_unlock_irqrestore
|
|--59.01%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--40.99%-- finish_wait
kswapd
kthread
kernel_thread_helper
0.39% kswapd0 [kernel.kallsyms] [k] global_reclaimable_pages
|
--- global_reclaimable_pages
determine_dirtyable_memory
global_dirty_limits
throttle_vm_writeout
shrink_zone
kswapd
kthread
kernel_thread_helper
0.37% kswapd0 [kernel.kallsyms] [k] finish_wait
|
--- finish_wait
|
|--95.03%-- kswapd
| kthread
| kernel_thread_helper
|
--4.97%-- kthread
kernel_thread_helper
0.33% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock
|
--- _raw_spin_lock
|
|--60.70%-- mem_cgroup_soft_limit_reclaim
| kswapd
| kthread
| kernel_thread_helper
|
--39.30%-- mb_cache_shrink_fn
shrink_slab
kswapd
kthread
kernel_thread_helper
0.31% kswapd0 [kernel.kallsyms] [k] arch_local_irq_enable
|
--- arch_local_irq_enable
|
|--76.94%-- _raw_spin_unlock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--23.06%-- shrink_zone
kswapd
kthread
kernel_thread_helper
0.28% kswapd0 [kernel.kallsyms] [k] arch_local_irq_save
|
--- arch_local_irq_save
|
|--93.18%-- _raw_spin_lock_irqsave
| |
| |--64.21%-- prepare_to_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --35.79%-- finish_wait
| kswapd
| kthread
| kernel_thread_helper
|
--6.82%-- finish_wait
kswapd
kthread
kernel_thread_helper
0.22% kswapd0 [kernel.kallsyms] [k] _raw_spin_unlock_irq
|
--- _raw_spin_unlock_irq
|
|--92.00%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--8.00%-- __remove_mapping
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.22% kswapd0 [kernel.kallsyms] [k] arch_local_irq_disable
|
--- arch_local_irq_disable
|
|--58.81%-- arch_local_irq_save
| _raw_spin_lock_irqsave
| |
| |--70.97%-- finish_wait
| | kswapd
| | kthread
| | kernel_thread_helper
| |
| --29.03%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
|--33.06%-- _raw_spin_lock_irq
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--8.13%-- shrink_zone
kswapd
kthread
kernel_thread_helper
0.22% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock_irq
|
--- _raw_spin_lock_irq
|
|--58.62%-- shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
|--33.19%-- kswapd
| kthread
| kernel_thread_helper
|
--8.19%-- __remove_mapping
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.22% swapper [kernel.kallsyms] [k] poll_idle
|
--- poll_idle
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.20% kswapd0 [kernel.kallsyms] [k] __list_add
|
--- __list_add
|
|--90.97%-- prepare_to_wait
| kswapd
| kthread
| kernel_thread_helper
|
--9.03%-- free_hot_cold_page
__pagevec_free
free_page_list
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.15% tar [kernel.kallsyms] [k] zero_user_segments
|
--- zero_user_segments
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.15% kswapd0 [kernel.kallsyms] [k] rb_last
|
--- rb_last
mem_cgroup_soft_limit_reclaim
kswapd
kthread
kernel_thread_helper
0.14% swapper [kernel.kallsyms] [k] intel_idle
|
--- intel_idle
cpuidle_idle_call
cpu_idle
|
|--65.05%-- rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--34.95%-- start_secondary
0.13% kswapd0 [kernel.kallsyms] [k] get_reclaim_stat
|
--- get_reclaim_stat
|
|--57.55%-- kswapd
| kthread
| kernel_thread_helper
|
--42.45%-- shrink_zone
kswapd
kthread
kernel_thread_helper
0.12% perf [kernel.kallsyms] [k] selinux_file_alloc_security
|
--- selinux_file_alloc_security
security_file_alloc
get_empty_filp
path_openat
do_filp_open
do_sys_open
sys_open
system_call_fastpath
__GI___libc_open
0x429beb
0x418709
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.12% tar [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--18.90%-- _raw_spin_lock
| |
| |--63.01%-- ext4_da_get_block_prep
| | __block_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| |--20.46%-- get_page_from_freelist
| | __alloc_pages_nodemask
| | alloc_pages_current
| | __page_cache_alloc
| | grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --16.53%-- ext4_ext_map_blocks
| ext4_map_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--13.68%-- __lru_cache_add
| add_to_page_cache_lru
| |
| |--51.23%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --48.77%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--13.32%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--12.87%-- alloc_buffer_head
| alloc_page_buffers
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--9.79%-- _raw_spin_lock_irq
| |
| |--62.37%-- add_to_page_cache_locked
| | add_to_page_cache_lru
| | grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --37.63%-- __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--7.52%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.32%-- lock_page_cgroup
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--3.69%-- __percpu_counter_add
| __add_bdi_stat
| account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--3.54%-- bit_spin_lock
| lock_page_cgroup
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--3.41%-- create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--3.32%-- _raw_read_lock_irqsave
| dm_get_live_table
| __split_and_process_bio
| dm_request
| generic_make_request
| submit_bio
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--3.29%-- _raw_read_lock
| start_this_handle
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--2.33%-- dput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.11% kswapd0 [kernel.kallsyms] [k] try_to_freeze
|
--- try_to_freeze
|
|--83.76%-- kswapd
| kthread
| kernel_thread_helper
|
--16.24%-- kthread
kernel_thread_helper
0.10% swapper [kernel.kallsyms] [k] ahci_interrupt
|
--- ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.09% kswapd0 [kernel.kallsyms] [k] determine_dirtyable_memory
|
--- determine_dirtyable_memory
|
|--80.46%-- global_dirty_limits
| throttle_vm_writeout
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--19.54%-- throttle_vm_writeout
shrink_zone
kswapd
kthread
kernel_thread_helper
0.08% tar [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
|
|--27.37%-- _raw_spin_unlock
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--12.90%-- blk_dequeue_request
| blk_start_request
| scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--9.76%-- bit_spin_unlock
| unlock_page_cgroup
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--6.85%-- _raw_spin_unlock_irq
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--6.81%-- alloc_buffer_head
| alloc_page_buffers
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--5.86%-- _raw_spin_unlock_irqrestore
| pagevec_lru_move_fn
| ____pagevec_lru_add
| __lru_cache_add
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--5.51%-- _raw_read_unlock
| start_this_handle
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--5.37%-- jbd_unlock_bh_state
| do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_free_inode
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
|--4.68%-- deactivate_slab
| __slab_alloc
| kmem_cache_alloc
| alloc_buffer_head
| alloc_page_buffers
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--4.15%-- __srcu_read_unlock
| fsnotify
| fsnotify_modify
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--3.72%-- radix_tree_preload_end
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--3.62%-- bit_spin_unlock
| jbd2_journal_add_journal_head
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--3.41%-- pagefault_enable
iov_iter_copy_from_user_atomic
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.08% swapper [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--52.45%-- _raw_spin_lock
| get_next_timer_interrupt
| tick_nohz_stop_sched_tick
| irq_exit
| smp_call_function_single_interrupt
| call_function_single_interrupt
| cpuidle_idle_call
| cpu_idle
| start_secondary
|
|--26.09%-- bit_spin_lock.constprop.22
| __slab_free
| kmem_cache_free
| ext4_free_io_end
| ext4_end_bio
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--21.46%-- __percpu_counter_add
__prop_inc_percpu_max
test_clear_page_writeback
end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.08% tar [kernel.kallsyms] [k] kmem_cache_alloc
|
--- kmem_cache_alloc
|
|--74.26%-- alloc_buffer_head
| alloc_page_buffers
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--12.98%-- jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--6.88%-- mempool_alloc_slab
| mempool_alloc
| scsi_sg_alloc
| __sg_alloc_table
| scsi_alloc_sgtable
| scsi_init_sgtable
| scsi_init_io
| scsi_setup_fs_cmnd
| sd_prep_fn
| blk_peek_request
| scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--5.87%-- selinux_inode_alloc_security
security_inode_alloc
inode_init_always
alloc_inode
new_inode
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.07% kswapd0 [kernel.kallsyms] [k] __free_one_page
|
--- __free_one_page
free_pcppages_bulk
free_hot_cold_page
__pagevec_free
free_page_list
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.07% swapper [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
|
|--56.43%-- __percpu_counter_add
| __add_bdi_stat
| test_clear_page_writeback
| end_page_writeback
| put_io_page
| ext4_end_bio
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--17.99%-- _raw_spin_unlock_irqrestore
| __hrtimer_start_range_ns
| hrtimer_start_range_ns
| hrtimer_start_expires.constprop.1
| tick_nohz_restart_sched_tick
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--15.61%-- _raw_spin_unlock
| |
| |--55.21%-- handle_irq_event
| | handle_edge_irq
| | handle_irq
| | do_IRQ
| | common_interrupt
| | cpuidle_idle_call
| | cpu_idle
| | rest_init
| | start_kernel
| | x86_64_start_reservations
| | x86_64_start_kernel
| |
| --44.79%-- scheduler_tick
| update_process_times
| tick_sched_timer
| __run_hrtimer
| hrtimer_interrupt
| smp_apic_timer_interrupt
| apic_timer_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--5.91%-- try_to_wake_up
| default_wake_function
| autoremove_wake_function
| wake_bit_function
| __wake_up_common
| __wake_up
| __wake_up_bit
| unlock_page
| mpage_end_io
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--4.06%-- default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.06% tar [kernel.kallsyms] [k] mark_page_accessed
|
--- mark_page_accessed
|
|--87.24%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--6.42%-- __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--6.35%-- __find_get_block
__getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] get_page_from_freelist
|
--- get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--54.79%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--45.21%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar [kernel.kallsyms] [k] __ext4_get_inode_loc
|
--- __ext4_get_inode_loc
ext4_get_inode_loc
|
|--55.97%-- ext4_reserve_inode_write
| ext4_mark_inode_dirty
| |
| |--76.52%-- ext4_dirty_inode
| | __mark_inode_dirty
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --23.48%-- ext4_ext_dirty
| ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--44.03%-- ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.06% tar tar [.] 0x2aa04
|
--- 0x40753f
|
--- 0x4055c7
|
--- 0x42aa04
|
--- 0x4166d8
0x417081
|
--- 0x40fdfb
|
--- 0x405f26
|
--- 0x4332d6
|
--- 0x40fe1a
|
--- 0x408ab1
|
--- 0x40fdb8
|
--- 0x408a87
|
--- 0x4100c5
|
--- 0x407564
0.06% perf [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__write_nocancel
0x4293b8
0x429c0a
0x418709
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.05% tar [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
|
|--79.42%-- _raw_spin_lock_irq
| |
| |--57.06%-- __set_page_dirty
| | mark_buffer_dirty
| | __block_commit_write
| | block_write_end
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --42.94%-- add_to_page_cache_locked
| add_to_page_cache_lru
| |
| |--52.51%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --47.49%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--15.99%-- _raw_spin_lock
| |
| |--60.63%-- ext4_ext_map_blocks
| | ext4_map_blocks
| | ext4_da_get_block_prep
| | __block_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --39.37%-- inode_add_rsv_space
| __dquot_alloc_space
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--4.59%-- inode_add_rsv_space
__dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] __mem_cgroup_commit_charge.constprop.28
|
--- __mem_cgroup_commit_charge.constprop.28
|
|--91.58%-- __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| |
| |--51.67%-- mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --48.33%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--8.42%-- mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.05% tar [kernel.kallsyms] [k] radix_tree_lookup_element
|
--- radix_tree_lookup_element
|
|--69.74%-- radix_tree_lookup_slot
| find_get_page
| |
| |--58.70%-- generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --41.30%-- find_lock_page
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--17.45%-- radix_tree_lookup
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--12.81%-- find_get_page
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% tar [kernel.kallsyms] [k] __alloc_pages_nodemask
|
--- __alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--71.39%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--28.61%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% tar [kernel.kallsyms] [k] put_mems_allowed
|
--- put_mems_allowed
alloc_pages_current
__page_cache_alloc
|
|--70.79%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--29.21%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% swapper [kernel.kallsyms] [k] load_balance
|
--- load_balance
rebalance_domains
run_rebalance_domains
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
start_secondary
0.04% flush-253:2 [kernel.kallsyms] [k] __bio_clone
|
--- __bio_clone
clone_bio
__split_and_process_bio
dm_request
generic_make_request
submit_bio
ext4_io_submit
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.04% tar [kernel.kallsyms] [k] __wake_up_bit
|
--- __wake_up_bit
|
|--89.40%-- unlock_page
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--10.60%-- wake_up_bit
unlock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.04% tar [kernel.kallsyms] [k] get_parent_ip
|
--- get_parent_ip
|
|--29.28%-- sub_preempt_count
| |
| |--59.65%-- pagefault_enable
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --40.35%-- bit_spin_unlock
| jbd2_journal_put_journal_head
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--19.16%-- avc_has_perm_noaudit
| avc_has_perm
| inode_has_perm
| selinux_inode_permission
| security_inode_exec_permission
| exec_permission
| link_path_walk
| path_openat
| do_filp_open
| do_sys_open
| sys_openat
| system_call_fastpath
| __GI___openat
|
|--19.01%-- _raw_spin_lock
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--17.20%-- bit_spin_lock
| lock_page_cgroup
| __mem_cgroup_commit_charge.constprop.28
| __mem_cgroup_commit_charge_lrucare
| mem_cgroup_cache_charge
| add_to_page_cache_locked
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--15.36%-- add_preempt_count
file_read_actor
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.04% tar [kernel.kallsyms] [k] ext4_mark_iloc_dirty
|
--- ext4_mark_iloc_dirty
ext4_mark_inode_dirty
|
|--77.33%-- ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--13.35%-- ext4_new_inode
| ext4_create
| vfs_create
| do_last
| path_openat
| do_filp_open
| do_sys_open
| sys_openat
| system_call_fastpath
| __GI___openat
|
--9.32%-- ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.04% kswapd0 [kernel.kallsyms] [k] native_write_msr_safe
|
--- native_write_msr_safe
paravirt_write_msr
intel_pmu_disable_all
x86_pmu_disable
perf_pmu_disable
perf_event_task_tick
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
|
|--50.07%-- zone_watermark_ok_safe
| sleeping_prematurely.part.12
| kswapd
| kthread
| kernel_thread_helper
|
--49.93%-- kswapd
kthread
kernel_thread_helper
0.04% swapper [kernel.kallsyms] [k] ext4_end_bio
|
--- ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.04% kswapd0 [kernel.kallsyms] [k] __list_del_entry
|
--- __list_del_entry
finish_wait
kswapd
kthread
kernel_thread_helper
0.04% kswapd0 [kernel.kallsyms] [k] mcount
|
--- mcount
_raw_spin_lock_irq
shrink_zone
kswapd
kthread
kernel_thread_helper
0.04% tar [kernel.kallsyms] [k] __percpu_counter_add
|
--- __percpu_counter_add
|
|--87.00%-- ext4_claim_free_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--13.00%-- __prop_inc_single
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.04% tar [kernel.kallsyms] [k] arch_read_unlock
|
--- arch_read_unlock
|
|--57.64%-- _raw_read_unlock_irqrestore
| dm_get_live_table
| dm_merge_bvec
| __bio_add_page.part.2
| bio_add_page
| do_mpage_readpage
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--31.54%-- _raw_read_unlock
| start_this_handle
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--10.82%-- start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.04% kswapd0 [kernel.kallsyms] [k] mem_cgroup_del_lru_list
|
--- mem_cgroup_del_lru_list
mem_cgroup_del_lru
isolate_lru_pages
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.04% kswapd0 [kernel.kallsyms] [k] test_and_set_bit
|
--- test_and_set_bit
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.04% tar [kernel.kallsyms] [k] __might_sleep
|
--- __might_sleep
|
|--22.45%-- kmem_cache_alloc
| |
| |--59.27%-- getname_flags
| | getname
| | do_sys_open
| | sys_openat
| | system_call_fastpath
| | __GI___openat
| |
| --40.73%-- jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--22.30%-- __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--54.37%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --45.63%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--15.00%-- lock_buffer
| do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--11.33%-- __getblk
| __ext4_get_inode_loc
| ext4_get_inode_loc
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--11.31%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--10.36%-- unmap_underlying_metadata
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--7.25%-- __generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.04% kswapd0 [kernel.kallsyms] [k] debug_smp_processor_id
|
--- debug_smp_processor_id
|
|--50.69%-- __pagevec_free
| free_page_list
| shrink_page_list
| shrink_inactive_list
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--49.31%-- free_hot_cold_page
__pagevec_free
free_page_list
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.04% kswapd0 [kernel.kallsyms] [k] __mem_cgroup_uncharge_common
|
--- __mem_cgroup_uncharge_common
mem_cgroup_uncharge_cache_page
__remove_mapping
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.04% kswapd0 [kernel.kallsyms] [k] tag_get
|
--- tag_get
|
|--50.59%-- radix_tree_delete
| __delete_from_page_cache
| __remove_mapping
| shrink_page_list
| shrink_inactive_list
| shrink_zone
| kswapd
| kthread
| kernel_thread_helper
|
--49.41%-- __delete_from_page_cache
__remove_mapping
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.03% flush-253:2 [kernel.kallsyms] [k] add_preempt_count
|
--- add_preempt_count
|
|--51.77%-- _raw_read_lock_irqsave
| dm_get_live_table
| dm_merge_bvec
| __bio_add_page.part.2
| bio_add_page
| ext4_bio_write_page
| mpage_da_submit_io
| mpage_da_map_and_submit
| ext4_da_writepages
| do_writepages
| writeback_single_inode
| writeback_sb_inodes
| writeback_inodes_wb
| wb_writeback
| wb_do_writeback
| bdi_writeback_thread
| kthread
| kernel_thread_helper
|
--48.23%-- blk_throtl_bio
generic_make_request
submit_bio
ext4_io_submit
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% tar [kernel.kallsyms] [k] __inc_zone_state
|
--- __inc_zone_state
|
|--49.98%-- __inc_zone_page_state
| add_to_page_cache_locked
| add_to_page_cache_lru
| |
| |--79.48%-- mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --20.52%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--32.04%-- zone_statistics
| get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--17.98%-- account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.03% tar [kernel.kallsyms] [k] __list_del_entry
|
--- __list_del_entry
|
|--56.86%-- list_del
| __rmqueue
| get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--78.10%-- __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --21.90%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--43.14%-- __rmqueue
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.03% flush-253:2 [kernel.kallsyms] [k] dm_table_find_target
|
--- dm_table_find_target
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% tar [kernel.kallsyms] [k] jbd_lock_bh_state
|
--- jbd_lock_bh_state
|
|--61.84%-- do_get_write_access
| jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| |
| |--72.93%-- generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --27.07%-- ext4_free_blocks
| ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--38.16%-- jbd2_journal_dirty_metadata
__ext4_handle_dirty_metadata
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.03% flush-253:2 [kernel.kallsyms] [k] page_waitqueue
|
--- page_waitqueue
ext4_num_dirty_pages
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% flush-253:2 [kernel.kallsyms] [k] radix_tree_lookup_element
|
--- radix_tree_lookup_element
radix_tree_lookup_slot
find_get_page
__find_get_block_slow
unmap_underlying_metadata
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% tar [kernel.kallsyms] [k] fcheck_files
|
--- fcheck_files
fget_light
|
|--71.87%-- sys_read
| system_call_fastpath
| __GI___libc_read
|
--28.13%-- sys_write
system_call_fastpath
__GI___libc_write
0.03% flush-253:2 [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% flush-253:2 [kernel.kallsyms] [k] lock_page
|
--- lock_page
ext4_num_dirty_pages
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% tar [kernel.kallsyms] [k] put_page_testzero
|
--- put_page_testzero
|
|--70.70%-- release_pages
| pagevec_lru_move_fn
| ____pagevec_lru_add
| __lru_cache_add
| add_to_page_cache_lru
| |
| |--77.62%-- mpage_readpages
| | ext4_readpages
| | __do_page_cache_readahead
| | ra_submit
| | ondemand_readahead
| | page_cache_async_readahead
| | generic_file_aio_read
| | do_sync_read
| | vfs_read
| | sys_read
| | system_call_fastpath
| | __GI___libc_read
| |
| --22.38%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--29.30%-- put_page
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.03% tar [kernel.kallsyms] [k] zone_watermark_ok
|
--- zone_watermark_ok
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
|
|--84.29%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--15.71%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.03% tar [kernel.kallsyms] [k] start_this_handle
|
--- start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
|
|--86.30%-- ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--13.70%-- ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.03% flush-253:2 [kernel.kallsyms] [k] ext4_num_dirty_pages
|
--- ext4_num_dirty_pages
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.03% tar [kernel.kallsyms] [k] put_mems_allowed
|
--- put_mems_allowed
|
|--74.79%-- __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--65.17%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --34.83%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--25.21%-- alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.03% tar [kernel.kallsyms] [k] debug_smp_processor_id
|
--- debug_smp_processor_id
|
|--57.28%-- load_balance
| schedule
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--25.57%-- radix_tree_preload
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--17.15%-- get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.03% swapper [kernel.kallsyms] [k] __wake_up_bit
|
--- __wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.03% tar [kernel.kallsyms] [k] avc_has_perm_noaudit
|
--- avc_has_perm_noaudit
avc_has_perm
|
|--82.82%-- inode_has_perm
| |
| |--71.72%-- dentry_has_perm
| | selinux_inode_getattr
| | security_inode_getattr
| | vfs_getattr
| | vfs_fstat
| | sys_newfstat
| | system_call_fastpath
| | __GI___fxstat64
| |
| --28.28%-- selinux_inode_permission
| security_inode_exec_permission
| exec_permission
| link_path_walk
| path_openat
| do_filp_open
| do_sys_open
| sys_openat
| system_call_fastpath
| __GI___openat
|
--17.18%-- may_create
selinux_inode_create
security_inode_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.03% tar [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.03% tar [kernel.kallsyms] [k] do_mpage_readpage
|
--- do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.03% tar [kernel.kallsyms] [k] list_del
|
--- list_del
|
|--55.44%-- get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| |
| |--67.14%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --32.86%-- __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--28.53%-- ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--16.02%-- __alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] __rmqueue
|
--- __rmqueue
|
|--77.01%-- get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--22.99%-- __alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] __mem_cgroup_commit_charge_lrucare
|
--- __mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--83.13%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--16.87%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] update_page_reclaim_stat
|
--- update_page_reclaim_stat
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
|
|--75.26%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--24.74%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.02% tar [kernel.kallsyms] [k] radix_tree_insert
|
--- radix_tree_insert
add_to_page_cache_locked
add_to_page_cache_lru
|
|--81.25%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--18.75%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] add_to_page_cache_locked
|
--- add_to_page_cache_locked
add_to_page_cache_lru
|
|--62.06%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--37.94%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] page_waitqueue
|
--- page_waitqueue
|
|--58.60%-- generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--41.40%-- unlock_page
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] up_read
|
--- up_read
|
|--52.38%-- ext4_map_blocks
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.62%-- ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% swapper [kernel.kallsyms] [k] elv_completed_request
|
--- elv_completed_request
blk_finish_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% tar [kernel.kallsyms] [k] arch_local_irq_save
|
--- arch_local_irq_save
|
|--84.09%-- task_dirty_inc
| account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--15.91%-- account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] ext4_map_blocks
|
--- ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% flush-253:2 [kernel.kallsyms] [k] mcount
|
--- mcount
_raw_spin_unlock_irqrestore
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] clear_buffer_dirty
|
--- clear_buffer_dirty
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] mem_cgroup_add_lru_list
|
--- mem_cgroup_add_lru_list
add_page_to_lru_list
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.02% swapper [kernel.kallsyms] [k] blk_complete_request
|
--- blk_complete_request
scsi_done
ata_scsi_qc_complete
__ata_qc_complete
ata_qc_complete
ata_qc_complete_multiple
ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% swapper [kernel.kallsyms] [k] sg_init_table
|
--- sg_init_table
scsi_alloc_sgtable
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
blk_run_queue
scsi_run_queue
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% tar [kernel.kallsyms] [k] _raw_spin_lock
|
--- _raw_spin_lock
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] touch_atime
|
--- touch_atime
|
|--90.35%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--9.65%-- do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.02% perf ld-2.13.90.so [.] _dl_fixup
|
--- _dl_fixup
0.02% tar [kernel.kallsyms] [k] __block_write_begin
|
--- __block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% kswapd0 [kernel.kallsyms] [k] irqtime_account_process_tick
|
--- irqtime_account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
i915_gem_inactive_shrink
shrink_slab
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] apic_timer_interrupt
|
--- apic_timer_interrupt
kswapd
kthread
kernel_thread_helper
0.02% kworker/2:1 [kernel.kallsyms] [k] ns_to_timeval
|
--- ns_to_timeval
get_cpu_iowait_time_us
do_dbs_timer
process_one_work
worker_thread
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] task_rq_lock
|
--- task_rq_lock
try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
zone_watermark_ok_safe
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] hrtimer_run_queues
|
--- hrtimer_run_queues
run_local_timers
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] next_zones_zonelist
|
--- next_zones_zonelist
|
|--71.41%-- get_page_from_freelist
| __alloc_pages_nodemask
| alloc_pages_current
| __page_cache_alloc
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--28.59%-- __alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% kswapd0 [kernel.kallsyms] [k] arch_local_irq_disable
|
--- arch_local_irq_disable
run_local_timers
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] __rcu_pending
|
--- __rcu_pending
rcu_check_callbacks
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
kswapd
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] write_cache_pages_da
|
--- write_cache_pages_da
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] intel_pmu_disable_all
|
--- intel_pmu_disable_all
x86_pmu_disable
perf_pmu_disable
perf_event_task_tick
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
sleeping_prematurely.part.12
kswapd
kthread
kernel_thread_helper
0.02% swapper [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
arch_local_irq_save
irqtime_account_process_tick
account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% kswapd0 [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
call_rcu
d_free
dentry_kill
shrink_dentry_list
__shrink_dcache_sb
shrink_dcache_memory
shrink_slab
kswapd
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] SetPageLRU
|
--- SetPageLRU
|
|--66.95%-- ____pagevec_lru_add_fn
| pagevec_lru_move_fn
| ____pagevec_lru_add
| __lru_cache_add
| add_to_page_cache_lru
| |
| |--56.88%-- grab_cache_page_write_begin
| | ext4_da_write_begin
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --43.12%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--33.05%-- pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.02% kswapd0 [kernel.kallsyms] [k] run_timer_softirq
|
--- run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
i915_gem_inactive_shrink
shrink_slab
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] __isolate_lru_page
|
--- __isolate_lru_page
isolate_lru_pages
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
__might_sleep
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] radix_tree_delete
|
--- radix_tree_delete
__delete_from_page_cache
__remove_mapping
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_clear
|
--- radix_tree_tag_clear
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] ext4_map_blocks
|
--- ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
__might_sleep
|
|--51.15%-- __getblk
| __ext4_get_inode_loc
| ext4_get_inode_loc
| |
| |--50.51%-- ext4_reserve_inode_write
| | ext4_mark_inode_dirty
| | ext4_ext_truncate
| | ext4_truncate
| | ext4_evict_inode
| | evict
| | iput
| | do_unlinkat
| | sys_unlinkat
| | system_call_fastpath
| | unlinkat
| |
| --49.49%-- ext4_xattr_get
| ext4_xattr_security_get
| generic_getxattr
| cap_inode_need_killpriv
| security_inode_need_killpriv
| file_remove_suid
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--48.85%-- kmem_cache_alloc
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% kswapd0 [kernel.kallsyms] [k] mem_cgroup_uncharge_cache_page
|
--- mem_cgroup_uncharge_cache_page
shrink_page_list
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] update_curr
|
--- update_curr
task_tick_fair
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] timerqueue_del
|
--- timerqueue_del
__remove_hrtimer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
_raw_spin_unlock_irqrestore
prepare_to_wait
kswapd
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] ack_APIC_irq
|
--- ack_APIC_irq
smp_apic_timer_interrupt
apic_timer_interrupt
kswapd
kthread
kernel_thread_helper
0.02% kworker/2:1 [kernel.kallsyms] [k] too_many_workers
|
--- too_many_workers
worker_enter_idle
worker_thread
kthread
kernel_thread_helper
0.02% kswapd0 [kernel.kallsyms] [k] update_isolated_counts
|
--- update_isolated_counts
shrink_inactive_list
shrink_zone
kswapd
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6
|
--- test_ti_thread_flag.constprop.6
_raw_spin_unlock_irqrestore
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] jbd2_journal_get_write_access
|
--- jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_ext_dirty
ext4_ext_insert_extent
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] radix_tree_tag_set
|
--- radix_tree_tag_set
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] __lookup
|
--- __lookup
radix_tree_gang_lookup_slot
find_get_pages
pagevec_lookup
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] page_cache_get_speculative
|
--- page_cache_get_speculative
find_get_pages_tag
pagevec_lookup_tag
ext4_num_dirty_pages
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
_raw_spin_unlock_irqrestore
test_set_page_writeback
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] ext4_ext_map_blocks
|
--- ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] jbd_lock_bh_state
|
--- jbd_lock_bh_state
jbd2_journal_dirty_metadata
__ext4_handle_dirty_metadata
ext4_mb_mark_diskspace_used
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] sub_preempt_count
|
--- sub_preempt_count
jbd_unlock_bh_state
jbd2_journal_dirty_metadata
__ext4_handle_dirty_metadata
ext4_mb_mark_diskspace_used
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] alloc_pages_current
|
--- alloc_pages_current
__page_cache_alloc
|
|--72.21%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--27.79%-- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.02% flush-253:2 [kernel.kallsyms] [k] ext4_bio_write_page
|
--- ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% flush-253:2 [kernel.kallsyms] [k] ext4_mb_complex_scan_group
|
--- ext4_mb_complex_scan_group
ext4_mb_regular_allocator
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% top [kernel.kallsyms] [k] __d_lookup
|
--- __d_lookup
d_lookup
proc_fill_cache
proc_pid_readdir
proc_root_readdir
vfs_readdir
sys_getdents
system_call_fastpath
__getdents64
0.02% flush-253:2 [kernel.kallsyms] [k] kmem_cache_alloc
|
--- kmem_cache_alloc
ext4_init_io_end
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] page_cache_get_speculative
|
--- page_cache_get_speculative
find_get_page
|
|--51.46%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--48.54%-- find_lock_page
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% flush-253:2 [kernel.kallsyms] [k] mb_find_extent.constprop.12
|
--- mb_find_extent.constprop.12
ext4_mb_complex_scan_group
ext4_mb_regular_allocator
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] generic_file_buffered_write
|
--- generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% top libc-2.13.90.so [.] __GI_vfprintf
|
--- __GI_vfprintf
___vsprintf_chk
0.02% tar [kernel.kallsyms] [k] ext4_da_get_block_prep
|
--- ext4_da_get_block_prep
|
|--75.27%-- __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--24.73%-- ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% top [kernel.kallsyms] [k] getname_flags
|
--- getname_flags
getname
do_sys_open
sys_open
system_call_fastpath
__GI___libc_open
0.02% swapper [kernel.kallsyms] [k] __count_vm_events
|
--- __count_vm_events
__free_pages
__free_slab
discard_slab
__slab_free
kmem_cache_free
ext4_free_io_end
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% swapper [kernel.kallsyms] [k] tick_check_idle
|
--- tick_check_idle
irq_enter
|
|--73.47%-- smp_call_function_single_interrupt
| call_function_single_interrupt
| cpuidle_idle_call
| cpu_idle
| start_secondary
|
--26.53%-- smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% flush-253:2 [kernel.kallsyms] [k] find_next_zero_bit
|
--- find_next_zero_bit
mb_find_next_zero_bit
ext4_mb_complex_scan_group
ext4_mb_regular_allocator
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% tar [kernel.kallsyms] [k] arch_read_lock
|
--- arch_read_lock
_raw_read_lock
start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% flush-253:2 [kernel.kallsyms] [k] mb_find_buddy
|
--- mb_find_buddy
mb_mark_used
ext4_mb_use_best_found
ext4_mb_check_limits
ext4_mb_complex_scan_group
ext4_mb_regular_allocator
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.02% swapper [kernel.kallsyms] [k] __rcu_read_unlock
|
--- __rcu_read_unlock
__prop_inc_percpu_max
test_clear_page_writeback
end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% tar [kernel.kallsyms] [k] mark_buffer_dirty
|
--- mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] new_slab
|
--- new_slab
__slab_alloc
kmem_cache_alloc
alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% swapper [kernel.kallsyms] [k] __do_softirq
|
--- __do_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% swapper [kernel.kallsyms] [k] select_task_rq_fair
|
--- select_task_rq_fair
select_task_rq
try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.02% tar [kernel.kallsyms] [k] do_get_write_access
|
--- do_get_write_access
|
|--70.88%-- jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| |
| |--54.37%-- ext4_new_inode
| | ext4_create
| | vfs_create
| | do_last
| | path_openat
| | do_filp_open
| | do_sys_open
| | sys_openat
| | system_call_fastpath
| | __GI___openat
| |
| --45.63%-- ext4_reserve_inode_write
| ext4_orphan_add
| ext4_unlink
| vfs_unlink
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--29.12%-- __ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] in_lock_functions
|
--- in_lock_functions
|
|--58.27%-- get_parent_ip
| add_preempt_count
| |
| |--56.37%-- jbd_lock_bh_state
| | jbd2_journal_dirty_metadata
| | __ext4_handle_dirty_metadata
| | ext4_mark_iloc_dirty
| | ext4_mark_inode_dirty
| | ext4_dirty_inode
| | __mark_inode_dirty
| | generic_write_end
| | ext4_da_write_end
| | generic_file_buffered_write
| | __generic_file_aio_write
| | generic_file_aio_write
| | ext4_file_write
| | do_sync_write
| | vfs_write
| | sys_write
| | system_call_fastpath
| | __GI___libc_write
| |
| --43.63%-- radix_tree_preload
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--41.73%-- sub_preempt_count
bit_spin_unlock
jbd2_journal_add_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] jbd2_journal_dirty_metadata
|
--- jbd2_journal_dirty_metadata
__ext4_handle_dirty_metadata
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] ext4_ext_map_blocks
|
--- ext4_ext_map_blocks
ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% tar [kernel.kallsyms] [k] block_write_end
|
--- block_write_end
|
|--67.22%-- ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--32.78%-- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.02% flush-253:2 [kernel.kallsyms] [k] mb_test_bit
|
--- mb_test_bit
mb_find_order_for_block
mb_find_extent.constprop.12
ext4_mb_complex_scan_group
ext4_mb_regular_allocator
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] attach_page_buffers
|
--- attach_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] __phys_addr
|
--- __phys_addr
virt_to_head_page
kmem_cache_free
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] do_sync_read
|
--- do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] swiotlb_map_sg_attrs
|
--- swiotlb_map_sg_attrs
ata_qc_issue
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] jbd2__journal_start
|
--- jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
|
|--73.97%-- ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--26.03%-- ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] iov_iter_advance
|
--- iov_iter_advance
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] generic_getxattr
|
--- generic_getxattr
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% jbd2/dm-2-8 [kernel.kallsyms] [k] elv_rq_merge_ok
|
--- elv_rq_merge_ok
elv_try_merge
__make_request
generic_make_request
submit_bio
submit_bh
jbd2_journal_commit_transaction
kjournald2
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] clear_bit_unlock
|
--- clear_bit_unlock
unlock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] page_waitqueue
|
--- page_waitqueue
|
|--52.26%-- unlock_page
| mpage_end_io
| bio_endio
| dec_pending
| clone_endio
| bio_endio
| req_bio_endio
| blk_update_request
| blk_update_bidi_request
| blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--47.74%-- mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] ext4_xattr_find_entry
|
--- ext4_xattr_find_entry
ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] bit_spin_lock
|
--- bit_spin_lock
lock_page_cgroup
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--76.11%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--23.89%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% flush-253:2 [kernel.kallsyms] [k] writeback_single_inode
|
--- writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] find_get_page
|
--- find_get_page
|
|--70.62%-- __find_get_block_slow
| __find_get_block
| ext4_free_blocks
| ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--29.38%-- generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% flush-253:2 [kernel.kallsyms] [k] __ext4_get_inode_loc
|
--- __ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] fsnotify_modify
|
--- fsnotify_modify
|
|--73.71%-- vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--26.29%-- sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] blk_rq_map_sg
|
--- blk_rq_map_sg
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
blk_run_queue
scsi_run_queue
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] get_page
|
--- get_page
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% swapper [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|
--- _raw_spin_lock_irqsave
|
|--44.23%-- blk_end_bidi_request
| blk_end_request
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
|--32.84%-- blk_run_queue
| scsi_run_queue
| scsi_next_command
| scsi_io_completion
| scsi_finish_command
| scsi_softirq_done
| blk_done_softirq
| __do_softirq
| call_softirq
| do_softirq
| irq_exit
| do_IRQ
| common_interrupt
| cpuidle_idle_call
| cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--22.93%-- pm_qos_request
menu_select
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] __bio_clone
|
--- __bio_clone
clone_bio
__split_and_process_bio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] sg_next
|
--- sg_next
|
|--36.30%-- ata_qc_issue
| __ata_scsi_queuecmd
| ata_scsi_queuecmd
| scsi_dispatch_cmd
| scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
|--32.06%-- swiotlb_map_sg_attrs
| ata_qc_issue
| __ata_scsi_queuecmd
| ata_scsi_queuecmd
| scsi_dispatch_cmd
| scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--31.64%-- blk_rq_map_sg
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] __rcu_read_unlock
|
--- __rcu_read_unlock
|
|--70.72%-- __prop_inc_single
| task_dirty_inc
| account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--29.28%-- avc_has_perm_noaudit
task_has_capability
selinux_capable
security_capable
ns_capable
capable
setattr_copy
ext4_setattr
notify_change
sys_fchmod
system_call_fastpath
__GI___fchmod
0.01% tar [kernel.kallsyms] [k] _raw_spin_unlock
|
--- _raw_spin_unlock
|
|--36.11%-- smp_apic_timer_interrupt
| apic_timer_interrupt
| __find_get_block_slow
| unmap_underlying_metadata
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
|--34.75%-- scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--29.14%-- __block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] generic_file_aio_read
|
--- generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] map_bh
|
--- map_bh
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] memcmp
|
--- memcmp
|
|--67.73%-- ext4_dx_find_entry
| ext4_find_entry
| ext4_lookup
| d_alloc_and_lookup
| __lookup_hash.part.3
| lookup_hash
| do_last
| path_openat
| do_filp_open
| do_sys_open
| sys_openat
| system_call_fastpath
| __GI___openat
|
--32.27%-- search_dirblock
ext4_dx_find_entry
ext4_find_entry
ext4_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% swapper [kernel.kallsyms] [k] test_ti_thread_flag
|
--- test_ti_thread_flag
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% swapper [kernel.kallsyms] [k] clone_endio
|
--- clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% swapper [kernel.kallsyms] [k] __slab_free
|
--- __slab_free
kmem_cache_free
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% swapper [kernel.kallsyms] [k] nr_iowait_cpu
|
--- nr_iowait_cpu
tick_nohz_stop_idle
tick_check_idle
irq_enter
smp_call_function_single_interrupt
call_function_single_interrupt
cpuidle_idle_call
cpu_idle
start_secondary
0.01% swapper [kernel.kallsyms] [k] timekeeping_get_ns
|
--- timekeeping_get_ns
getnstimeofday
ktime_get_real
intel_idle
cpuidle_idle_call
cpu_idle
start_secondary
0.01% flush-253:2 [kernel.kallsyms] [k] crc16
|
--- crc16
ext4_group_desc_csum
ext4_mb_mark_diskspace_used
ext4_mb_new_blocks
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] ata_scsi_queuecmd
|
--- ata_scsi_queuecmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] jbd2_journal_put_journal_head
|
--- jbd2_journal_put_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
|
|--60.89%-- generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--39.11%-- ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] ext4_da_write_end
|
--- ext4_da_write_end
|
|--68.79%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--31.21%-- __generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] acct_update_integrals
|
--- acct_update_integrals
__account_system_time
irqtime_account_process_tick
account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
|
|--51.45%-- get_parent_ip
| add_preempt_count
| __lru_cache_add
| add_to_page_cache_lru
| mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--48.55%-- find_get_page
__find_get_block_slow
__find_get_block
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] __inc_zone_page_state
|
--- __inc_zone_page_state
|
|--61.16%-- account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--38.84%-- add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] account_idle_time
|
--- account_idle_time
account_idle_ticks
tick_nohz_restart_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] recalc_bh_state
|
--- recalc_bh_state
|
|--62.12%-- alloc_buffer_head
| alloc_page_buffers
| create_empty_buffers
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--37.88%-- alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] put_page_testzero
|
--- put_page_testzero
put_page
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] generic_segment_checks
|
--- generic_segment_checks
|
|--56.32%-- generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--43.68%-- __generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] arch_local_irq_save
|
--- arch_local_irq_save
__free_pages
__free_slab
discard_slab
__slab_free
kmem_cache_free
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] rw_verify_area
|
--- rw_verify_area
|
|--65.20%-- vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--34.80%-- vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] file_update_time
|
--- file_update_time
|
|--70.97%-- __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--29.03%-- generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% flush-253:2 [kernel.kallsyms] [k] alloc_tio
|
--- alloc_tio
dm_request
generic_make_request
submit_bio
ext4_io_submit
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] __mark_inode_dirty
|
--- __mark_inode_dirty
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore
|
--- _raw_spin_unlock_irqrestore
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] jbd2_journal_cancel_revoke
|
--- jbd2_journal_cancel_revoke
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] lookup_page_cgroup
|
--- lookup_page_cgroup
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--53.32%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--46.68%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] _raw_spin_unlock_irq
|
--- _raw_spin_unlock_irq
add_to_page_cache_locked
add_to_page_cache_lru
|
|--69.09%-- mpage_readpages
| ext4_readpages
| __do_page_cache_readahead
| ra_submit
| ondemand_readahead
| page_cache_async_readahead
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--30.91%-- grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] __do_page_cache_readahead
|
--- __do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] ext4_acl_chmod
|
--- ext4_acl_chmod
notify_change
sys_fchmod
system_call_fastpath
__GI___fchmod
0.01% flush-253:2 [kernel.kallsyms] [k] start_this_handle
|
--- start_this_handle
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] _raw_read_lock
|
--- _raw_read_lock
|
|--56.63%-- start_this_handle
| jbd2__journal_start
| jbd2_journal_start
| ext4_journal_start_sb
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--43.37%-- jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] mix_pool_bytes_extract
|
--- mix_pool_bytes_extract
add_timer_randomness
add_disk_randomness
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] system_call_after_swapgs
|
--- system_call_after_swapgs
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] ext4_claim_free_blocks
|
--- ext4_claim_free_blocks
|
|--53.08%-- __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--46.92%-- ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] sys_write
|
--- sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] mcount
|
--- mcount
|
|--52.17%-- radix_tree_preload_end
| add_to_page_cache_locked
| add_to_page_cache_lru
| grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.83%-- _raw_spin_unlock
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] pagevec_lru_move_fn
|
--- pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] jbd2_journal_add_journal_head
|
--- jbd2_journal_add_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] __block_commit_write
|
--- __block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] virt_to_head_page
|
--- virt_to_head_page
|
|--57.98%-- kmem_cache_free
| jbd2_journal_stop
| __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--42.02%-- jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] mb_test_bit
|
--- mb_test_bit
|
|--53.79%-- ext4_free_blocks
| ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--46.21%-- mb_free_blocks
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
|
|--57.00%-- ext4_mark_iloc_dirty
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--43.00%-- ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ahci_qc_prep
|
--- ahci_qc_prep
|
|--56.69%-- __ata_scsi_queuecmd
| ata_scsi_queuecmd
| scsi_dispatch_cmd
| scsi_request_fn
| __blk_run_queue
| queue_unplugged
| blk_flush_plug_list
| io_schedule
| sleep_on_page_killable
| __wait_on_bit_lock
| __lock_page_killable
| lock_page_killable
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--43.31%-- ata_qc_issue
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] generic_write_end
|
--- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] iov_iter_fault_in_readable
|
--- iov_iter_fault_in_readable
|
|--51.45%-- generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--48.55%-- __generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] inode_reserved_space
|
--- inode_reserved_space
|
|--54.74%-- inode_add_rsv_space
| __dquot_alloc_space
| ext4_da_get_block_prep
| __block_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--45.26%-- __dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] update_curr
|
--- update_curr
dequeue_task_fair
dequeue_task
deactivate_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] atomic_sub
|
--- atomic_sub
|
|--56.22%-- __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--43.78%-- jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_get_inode_flags
|
--- ext4_get_inode_flags
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] prop_norm_single
|
--- prop_norm_single
|
|--55.85%-- task_dirty_inc
| account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--44.15%-- __prop_inc_single
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] jbd_unlock_bh_state
|
--- jbd_unlock_bh_state
|
|--50.48%-- jbd2_journal_dirty_metadata
| __ext4_handle_dirty_metadata
| ext4_mark_iloc_dirty
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--49.52%-- do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] arch_local_irq_disable
|
--- arch_local_irq_disable
arch_local_irq_save
_raw_spin_lock_irqsave
|
|--65.53%-- __wake_up
| jbd2_journal_stop
| __ext4_journal_stop
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--34.47%-- pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] __generic_file_aio_write
|
--- __generic_file_aio_write
|
|--53.33%-- generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--46.67%-- ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] test_and_set_bit
|
--- test_and_set_bit
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% flush-253:2 [kernel.kallsyms] [k] _raw_read_lock_irqsave
|
--- _raw_read_lock_irqsave
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
ext4_bio_write_page
mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] __find_get_block
|
--- __find_get_block
__getblk
__ext4_get_inode_loc
ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
|
|--64.27%-- file_update_time
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--35.73%-- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] __split_and_process_bio
|
--- __split_and_process_bio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] memcg_check_events
|
--- memcg_check_events
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
|
|--52.26%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.74%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] xattr_resolve_name
|
--- xattr_resolve_name
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% kworker/0:2 [kernel.kallsyms] [k] bit_cursor
|
--- bit_cursor
fb_flashcursor
process_one_work
worker_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] __set_page_dirty
|
--- __set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_file_write
|
--- ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] bit_spin_unlock
|
--- bit_spin_unlock
jbd2_journal_put_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] fget_light
|
--- fget_light
sys_read
system_call_fastpath
__GI___libc_read
0.01% flush-253:2 [kernel.kallsyms] [k] ext4_clear_inode_state
|
--- ext4_clear_inode_state
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% swapper [kernel.kallsyms] [k] ahci_scr_read
|
--- ahci_scr_read
sata_scr_read
sata_async_notification
ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] add_to_page_cache_lru
|
--- add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] arch_local_save_flags
|
--- arch_local_save_flags
_local_bh_enable
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] current_fs_time
|
--- current_fs_time
|
|--50.34%-- touch_atime
| generic_file_aio_read
| do_sync_read
| vfs_read
| sys_read
| system_call_fastpath
| __GI___libc_read
|
--49.66%-- file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] need_resched
|
--- need_resched
_cond_resched
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] mb_clear_bit
|
--- mb_clear_bit
|
|--57.54%-- ext4_free_blocks
| ext4_ext_truncate
| ext4_truncate
| ext4_evict_inode
| evict
| iput
| do_unlinkat
| sys_unlinkat
| system_call_fastpath
| unlinkat
|
--42.46%-- mb_free_blocks
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] __list_add
|
--- __list_add
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] task_dirty_inc
|
--- task_dirty_inc
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] vfs_read
|
--- vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] bit_spin_lock
|
--- bit_spin_lock
|
|--55.69%-- jbd2_journal_get_write_access
| __ext4_journal_get_write_access
| ext4_reserve_inode_write
| ext4_mark_inode_dirty
| ext4_dirty_inode
| __mark_inode_dirty
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--44.31%-- jbd2_journal_add_journal_head
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] paravirt_read_tsc
|
--- paravirt_read_tsc
read_tsc
timekeeping_get_ns
ktime_get
tick_nohz_stop_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% swapper [kernel.kallsyms] [k] irq_entries_start
|
--- irq_entries_start
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] iov_iter_copy_from_user_atomic
|
--- iov_iter_copy_from_user_atomic
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] clear_buffer_new
|
--- clear_buffer_new
|
|--57.80%-- __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--42.20%-- block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] _raw_read_lock_irqsave
|
--- _raw_read_lock_irqsave
dm_get_live_table
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] __prop_inc_single
|
--- __prop_inc_single
|
|--52.30%-- account_page_dirtied
| __set_page_dirty
| mark_buffer_dirty
| __block_commit_write
| block_write_end
| generic_write_end
| ext4_da_write_end
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--47.70%-- task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] dm_merge_bvec
|
--- dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% swapper [kernel.kallsyms] [k] menu_select
|
--- menu_select
|
|--59.21%-- cpu_idle
| rest_init
| start_kernel
| x86_64_start_reservations
| x86_64_start_kernel
|
--40.79%-- cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% swapper [kernel.kallsyms] [k] scsi_decide_disposition
|
--- scsi_decide_disposition
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] bit_spin_trylock.constprop.23
|
--- bit_spin_trylock.constprop.23
get_partial_node
__slab_alloc
kmem_cache_alloc
alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] exit_idle
|
--- exit_idle
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% flush-253:2 [kernel.kallsyms] [k] mpage_da_submit_io
|
--- mpage_da_submit_io
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% swapper [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
_raw_spin_lock_irq
schedule
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] get_cwq
|
--- get_cwq
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
_raw_spin_lock
inode_add_rsv_space
__dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ____pagevec_lru_add_fn
|
--- ____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
|
|--57.19%-- grab_cache_page_write_begin
| ext4_da_write_begin
| generic_file_buffered_write
| __generic_file_aio_write
| generic_file_aio_write
| ext4_file_write
| do_sync_write
| vfs_write
| sys_write
| system_call_fastpath
| __GI___libc_write
|
--42.81%-- mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] alloc_buffer_head
|
--- alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] generic_make_request
|
--- generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% swapper [kernel.kallsyms] [k] rcu_irq_enter
|
--- rcu_irq_enter
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% swapper [kernel.kallsyms] [k] leave_mm
|
--- leave_mm
intel_idle
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] __lru_cache_add
|
--- __lru_cache_add
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar libc-2.13.90.so [.] __GI___libc_write
|
--- __GI___libc_write
0.01% swapper [kernel.kallsyms] [k] task_rq_lock
|
--- task_rq_lock
try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] __elv_add_request
|
--- __elv_add_request
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] may_link
|
--- may_link
selinux_inode_unlink
security_inode_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] __find_get_block_slow
|
--- __find_get_block_slow
unmap_underlying_metadata
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] radix_tree_tag_set
|
--- radix_tree_tag_set
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% flush-253:2 [kernel.kallsyms] [k] do_raw_spin_lock
|
--- do_raw_spin_lock
_raw_spin_lock
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] delayacct_blkio_start
|
--- delayacct_blkio_start
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] current_umask
|
--- current_umask
ext4_init_acl
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.01% tar [kernel.kallsyms] [k] linear_merge
|
--- linear_merge
dm_merge_bvec
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% flush-253:2 [kernel.kallsyms] [k] get_parent_ip
|
--- get_parent_ip
_raw_spin_lock
ext4_discard_preallocations
ext4_da_update_reserve_space
ext4_ext_map_blocks
ext4_map_blocks
mpage_da_map_and_submit
ext4_da_writepages
do_writepages
writeback_single_inode
writeback_sb_inodes
writeback_inodes_wb
wb_writeback
wb_do_writeback
bdi_writeback_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] __blk_run_queue
|
--- __blk_run_queue
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] __page_cache_alloc
|
--- __page_cache_alloc
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
account_system_vtime
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] ext4_new_inode
|
--- ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.01% tar [kernel.kallsyms] [k] __slab_alloc
|
--- __slab_alloc
kmem_cache_alloc
alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_ext_calc_metadata_amount
|
--- ext4_ext_calc_metadata_amount
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] task_of
|
--- task_of
dequeue_task_fair
dequeue_task
deactivate_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% swapper [kernel.kallsyms] [k] tick_program_event
|
--- tick_program_event
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% perf [kernel.kallsyms] [k] __wake_up_bit
|
--- __wake_up_bit
unlock_page
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__write_nocancel
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.01% perf [kernel.kallsyms] [k] generic_write_end
|
--- generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__write_nocancel
0x4191c6
0x40f7a9
0x40ef8c
__libc_start_main
0.01% kworker/0:2 [kernel.kallsyms] [k] native_read_msr_safe
|
--- native_read_msr_safe
acpi_cpufreq_init
do_drv_write
acpi_cpufreq_target
__cpufreq_driver_target
do_dbs_timer
process_one_work
worker_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] kmem_cache_free
|
--- kmem_cache_free
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_journal_start_sb
|
--- ext4_journal_start_sb
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_get_inode_loc
|
--- ext4_get_inode_loc
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] __mem_cgroup_try_charge.constprop.27
|
--- __mem_cgroup_try_charge.constprop.27
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] hrtick_update
|
--- hrtick_update
enqueue_task
activate_task
try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] mem_cgroup_get_reclaim_stat_from_page
|
--- mem_cgroup_get_reclaim_stat_from_page
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% swapper [kernel.kallsyms] [k] bio_endio
|
--- bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] kfree
|
--- kfree
jbd2__journal_start
jbd2_journal_start
ext4_journal_start_sb
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_free_blocks
|
--- ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] ext4_test_inode_flag
|
--- ext4_test_inode_flag
ext4_map_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% swapper [kernel.kallsyms] [k] scsi_run_queue
|
--- scsi_run_queue
scsi_next_command
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% kworker/0:2 [cpufreq_ondemand] [k] do_dbs_timer
|
--- do_dbs_timer
worker_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] alloc_page_buffers
|
--- alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] linear_map
|
--- linear_map
__split_and_process_bio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% swapper [kernel.kallsyms] [k] test_ti_thread_flag.constprop.7
|
--- test_ti_thread_flag.constprop.7
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.01% tar [kernel.kallsyms] [k] delayacct_blkio_end
|
--- delayacct_blkio_end
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.01% tar [kernel.kallsyms] [k] inode_permission
|
--- inode_permission
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.01% kworker/0:2 [kernel.kallsyms] [k] kobject_get
|
--- kobject_get
__cpufreq_driver_getavg
do_dbs_timer
process_one_work
worker_thread
kthread
kernel_thread_helper
0.01% tar [kernel.kallsyms] [k] __wake_up_common
|
--- __wake_up_common
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_dirty_inode
__mark_inode_dirty
file_update_time
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.01% tar [kernel.kallsyms] [k] ext4_read_inode_bitmap
|
--- ext4_read_inode_bitmap
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.01% tar [kernel.kallsyms] [k] mnt_add_count
|
--- mnt_add_count
mntget
path_get
nameidata_to_filp
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.01% tar [kernel.kallsyms] [k] schedule
|
--- schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] try_to_wake_up
|
--- try_to_wake_up
wake_up_process
wake_up_worker
insert_work
__queue_work
delayed_work_timer_fn
run_timer_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] inode_doinit_with_dentry
|
--- inode_doinit_with_dentry
selinux_d_instantiate
security_d_instantiate
d_instantiate
ext4_add_nondir
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.00% tar [kernel.kallsyms] [k] radix_tree_preload
|
--- radix_tree_preload
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] __mod_zone_page_state
|
--- __mod_zone_page_state
add_page_to_lru_list
____pagevec_lru_add_fn
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] unmap_single
|
--- unmap_single
swiotlb_unmap_sg_attrs
ata_sg_clean
__ata_qc_complete
ata_qc_complete
ata_qc_complete_multiple
ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar libc-2.13.90.so [.] _int_malloc
|
--- _int_malloc
0.00% tar [kernel.kallsyms] [k] ext4_test_inode_state
|
--- ext4_test_inode_state
ext4_xattr_get
ext4_xattr_security_get
generic_getxattr
cap_inode_need_killpriv
security_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.6
|
--- test_ti_thread_flag.constprop.6
__wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] __ext4_handle_dirty_metadata
|
--- __ext4_handle_dirty_metadata
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] queue_unplugged
|
--- queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] mutex_lock
|
--- mutex_lock
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] crc16
|
--- crc16
ext4_group_desc_csum
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.00% tar [kernel.kallsyms] [k] lock_page_killable
|
--- lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] lock_page
|
--- lock_page
find_lock_page
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] balance_dirty_pages_ratelimited_nr
|
--- balance_dirty_pages_ratelimited_nr
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] setup_object
|
--- setup_object
new_slab
__slab_alloc
kmem_cache_alloc
alloc_buffer_head
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] sd_prep_fn
|
--- sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] system_call_fastpath
|
--- system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] unlock_page
|
--- unlock_page
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] blk_throtl_bio
|
--- blk_throtl_bio
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] ihold
|
--- ihold
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.00% tar [kernel.kallsyms] [k] mb_find_buddy
|
--- mb_find_buddy
mb_free_blocks
ext4_free_blocks
ext4_ext_truncate
ext4_truncate
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.00% tar [kernel.kallsyms] [k] jbd2_journal_get_write_access
|
--- jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] _raw_spin_unlock
|
--- _raw_spin_unlock
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] bit_spin_unlock
|
--- bit_spin_unlock
unlock_page_cgroup
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] fput_light
|
--- fput_light
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] audit_inode.constprop.6
|
--- audit_inode.constprop.6
system_call_fastpath
__GI___fchown
0.00% tar [kernel.kallsyms] [k] ext4_test_inode_state
|
--- ext4_test_inode_state
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] system_call
|
--- system_call
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] ktime_get
|
--- ktime_get
tick_check_idle
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] radix_tree_lookup_slot
|
--- radix_tree_lookup_slot
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] apic_timer_interrupt
|
--- apic_timer_interrupt
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] __rcu_read_lock
|
--- __rcu_read_lock
__prop_inc_single
task_dirty_inc
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] arch_local_irq_enable
|
--- arch_local_irq_enable
_raw_spin_unlock_irq
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] file_read_actor
|
--- file_read_actor
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] dquot_active
|
--- dquot_active
__dquot_alloc_space
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% sleep [kernel.kallsyms] [k] tlb_gather_mmu
|
--- tlb_gather_mmu
mmput
exit_mm
do_exit
do_group_exit
__wake_up_parent
system_call_fastpath
0.00% tar [kernel.kallsyms] [k] test_ti_thread_flag
|
--- test_ti_thread_flag
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] bvec_free_bs
|
--- bvec_free_bs
dm_bio_destructor
bio_put
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% swapper [kernel.kallsyms] [k] ktime_get_real
|
--- ktime_get_real
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] generic_write_sync
|
--- generic_write_sync
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] _raw_spin_lock_irq
|
--- _raw_spin_lock_irq
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] posix_acl_release
|
--- posix_acl_release
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.00% tar [kernel.kallsyms] [k] ext4_dirty_inode
|
--- ext4_dirty_inode
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] test_clear_page_writeback
|
--- test_clear_page_writeback
end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] deadline_dispatch_requests
|
--- deadline_dispatch_requests
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] deadline_move_request
|
--- deadline_move_request
deadline_dispatch_requests
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% kworker/0:2 [kernel.kallsyms] [k] schedule
|
--- schedule
worker_thread
kthread
kernel_thread_helper
0.00% tar [kernel.kallsyms] [k] blk_start_plug
|
--- blk_start_plug
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] set_buffer_uptodate
|
--- set_buffer_uptodate
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] strncpy_from_user
|
--- strncpy_from_user
getname
user_path_parent
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.00% tar [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
drive_stat_acct
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] update_curr
|
--- update_curr
enqueue_task_fair
enqueue_task
activate_task
try_to_wake_up
default_wake_function
autoremove_wake_function
wake_bit_function
__wake_up_common
__wake_up
__wake_up_bit
unlock_page
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% swapper [kernel.kallsyms] [k] debug_smp_processor_id
|
--- debug_smp_processor_id
rcu_process_callbacks
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] page_cache_async_readahead
|
--- page_cache_async_readahead
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] bvec_alloc_bs
|
--- bvec_alloc_bs
clone_bio
__split_and_process_bio
dm_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] page_mapping
|
--- page_mapping
test_clear_page_writeback
end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% swapper [kernel.kallsyms] [k] kmem_cache_free
|
--- kmem_cache_free
mempool_free_slab
mempool_free
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% swapper [kernel.kallsyms] [k] prop_norm_percpu
|
--- prop_norm_percpu
__prop_inc_percpu_max
test_clear_page_writeback
end_page_writeback
put_io_page
ext4_end_bio
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] sg_init_table
|
--- sg_init_table
scsi_alloc_sgtable
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] atomic_inc
|
--- atomic_inc
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] bit_waitqueue
|
--- bit_waitqueue
wake_up_bit
unlock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] apic_write
|
--- apic_write
ack_APIC_irq
ack_apic_edge
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% swapper [kernel.kallsyms] [k] native_write_msr_safe
|
--- native_write_msr_safe
0.00% swapper [kernel.kallsyms] [k] req_bio_endio
|
--- req_bio_endio
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] __wake_up
|
--- __wake_up
jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] mem_cgroup_disabled
|
--- mem_cgroup_disabled
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] __make_request
|
--- __make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] tick_check_oneshot_broadcast
|
--- tick_check_oneshot_broadcast
tick_check_idle
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] raise_softirq
|
--- raise_softirq
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
radix_tree_preload_end
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] ext4_setattr
|
--- ext4_setattr
notify_change
utimes_common
do_utimes
sys_utimensat
system_call_fastpath
futimens
0.00% swapper [kernel.kallsyms] [k] native_sched_clock
|
--- native_sched_clock
sched_clock_cpu
account_system_vtime
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% swapper [kernel.kallsyms] [k] read_tsc
|
--- read_tsc
timekeeping_get_ns
ktime_get
tick_check_idle
irq_enter
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] cap_inode_need_killpriv
|
--- cap_inode_need_killpriv
file_remove_suid
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] __cycles_2_ns
|
--- __cycles_2_ns
native_sched_clock
sched_clock
blk_rq_init
get_request
get_request_wait
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] load_balance
|
--- load_balance
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] read_tsc
|
--- read_tsc
ktime_get_ts
delayacct_end
__delayacct_blkio_end
delayacct_blkio_end
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] scsi_sg_alloc
|
--- scsi_sg_alloc
__sg_alloc_table
scsi_alloc_sgtable
scsi_init_sgtable
scsi_init_io
scsi_setup_fs_cmnd
sd_prep_fn
blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] cpuacct_charge
|
--- cpuacct_charge
update_curr
dequeue_task_fair
dequeue_task
deactivate_task
schedule
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] elv_rqhash_add
|
--- elv_rqhash_add
__elv_add_request
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] test_ti_thread_flag.constprop.22
|
--- test_ti_thread_flag.constprop.22
alloc_page_buffers
create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] ata_find_dev.part.3
|
--- ata_find_dev.part.3
__ata_scsi_find_dev
ata_scsi_find_dev
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] fsnotify
|
--- fsnotify
fsnotify_modify
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] arch_local_irq_restore
|
--- arch_local_irq_restore
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% kworker/0:2 [kernel.kallsyms] [k] ns_to_timeval
|
--- ns_to_timeval
get_cpu_iowait_time_us
do_dbs_timer
process_one_work
worker_thread
kthread
kernel_thread_helper
0.00% tar [kernel.kallsyms] [k] expand
|
--- expand
__rmqueue
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] preempt_schedule
|
--- preempt_schedule
blk_finish_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] max_io_len
|
--- max_io_len
__bio_add_page.part.2
bio_add_page
do_mpage_readpage
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] release_pages
|
--- release_pages
pagevec_lru_move_fn
____pagevec_lru_add
__lru_cache_add
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] irqtime_account_process_tick
|
--- irqtime_account_process_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] atomic_long_add
|
--- atomic_long_add
zone_page_state_add
__inc_zone_state
__inc_zone_page_state
account_page_dirtied
__set_page_dirty
mark_buffer_dirty
__block_commit_write
block_write_end
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] zone_statistics
|
--- zone_statistics
__alloc_pages_nodemask
alloc_pages_current
__page_cache_alloc
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] str2hashbuf_signed
|
--- str2hashbuf_signed
ext4fs_dirhash
dx_probe
ext4_dx_find_entry
ext4_find_entry
ext4_unlink
vfs_unlink
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.00% swapper [kernel.kallsyms] [k] ata_scsi_qc_complete
|
--- ata_scsi_qc_complete
__ata_qc_complete
ata_qc_complete
ata_qc_complete_multiple
ahci_interrupt
handle_irq_event_percpu
handle_irq_event
handle_edge_irq
handle_irq
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] ext4_has_free_blocks
|
--- ext4_has_free_blocks
ext4_da_get_block_prep
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] deadline_add_request
|
--- deadline_add_request
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] pagefault_enable
|
--- pagefault_enable
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] deadline_merge
|
--- deadline_merge
elv_merge
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] rb_next
|
--- rb_next
timerqueue_del
__remove_hrtimer
remove_hrtimer.part.4
hrtimer_try_to_cancel
hrtimer_cancel
tick_nohz_restart_sched_tick
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] mem_cgroup_cache_charge
|
--- mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] sched_clock_tick
|
--- sched_clock_tick
sched_clock_idle_wakeup_event
tick_nohz_stop_idle
tick_check_idle
irq_enter
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] put_bh
|
--- put_bh
__brelse
brelse
ext4_mark_iloc_dirty
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] get_page
|
--- get_page
__lru_cache_add
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] hashtab_search
|
--- hashtab_search
string_to_context_struct
security_context_to_sid_core
security_context_to_sid_default
inode_doinit_with_dentry
selinux_d_instantiate
security_d_instantiate
d_instantiate
d_splice_alias
ext4_lookup
d_alloc_and_lookup
__lookup_hash.part.3
lookup_hash
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.00% tar [kernel.kallsyms] [k] ata_qc_new_init
|
--- ata_qc_new_init
__ata_scsi_queuecmd
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] sd_done
|
--- sd_done
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% sleep [kernel.kallsyms] [k] get_parent_ip
|
--- get_parent_ip
add_preempt_count
get_page_from_freelist
__alloc_pages_nodemask
alloc_pages_current
pte_alloc_one
__pte_alloc
handle_mm_fault
do_page_fault
page_fault
_dl_vdso_vsym
0x7fff16d617f5
0x535345535f474458
0.00% sleep [kernel.kallsyms] [k] policy_zonelist
|
--- policy_zonelist
pte_alloc_one
__pte_alloc
handle_mm_fault
do_page_fault
page_fault
_dl_init_paths
_dl_sysdep_start
0.00% tar [kernel.kallsyms] [k] find_lock_page
|
--- find_lock_page
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] lock_buffer
|
--- lock_buffer
do_get_write_access
jbd2_journal_get_write_access
__ext4_journal_get_write_access
ext4_reserve_inode_write
ext4_mark_inode_dirty
ext4_dirty_inode
__mark_inode_dirty
generic_write_end
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] _cond_resched
|
--- _cond_resched
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] ext4_da_write_begin
|
--- ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] dquot_free_inode
|
--- dquot_free_inode
ext4_free_inode
ext4_evict_inode
evict
iput
do_unlinkat
sys_unlinkat
system_call_fastpath
unlinkat
0.00% tar [kernel.kallsyms] [k] get_request_wait
|
--- get_request_wait
__make_request
generic_make_request
submit_bio
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% swapper [kernel.kallsyms] [k] __local_bh_enable
|
--- __local_bh_enable
__do_softirq
call_softirq
do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] mutex_unlock
|
--- mutex_unlock
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% swapper [kernel.kallsyms] [k] hrtimer_interrupt
|
--- hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] create_empty_buffers
|
--- create_empty_buffers
__block_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] radix_tree_preload_end
|
--- radix_tree_preload_end
add_to_page_cache_locked
add_to_page_cache_lru
grab_cache_page_write_begin
ext4_da_write_begin
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] ext4_xattr_set_entry
|
--- ext4_xattr_set_entry
ext4_xattr_ibody_set
ext4_xattr_set_handle
ext4_init_security
ext4_new_inode
ext4_create
vfs_create
do_last
path_openat
do_filp_open
do_sys_open
sys_openat
system_call_fastpath
__GI___openat
0.00% swapper [kernel.kallsyms] [k] SetPageUptodate
|
--- SetPageUptodate
mpage_end_io
bio_endio
dec_pending
clone_endio
bio_endio
req_bio_endio
blk_update_request
blk_update_bidi_request
blk_end_bidi_request
blk_end_request
scsi_io_completion
scsi_finish_command
scsi_softirq_done
blk_done_softirq
__do_softirq
call_softirq
do_softirq
irq_exit
do_IRQ
common_interrupt
cpuidle_idle_call
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] mem_cgroup_charge_statistics
|
--- mem_cgroup_charge_statistics
__mem_cgroup_commit_charge.constprop.28
__mem_cgroup_commit_charge_lrucare
mem_cgroup_cache_charge
add_to_page_cache_locked
add_to_page_cache_lru
mpage_readpages
ext4_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_async_readahead
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar libc-2.13.90.so [.] _int_free
|
--- _int_free
0.00% swapper [kernel.kallsyms] [k] schedule
|
--- schedule
cpu_idle
rest_init
start_kernel
x86_64_start_reservations
x86_64_start_kernel
0.00% tar [kernel.kallsyms] [k] sysret_check
|
--- sysret_check
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] blk_peek_request
|
--- blk_peek_request
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] ata_scsi_rw_xlat
|
--- ata_scsi_rw_xlat
ata_scsi_queuecmd
scsi_dispatch_cmd
scsi_request_fn
__blk_run_queue
queue_unplugged
blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] blk_flush_plug_list
|
--- blk_flush_plug_list
io_schedule
sleep_on_page_killable
__wait_on_bit_lock
__lock_page_killable
lock_page_killable
generic_file_aio_read
do_sync_read
vfs_read
sys_read
system_call_fastpath
__GI___libc_read
0.00% tar [kernel.kallsyms] [k] jbd2_journal_stop
|
--- jbd2_journal_stop
__ext4_journal_stop
ext4_da_write_end
generic_file_buffered_write
__generic_file_aio_write
generic_file_aio_write
ext4_file_write
do_sync_write
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
0.00% tar [kernel.kallsyms] [k] __srcu_read_lock
|
--- __srcu_read_lock
fsnotify
fsnotify_modify
vfs_write
sys_write
system_call_fastpath
__GI___libc_write
#
# (For a higher level overview, try: perf report --sort comm,dso)
#
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 15:08 ` Mel Gorman
@ 2011-04-28 17:10 ` Colin Ian King
-1 siblings, 0 replies; 138+ messages in thread
From: Colin Ian King @ 2011-04-28 17:10 UTC (permalink / raw)
To: Mel Gorman
Cc: James Bottomley, Jan Kara, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 16:08 +0100, Mel Gorman wrote:
[ text deleted ]
> Another consequence of this patch is that when high order allocations
> are in progress (is the test case fork heavy in any way for
> example? alternatively, it might be something in the storage stack
> that requires high-order allocs) we are no longer necessarily going
> to sleep because of should_reclaim_continue() check. This could
> explain kswapd-at-99% but would only apply if CONFIG_COMPACTION is
> set (does unsetting CONFIG_COMPACTION help). If the bug only triggers
> for CONFIG_COMPACTION, does the following *untested* patch help any?
Afraid to report this patch didn't help either.
>
> (as a warning, I'm offline Friday until Tuesday morning. I'll try
> check mail over the weekend but it's unlikely I'll find a terminal
> or be allowed to use it without an ass kicking)
Ditto, me, to, I will pick this up Tuesday.
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 148c6e6..c74a501 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -1842,15 +1842,22 @@ static inline bool should_continue_reclaim(struct zone *zone,
> return false;
>
> /*
> - * If we failed to reclaim and have scanned the full list, stop.
> - * NOTE: Checking just nr_reclaimed would exit reclaim/compaction far
> - * faster but obviously would be less likely to succeed
> - * allocation. If this is desirable, use GFP_REPEAT to decide
> - * if both reclaimed and scanned should be checked or just
> - * reclaimed
> + * For direct reclaimers
> + * If we failed to reclaim and have scanned the full list, stop.
> + * The caller will check congestion and sleep if necessary until
> + * some IO completes.
> + * For kswapd
> + * Check just nr_reclaimed. If we are failing to reclaim, we
> + * want to stop this reclaim loop, increase the priority and
> + * go to sleep if necessary to allow IO a change to complete.
> + * This avoids kswapd going into a busy loop in shrink_zone()
> */
> - if (!nr_reclaimed && !nr_scanned)
> - return false;
> + if (!nr_reclaimed) {
> + if (current_is_kswapd())
> + return false;
> + else if (!nr_scanned)
> + return false;
> + }
>
> /*
> * If we have not reclaimed enough pages for compaction and the
> @@ -1924,8 +1931,13 @@ restart:
>
> /* reclaim/compaction might need reclaim to continue */
> if (should_continue_reclaim(zone, nr_reclaimed,
> - sc->nr_scanned - nr_scanned, sc))
> + sc->nr_scanned - nr_scanned, sc)) {
> + /* Throttle direct reclaimers if congested */
> + if (!current_is_kswapd())
> + wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
> +
> goto restart;
> + }
>
> throttle_vm_writeout(sc->gfp_mask);
> }
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 17:10 ` Colin Ian King
0 siblings, 0 replies; 138+ messages in thread
From: Colin Ian King @ 2011-04-28 17:10 UTC (permalink / raw)
To: Mel Gorman
Cc: James Bottomley, Jan Kara, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 16:08 +0100, Mel Gorman wrote:
[ text deleted ]
> Another consequence of this patch is that when high order allocations
> are in progress (is the test case fork heavy in any way for
> example? alternatively, it might be something in the storage stack
> that requires high-order allocs) we are no longer necessarily going
> to sleep because of should_reclaim_continue() check. This could
> explain kswapd-at-99% but would only apply if CONFIG_COMPACTION is
> set (does unsetting CONFIG_COMPACTION help). If the bug only triggers
> for CONFIG_COMPACTION, does the following *untested* patch help any?
Afraid to report this patch didn't help either.
>
> (as a warning, I'm offline Friday until Tuesday morning. I'll try
> check mail over the weekend but it's unlikely I'll find a terminal
> or be allowed to use it without an ass kicking)
Ditto, me, to, I will pick this up Tuesday.
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 148c6e6..c74a501 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -1842,15 +1842,22 @@ static inline bool should_continue_reclaim(struct zone *zone,
> return false;
>
> /*
> - * If we failed to reclaim and have scanned the full list, stop.
> - * NOTE: Checking just nr_reclaimed would exit reclaim/compaction far
> - * faster but obviously would be less likely to succeed
> - * allocation. If this is desirable, use GFP_REPEAT to decide
> - * if both reclaimed and scanned should be checked or just
> - * reclaimed
> + * For direct reclaimers
> + * If we failed to reclaim and have scanned the full list, stop.
> + * The caller will check congestion and sleep if necessary until
> + * some IO completes.
> + * For kswapd
> + * Check just nr_reclaimed. If we are failing to reclaim, we
> + * want to stop this reclaim loop, increase the priority and
> + * go to sleep if necessary to allow IO a change to complete.
> + * This avoids kswapd going into a busy loop in shrink_zone()
> */
> - if (!nr_reclaimed && !nr_scanned)
> - return false;
> + if (!nr_reclaimed) {
> + if (current_is_kswapd())
> + return false;
> + else if (!nr_scanned)
> + return false;
> + }
>
> /*
> * If we have not reclaimed enough pages for compaction and the
> @@ -1924,8 +1931,13 @@ restart:
>
> /* reclaim/compaction might need reclaim to continue */
> if (should_continue_reclaim(zone, nr_reclaimed,
> - sc->nr_scanned - nr_scanned, sc))
> + sc->nr_scanned - nr_scanned, sc)) {
> + /* Throttle direct reclaimers if congested */
> + if (!current_is_kswapd())
> + wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
> +
> goto restart;
> + }
>
> throttle_vm_writeout(sc->gfp_mask);
> }
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 16:56 ` James Bottomley
@ 2011-04-28 17:18 ` Mel Gorman
-1 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-04-28 17:18 UTC (permalink / raw)
To: James Bottomley
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote:
> On Thu, 2011-04-28 at 11:50 -0500, James Bottomley wrote:
> > This is the output of perf record -g -a -f sleep 5
> >
> > (hopefully the list won't choke)
>
> Um, this one actually shows kswapd
>
> James
>
> ---
>
> # Events: 6K cycles
> #
> # Overhead Command Shared Object Symbol
> # ........ ........... ................... .......................................
> #
> 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab
> |
> --- shrink_slab
> |
> |--99.91%-- kswapd
> | kthread
> | kernel_thread_helper
> --0.09%-- [...]
>
Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use
compaction instead of lumpy reclaim" is related unless we are seeing
two problems that happen to manifest in a similar manner.
However, there were a number of changes made to dcache in particular
for 2.6.38. Specifically thinks like dentry_kill use trylock and is
happy to loop around if it fails to acquire anything. See things like
this for example;
static void try_prune_one_dentry(struct dentry *dentry)
__releases(dentry->d_lock)
{
struct dentry *parent;
parent = dentry_kill(dentry, 0);
/*
* If dentry_kill returns NULL, we have nothing more to do.
* if it returns the same dentry, trylocks failed. In either
* case, just loop again.
If this in combination with many inodes being locked for whatever
reason (writeback locking them maybe?) is causing the shrinker to
return after zero progress, it could in turn cause kswapd to enter
into a loop for longish periods of time in shrink_slab here;
while (total_scan >= SHRINK_BATCH) {
long this_scan = SHRINK_BATCH;
int shrink_ret;
int nr_before;
nr_before = (*shrinker->shrink)(shrinker, 0, gfp_mask);
shrink_ret = (*shrinker->shrink)(shrinker, this_scan,
gfp_mask);
if (shrink_ret == -1)
break;
if (shrink_ret < nr_before)
ret += nr_before - shrink_ret;
count_vm_events(SLABS_SCANNED, this_scan);
total_scan -= this_scan;
cond_resched();
}
That would explain this trace.
> 9.98% kswapd0 [kernel.kallsyms] [k] shrink_zone
> |
> --- shrink_zone
> |
> |--99.46%-- kswapd
> | kthread
> | kernel_thread_helper
> |
> --0.54%-- kthread
> kernel_thread_helper
>
> 7.70% kswapd0 [kernel.kallsyms] [k] kswapd
> |
> --- kswapd
> kthread
> kernel_thread_helper
>
> 5.40% kswapd0 [kernel.kallsyms] [k] zone_watermark_ok_safe
> |
> --- zone_watermark_ok_safe
> |
> |--72.66%-- kswapd
> | kthread
> | kernel_thread_helper
> |
> |--20.88%-- sleeping_prematurely.part.12
> | kswapd
> | kthread
> | kernel_thread_helper
> |
> --6.46%-- kthread
> kernel_thread_helper
>
We are also spending an astonishing amount of time in
sleeping_prematurely leading me to believe we are failing to balance the
zones and are continually under the min watermark for one of the zones.
We are never going to sleep because of this check;
if (total_scanned && (priority < DEF_PRIORITY - 2)) {
if (has_under_min_watermark_zone)
count_vm_event(KSWAPD_SKIP_CONGESTION_WAIT);
else
congestion_wait(BLK_RW_ASYNC, HZ/10);
}
However, I think this is a secondary effect to the failure of shrinkers
to do their work. If slabs were being shrunk, one would expect us to
be getting over the min watermark.
> 4.25% kswapd0 [kernel.kallsyms] [k] do_raw_spin_lock
> |
> --- do_raw_spin_lock
> |
> |--77.49%-- _raw_spin_lock
> | |
> | |--51.85%-- mb_cache_shrink_fn
> | | shrink_slab
> | | kswapd
> | | kthread
> | | kernel_thread_helper
> | |
> | --48.15%-- mem_cgroup_soft_limit_reclaim
> | kswapd
> | kthread
> | kernel_thread_helper
> |
Way hey, cgroups are also in the mix. How jolly.
Is systemd a common element of the machines hitting this bug by any
chance?
The remaining traces seem to be follow-on damage related to the three
issues of "shrinkers are bust in some manner" causing "we are not
getting over the min watermark" and as a side-show "we are spending lots
of time doing something unspecified but unhelpful in cgroups".
> <SNIP>
--
Mel Gorman
SUSE Labs
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 17:18 ` Mel Gorman
0 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-04-28 17:18 UTC (permalink / raw)
To: James Bottomley
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote:
> On Thu, 2011-04-28 at 11:50 -0500, James Bottomley wrote:
> > This is the output of perf record -g -a -f sleep 5
> >
> > (hopefully the list won't choke)
>
> Um, this one actually shows kswapd
>
> James
>
> ---
>
> # Events: 6K cycles
> #
> # Overhead Command Shared Object Symbol
> # ........ ........... ................... .......................................
> #
> 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab
> |
> --- shrink_slab
> |
> |--99.91%-- kswapd
> | kthread
> | kernel_thread_helper
> --0.09%-- [...]
>
Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use
compaction instead of lumpy reclaim" is related unless we are seeing
two problems that happen to manifest in a similar manner.
However, there were a number of changes made to dcache in particular
for 2.6.38. Specifically thinks like dentry_kill use trylock and is
happy to loop around if it fails to acquire anything. See things like
this for example;
static void try_prune_one_dentry(struct dentry *dentry)
__releases(dentry->d_lock)
{
struct dentry *parent;
parent = dentry_kill(dentry, 0);
/*
* If dentry_kill returns NULL, we have nothing more to do.
* if it returns the same dentry, trylocks failed. In either
* case, just loop again.
If this in combination with many inodes being locked for whatever
reason (writeback locking them maybe?) is causing the shrinker to
return after zero progress, it could in turn cause kswapd to enter
into a loop for longish periods of time in shrink_slab here;
while (total_scan >= SHRINK_BATCH) {
long this_scan = SHRINK_BATCH;
int shrink_ret;
int nr_before;
nr_before = (*shrinker->shrink)(shrinker, 0, gfp_mask);
shrink_ret = (*shrinker->shrink)(shrinker, this_scan,
gfp_mask);
if (shrink_ret == -1)
break;
if (shrink_ret < nr_before)
ret += nr_before - shrink_ret;
count_vm_events(SLABS_SCANNED, this_scan);
total_scan -= this_scan;
cond_resched();
}
That would explain this trace.
> 9.98% kswapd0 [kernel.kallsyms] [k] shrink_zone
> |
> --- shrink_zone
> |
> |--99.46%-- kswapd
> | kthread
> | kernel_thread_helper
> |
> --0.54%-- kthread
> kernel_thread_helper
>
> 7.70% kswapd0 [kernel.kallsyms] [k] kswapd
> |
> --- kswapd
> kthread
> kernel_thread_helper
>
> 5.40% kswapd0 [kernel.kallsyms] [k] zone_watermark_ok_safe
> |
> --- zone_watermark_ok_safe
> |
> |--72.66%-- kswapd
> | kthread
> | kernel_thread_helper
> |
> |--20.88%-- sleeping_prematurely.part.12
> | kswapd
> | kthread
> | kernel_thread_helper
> |
> --6.46%-- kthread
> kernel_thread_helper
>
We are also spending an astonishing amount of time in
sleeping_prematurely leading me to believe we are failing to balance the
zones and are continually under the min watermark for one of the zones.
We are never going to sleep because of this check;
if (total_scanned && (priority < DEF_PRIORITY - 2)) {
if (has_under_min_watermark_zone)
count_vm_event(KSWAPD_SKIP_CONGESTION_WAIT);
else
congestion_wait(BLK_RW_ASYNC, HZ/10);
}
However, I think this is a secondary effect to the failure of shrinkers
to do their work. If slabs were being shrunk, one would expect us to
be getting over the min watermark.
> 4.25% kswapd0 [kernel.kallsyms] [k] do_raw_spin_lock
> |
> --- do_raw_spin_lock
> |
> |--77.49%-- _raw_spin_lock
> | |
> | |--51.85%-- mb_cache_shrink_fn
> | | shrink_slab
> | | kswapd
> | | kthread
> | | kernel_thread_helper
> | |
> | --48.15%-- mem_cgroup_soft_limit_reclaim
> | kswapd
> | kthread
> | kernel_thread_helper
> |
Way hey, cgroups are also in the mix. How jolly.
Is systemd a common element of the machines hitting this bug by any
chance?
The remaining traces seem to be follow-on damage related to the three
issues of "shrinkers are bust in some manner" causing "we are not
getting over the min watermark" and as a side-show "we are spending lots
of time doing something unspecified but unhelpful in cgroups".
> <SNIP>
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 17:18 ` Mel Gorman
@ 2011-04-28 18:30 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 18:30 UTC (permalink / raw)
To: Mel Gorman
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 18:18 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote:
> > # Events: 6K cycles
> > #
> > # Overhead Command Shared Object Symbol
> > # ........ ........... ................... .......................................
> > #
> > 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab
> > |
> > --- shrink_slab
> > |
> > |--99.91%-- kswapd
> > | kthread
> > | kernel_thread_helper
> > --0.09%-- [...]
> >
>
> Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use
> compaction instead of lumpy reclaim" is related unless we are seeing
> two problems that happen to manifest in a similar manner.
>
> However, there were a number of changes made to dcache in particular
> for 2.6.38. Specifically thinks like dentry_kill use trylock and is
> happy to loop around if it fails to acquire anything. See things like
> this for example;
OK, so for this, I tried a 2.6.37 kernel. It doesn't work very well,
networking is hosed for no reason I can see (probably systemd / cgroups
problems).
However, it runs enough for me to say that the tar proceeds to
completion in a non-PREEMPT kernel. (I tried several times for good
measure). That makes this definitely a regression of some sort, but it
doesn't definitively identify the dcache code ... it could be an ext4
bug that got introduced in 2.6.38 either.
> static void try_prune_one_dentry(struct dentry *dentry)
> __releases(dentry->d_lock)
> {
> struct dentry *parent;
>
> parent = dentry_kill(dentry, 0);
> /*
> * If dentry_kill returns NULL, we have nothing more to do.
> * if it returns the same dentry, trylocks failed. In either
> * case, just loop again.
>
>
> If this in combination with many inodes being locked for whatever
> reason (writeback locking them maybe?) is causing the shrinker to
> return after zero progress, it could in turn cause kswapd to enter
> into a loop for longish periods of time in shrink_slab here;
>
> while (total_scan >= SHRINK_BATCH) {
> long this_scan = SHRINK_BATCH;
> int shrink_ret;
> int nr_before;
>
> nr_before = (*shrinker->shrink)(shrinker, 0, gfp_mask);
> shrink_ret = (*shrinker->shrink)(shrinker, this_scan,
> gfp_mask);
> if (shrink_ret == -1)
> break;
> if (shrink_ret < nr_before)
> ret += nr_before - shrink_ret;
> count_vm_events(SLABS_SCANNED, this_scan);
> total_scan -= this_scan;
>
> cond_resched();
> }
>
> That would explain this trace.
>
> > 9.98% kswapd0 [kernel.kallsyms] [k] shrink_zone
> > |
> > --- shrink_zone
> > |
> > |--99.46%-- kswapd
> > | kthread
> > | kernel_thread_helper
> > |
> > --0.54%-- kthread
> > kernel_thread_helper
> >
> > 7.70% kswapd0 [kernel.kallsyms] [k] kswapd
> > |
> > --- kswapd
> > kthread
> > kernel_thread_helper
> >
> > 5.40% kswapd0 [kernel.kallsyms] [k] zone_watermark_ok_safe
> > |
> > --- zone_watermark_ok_safe
> > |
> > |--72.66%-- kswapd
> > | kthread
> > | kernel_thread_helper
> > |
> > |--20.88%-- sleeping_prematurely.part.12
> > | kswapd
> > | kthread
> > | kernel_thread_helper
> > |
> > --6.46%-- kthread
> > kernel_thread_helper
> >
>
> We are also spending an astonishing amount of time in
> sleeping_prematurely leading me to believe we are failing to balance the
> zones and are continually under the min watermark for one of the zones.
> We are never going to sleep because of this check;
>
> if (total_scanned && (priority < DEF_PRIORITY - 2)) {
> if (has_under_min_watermark_zone)
> count_vm_event(KSWAPD_SKIP_CONGESTION_WAIT);
> else
> congestion_wait(BLK_RW_ASYNC, HZ/10);
> }
>
> However, I think this is a secondary effect to the failure of shrinkers
> to do their work. If slabs were being shrunk, one would expect us to
> be getting over the min watermark.
>
> > 4.25% kswapd0 [kernel.kallsyms] [k] do_raw_spin_lock
> > |
> > --- do_raw_spin_lock
> > |
> > |--77.49%-- _raw_spin_lock
> > | |
> > | |--51.85%-- mb_cache_shrink_fn
> > | | shrink_slab
> > | | kswapd
> > | | kthread
> > | | kernel_thread_helper
> > | |
> > | --48.15%-- mem_cgroup_soft_limit_reclaim
> > | kswapd
> > | kthread
> > | kernel_thread_helper
> > |
>
> Way hey, cgroups are also in the mix. How jolly.
>
> Is systemd a common element of the machines hitting this bug by any
> chance?
Well, yes, the bug report is against FC15, which needs cgroups for
systemd.
> The remaining traces seem to be follow-on damage related to the three
> issues of "shrinkers are bust in some manner" causing "we are not
> getting over the min watermark" and as a side-show "we are spending lots
> of time doing something unspecified but unhelpful in cgroups".
Heh, well find a way for me to verify this: I can't turn off cgroups
because systemd then won't work and the machine won't boot ...
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 18:30 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 18:30 UTC (permalink / raw)
To: Mel Gorman
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 18:18 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote:
> > # Events: 6K cycles
> > #
> > # Overhead Command Shared Object Symbol
> > # ........ ........... ................... .......................................
> > #
> > 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab
> > |
> > --- shrink_slab
> > |
> > |--99.91%-- kswapd
> > | kthread
> > | kernel_thread_helper
> > --0.09%-- [...]
> >
>
> Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use
> compaction instead of lumpy reclaim" is related unless we are seeing
> two problems that happen to manifest in a similar manner.
>
> However, there were a number of changes made to dcache in particular
> for 2.6.38. Specifically thinks like dentry_kill use trylock and is
> happy to loop around if it fails to acquire anything. See things like
> this for example;
OK, so for this, I tried a 2.6.37 kernel. It doesn't work very well,
networking is hosed for no reason I can see (probably systemd / cgroups
problems).
However, it runs enough for me to say that the tar proceeds to
completion in a non-PREEMPT kernel. (I tried several times for good
measure). That makes this definitely a regression of some sort, but it
doesn't definitively identify the dcache code ... it could be an ext4
bug that got introduced in 2.6.38 either.
> static void try_prune_one_dentry(struct dentry *dentry)
> __releases(dentry->d_lock)
> {
> struct dentry *parent;
>
> parent = dentry_kill(dentry, 0);
> /*
> * If dentry_kill returns NULL, we have nothing more to do.
> * if it returns the same dentry, trylocks failed. In either
> * case, just loop again.
>
>
> If this in combination with many inodes being locked for whatever
> reason (writeback locking them maybe?) is causing the shrinker to
> return after zero progress, it could in turn cause kswapd to enter
> into a loop for longish periods of time in shrink_slab here;
>
> while (total_scan >= SHRINK_BATCH) {
> long this_scan = SHRINK_BATCH;
> int shrink_ret;
> int nr_before;
>
> nr_before = (*shrinker->shrink)(shrinker, 0, gfp_mask);
> shrink_ret = (*shrinker->shrink)(shrinker, this_scan,
> gfp_mask);
> if (shrink_ret == -1)
> break;
> if (shrink_ret < nr_before)
> ret += nr_before - shrink_ret;
> count_vm_events(SLABS_SCANNED, this_scan);
> total_scan -= this_scan;
>
> cond_resched();
> }
>
> That would explain this trace.
>
> > 9.98% kswapd0 [kernel.kallsyms] [k] shrink_zone
> > |
> > --- shrink_zone
> > |
> > |--99.46%-- kswapd
> > | kthread
> > | kernel_thread_helper
> > |
> > --0.54%-- kthread
> > kernel_thread_helper
> >
> > 7.70% kswapd0 [kernel.kallsyms] [k] kswapd
> > |
> > --- kswapd
> > kthread
> > kernel_thread_helper
> >
> > 5.40% kswapd0 [kernel.kallsyms] [k] zone_watermark_ok_safe
> > |
> > --- zone_watermark_ok_safe
> > |
> > |--72.66%-- kswapd
> > | kthread
> > | kernel_thread_helper
> > |
> > |--20.88%-- sleeping_prematurely.part.12
> > | kswapd
> > | kthread
> > | kernel_thread_helper
> > |
> > --6.46%-- kthread
> > kernel_thread_helper
> >
>
> We are also spending an astonishing amount of time in
> sleeping_prematurely leading me to believe we are failing to balance the
> zones and are continually under the min watermark for one of the zones.
> We are never going to sleep because of this check;
>
> if (total_scanned && (priority < DEF_PRIORITY - 2)) {
> if (has_under_min_watermark_zone)
> count_vm_event(KSWAPD_SKIP_CONGESTION_WAIT);
> else
> congestion_wait(BLK_RW_ASYNC, HZ/10);
> }
>
> However, I think this is a secondary effect to the failure of shrinkers
> to do their work. If slabs were being shrunk, one would expect us to
> be getting over the min watermark.
>
> > 4.25% kswapd0 [kernel.kallsyms] [k] do_raw_spin_lock
> > |
> > --- do_raw_spin_lock
> > |
> > |--77.49%-- _raw_spin_lock
> > | |
> > | |--51.85%-- mb_cache_shrink_fn
> > | | shrink_slab
> > | | kswapd
> > | | kthread
> > | | kernel_thread_helper
> > | |
> > | --48.15%-- mem_cgroup_soft_limit_reclaim
> > | kswapd
> > | kthread
> > | kernel_thread_helper
> > |
>
> Way hey, cgroups are also in the mix. How jolly.
>
> Is systemd a common element of the machines hitting this bug by any
> chance?
Well, yes, the bug report is against FC15, which needs cgroups for
systemd.
> The remaining traces seem to be follow-on damage related to the three
> issues of "shrinkers are bust in some manner" causing "we are not
> getting over the min watermark" and as a side-show "we are spending lots
> of time doing something unspecified but unhelpful in cgroups".
Heh, well find a way for me to verify this: I can't turn off cgroups
because systemd then won't work and the machine won't boot ...
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 18:30 ` James Bottomley
@ 2011-04-28 19:21 ` Mel Gorman
-1 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-04-28 19:21 UTC (permalink / raw)
To: James Bottomley
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote:
> On Thu, 2011-04-28 at 18:18 +0100, Mel Gorman wrote:
> > On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote:
> > > # Events: 6K cycles
> > > #
> > > # Overhead Command Shared Object Symbol
> > > # ........ ........... ................... .......................................
> > > #
> > > 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab
> > > |
> > > --- shrink_slab
> > > |
> > > |--99.91%-- kswapd
> > > | kthread
> > > | kernel_thread_helper
> > > --0.09%-- [...]
> > >
> >
> > Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use
> > compaction instead of lumpy reclaim" is related unless we are seeing
> > two problems that happen to manifest in a similar manner.
> >
> > However, there were a number of changes made to dcache in particular
> > for 2.6.38. Specifically thinks like dentry_kill use trylock and is
> > happy to loop around if it fails to acquire anything. See things like
> > this for example;
>
> OK, so for this, I tried a 2.6.37 kernel. It doesn't work very well,
> networking is hosed for no reason I can see (probably systemd / cgroups
> problems).
>
> However, it runs enough for me to say that the tar proceeds to
> completion in a non-PREEMPT kernel. (I tried several times for good
> measure). That makes this definitely a regression of some sort, but it
> doesn't definitively identify the dcache code ... it could be an ext4
> bug that got introduced in 2.6.38 either.
>
True, it could be any shrinker and dcache is just a guess.
> > <SNIP>
> >
> > Way hey, cgroups are also in the mix. How jolly.
> >
> > Is systemd a common element of the machines hitting this bug by any
> > chance?
>
> Well, yes, the bug report is against FC15, which needs cgroups for
> systemd.
>
Ok although we do not have direct evidence that it's the problem yet. A
broken shrinker could just mean we are also trying to aggressively
reclaim in cgroups.
> > The remaining traces seem to be follow-on damage related to the three
> > issues of "shrinkers are bust in some manner" causing "we are not
> > getting over the min watermark" and as a side-show "we are spending lots
> > of time doing something unspecified but unhelpful in cgroups".
>
> Heh, well find a way for me to verify this: I can't turn off cgroups
> because systemd then won't work and the machine won't boot ...
>
Same testcase, same kernel but a distro that is not using systemd to
verify if cgroups are the problem. Not ideal I know. When I'm back
online Tuesday, I'll try reproducing this on a !Fedora distribution. In
the meantime, the following untested hatchet job might spit out
which shrinker we are getting stuck in. It is also breaking out of
the shrink_slab loop so it'd even be interesting to see if the bug
is mitigated in any way.
diff --git a/mm/vmscan.c b/mm/vmscan.c
index c74a501..ed99104 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -225,6 +225,7 @@ unsigned long shrink_slab(unsigned long scanned, gfp_t gfp_mask,
{
struct shrinker *shrinker;
unsigned long ret = 0;
+ unsigned long shrink_expired = jiffies + HZ;
if (scanned == 0)
scanned = SWAP_CLUSTER_MAX;
@@ -270,6 +271,14 @@ unsigned long shrink_slab(unsigned long scanned, gfp_t gfp_mask,
gfp_mask);
if (shrink_ret == -1)
break;
+ if (time_after(jiffies, shrink_expired)) {
+ printk(KERN_WARNING "Slab shrinker %p gone mental"
+ " comm=%s nr=%ld\n",
+ shrinker->shrink,
+ current->comm,
+ shrinker->nr);
+ break;
+ }
if (shrink_ret < nr_before)
ret += nr_before - shrink_ret;
count_vm_events(SLABS_SCANNED, this_scan);
^ permalink raw reply related [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 19:21 ` Mel Gorman
0 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-04-28 19:21 UTC (permalink / raw)
To: James Bottomley
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote:
> On Thu, 2011-04-28 at 18:18 +0100, Mel Gorman wrote:
> > On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote:
> > > # Events: 6K cycles
> > > #
> > > # Overhead Command Shared Object Symbol
> > > # ........ ........... ................... .......................................
> > > #
> > > 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab
> > > |
> > > --- shrink_slab
> > > |
> > > |--99.91%-- kswapd
> > > | kthread
> > > | kernel_thread_helper
> > > --0.09%-- [...]
> > >
> >
> > Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use
> > compaction instead of lumpy reclaim" is related unless we are seeing
> > two problems that happen to manifest in a similar manner.
> >
> > However, there were a number of changes made to dcache in particular
> > for 2.6.38. Specifically thinks like dentry_kill use trylock and is
> > happy to loop around if it fails to acquire anything. See things like
> > this for example;
>
> OK, so for this, I tried a 2.6.37 kernel. It doesn't work very well,
> networking is hosed for no reason I can see (probably systemd / cgroups
> problems).
>
> However, it runs enough for me to say that the tar proceeds to
> completion in a non-PREEMPT kernel. (I tried several times for good
> measure). That makes this definitely a regression of some sort, but it
> doesn't definitively identify the dcache code ... it could be an ext4
> bug that got introduced in 2.6.38 either.
>
True, it could be any shrinker and dcache is just a guess.
> > <SNIP>
> >
> > Way hey, cgroups are also in the mix. How jolly.
> >
> > Is systemd a common element of the machines hitting this bug by any
> > chance?
>
> Well, yes, the bug report is against FC15, which needs cgroups for
> systemd.
>
Ok although we do not have direct evidence that it's the problem yet. A
broken shrinker could just mean we are also trying to aggressively
reclaim in cgroups.
> > The remaining traces seem to be follow-on damage related to the three
> > issues of "shrinkers are bust in some manner" causing "we are not
> > getting over the min watermark" and as a side-show "we are spending lots
> > of time doing something unspecified but unhelpful in cgroups".
>
> Heh, well find a way for me to verify this: I can't turn off cgroups
> because systemd then won't work and the machine won't boot ...
>
Same testcase, same kernel but a distro that is not using systemd to
verify if cgroups are the problem. Not ideal I know. When I'm back
online Tuesday, I'll try reproducing this on a !Fedora distribution. In
the meantime, the following untested hatchet job might spit out
which shrinker we are getting stuck in. It is also breaking out of
the shrink_slab loop so it'd even be interesting to see if the bug
is mitigated in any way.
diff --git a/mm/vmscan.c b/mm/vmscan.c
index c74a501..ed99104 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -225,6 +225,7 @@ unsigned long shrink_slab(unsigned long scanned, gfp_t gfp_mask,
{
struct shrinker *shrinker;
unsigned long ret = 0;
+ unsigned long shrink_expired = jiffies + HZ;
if (scanned == 0)
scanned = SWAP_CLUSTER_MAX;
@@ -270,6 +271,14 @@ unsigned long shrink_slab(unsigned long scanned, gfp_t gfp_mask,
gfp_mask);
if (shrink_ret == -1)
break;
+ if (time_after(jiffies, shrink_expired)) {
+ printk(KERN_WARNING "Slab shrinker %p gone mental"
+ " comm=%s nr=%ld\n",
+ shrinker->shrink,
+ current->comm,
+ shrinker->nr);
+ break;
+ }
if (shrink_ret < nr_before)
ret += nr_before - shrink_ret;
count_vm_events(SLABS_SCANNED, this_scan);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 19:21 ` Mel Gorman
@ 2011-04-28 19:59 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 19:59 UTC (permalink / raw)
To: Mel Gorman
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 20:21 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote:
> > > Way hey, cgroups are also in the mix. How jolly.
> > >
> > > Is systemd a common element of the machines hitting this bug by any
> > > chance?
> >
> > Well, yes, the bug report is against FC15, which needs cgroups for
> > systemd.
> >
>
> Ok although we do not have direct evidence that it's the problem yet. A
> broken shrinker could just mean we are also trying to aggressively
> reclaim in cgroups.
>
> > > The remaining traces seem to be follow-on damage related to the three
> > > issues of "shrinkers are bust in some manner" causing "we are not
> > > getting over the min watermark" and as a side-show "we are spending lots
> > > of time doing something unspecified but unhelpful in cgroups".
> >
> > Heh, well find a way for me to verify this: I can't turn off cgroups
> > because systemd then won't work and the machine won't boot ...
> >
>
> Same testcase, same kernel but a distro that is not using systemd to
> verify if cgroups are the problem. Not ideal I know. When I'm back
> online Tuesday, I'll try reproducing this on a !Fedora distribution. In
> the meantime, the following untested hatchet job might spit out
> which shrinker we are getting stuck in. It is also breaking out of
> the shrink_slab loop so it'd even be interesting to see if the bug
> is mitigated in any way.
Actually, talking to Chris, I think I can get the system up using
init=/bin/bash without systemd, so I can try the no cgroup config.
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index c74a501..ed99104 100644
In the mean time, this patch produces:
(that's nothing ... apparently the trace doesn't activate when kswapd
goes mad).
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 19:59 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 19:59 UTC (permalink / raw)
To: Mel Gorman
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 20:21 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote:
> > > Way hey, cgroups are also in the mix. How jolly.
> > >
> > > Is systemd a common element of the machines hitting this bug by any
> > > chance?
> >
> > Well, yes, the bug report is against FC15, which needs cgroups for
> > systemd.
> >
>
> Ok although we do not have direct evidence that it's the problem yet. A
> broken shrinker could just mean we are also trying to aggressively
> reclaim in cgroups.
>
> > > The remaining traces seem to be follow-on damage related to the three
> > > issues of "shrinkers are bust in some manner" causing "we are not
> > > getting over the min watermark" and as a side-show "we are spending lots
> > > of time doing something unspecified but unhelpful in cgroups".
> >
> > Heh, well find a way for me to verify this: I can't turn off cgroups
> > because systemd then won't work and the machine won't boot ...
> >
>
> Same testcase, same kernel but a distro that is not using systemd to
> verify if cgroups are the problem. Not ideal I know. When I'm back
> online Tuesday, I'll try reproducing this on a !Fedora distribution. In
> the meantime, the following untested hatchet job might spit out
> which shrinker we are getting stuck in. It is also breaking out of
> the shrink_slab loop so it'd even be interesting to see if the bug
> is mitigated in any way.
Actually, talking to Chris, I think I can get the system up using
init=/bin/bash without systemd, so I can try the no cgroup config.
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index c74a501..ed99104 100644
In the mean time, this patch produces:
(that's nothing ... apparently the trace doesn't activate when kswapd
goes mad).
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 19:59 ` James Bottomley
@ 2011-04-28 20:27 ` Mel Gorman
-1 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-04-28 20:27 UTC (permalink / raw)
To: James Bottomley
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, Apr 28, 2011 at 02:59:27PM -0500, James Bottomley wrote:
> On Thu, 2011-04-28 at 20:21 +0100, Mel Gorman wrote:
> > On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote:
> > > > Way hey, cgroups are also in the mix. How jolly.
> > > >
> > > > Is systemd a common element of the machines hitting this bug by any
> > > > chance?
> > >
> > > Well, yes, the bug report is against FC15, which needs cgroups for
> > > systemd.
> > >
> >
> > Ok although we do not have direct evidence that it's the problem yet. A
> > broken shrinker could just mean we are also trying to aggressively
> > reclaim in cgroups.
> >
> > > > The remaining traces seem to be follow-on damage related to the three
> > > > issues of "shrinkers are bust in some manner" causing "we are not
> > > > getting over the min watermark" and as a side-show "we are spending lots
> > > > of time doing something unspecified but unhelpful in cgroups".
> > >
> > > Heh, well find a way for me to verify this: I can't turn off cgroups
> > > because systemd then won't work and the machine won't boot ...
> > >
> >
> > Same testcase, same kernel but a distro that is not using systemd to
> > verify if cgroups are the problem. Not ideal I know. When I'm back
> > online Tuesday, I'll try reproducing this on a !Fedora distribution. In
> > the meantime, the following untested hatchet job might spit out
> > which shrinker we are getting stuck in. It is also breaking out of
> > the shrink_slab loop so it'd even be interesting to see if the bug
> > is mitigated in any way.
>
> Actually, talking to Chris, I think I can get the system up using
> init=/bin/bash without systemd, so I can try the no cgroup config.
>
> > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > index c74a501..ed99104 100644
>
> In the mean time, this patch produces:
>
> (that's nothing ... apparently the trace doesn't activate when kswapd
> goes mad).
>
Or is looping there for shorter than we expect. HZ/10?
--
Mel Gorman
SUSE Labs
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 20:27 ` Mel Gorman
0 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-04-28 20:27 UTC (permalink / raw)
To: James Bottomley
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, Apr 28, 2011 at 02:59:27PM -0500, James Bottomley wrote:
> On Thu, 2011-04-28 at 20:21 +0100, Mel Gorman wrote:
> > On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote:
> > > > Way hey, cgroups are also in the mix. How jolly.
> > > >
> > > > Is systemd a common element of the machines hitting this bug by any
> > > > chance?
> > >
> > > Well, yes, the bug report is against FC15, which needs cgroups for
> > > systemd.
> > >
> >
> > Ok although we do not have direct evidence that it's the problem yet. A
> > broken shrinker could just mean we are also trying to aggressively
> > reclaim in cgroups.
> >
> > > > The remaining traces seem to be follow-on damage related to the three
> > > > issues of "shrinkers are bust in some manner" causing "we are not
> > > > getting over the min watermark" and as a side-show "we are spending lots
> > > > of time doing something unspecified but unhelpful in cgroups".
> > >
> > > Heh, well find a way for me to verify this: I can't turn off cgroups
> > > because systemd then won't work and the machine won't boot ...
> > >
> >
> > Same testcase, same kernel but a distro that is not using systemd to
> > verify if cgroups are the problem. Not ideal I know. When I'm back
> > online Tuesday, I'll try reproducing this on a !Fedora distribution. In
> > the meantime, the following untested hatchet job might spit out
> > which shrinker we are getting stuck in. It is also breaking out of
> > the shrink_slab loop so it'd even be interesting to see if the bug
> > is mitigated in any way.
>
> Actually, talking to Chris, I think I can get the system up using
> init=/bin/bash without systemd, so I can try the no cgroup config.
>
> > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > index c74a501..ed99104 100644
>
> In the mean time, this patch produces:
>
> (that's nothing ... apparently the trace doesn't activate when kswapd
> goes mad).
>
Or is looping there for shorter than we expect. HZ/10?
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 19:59 ` James Bottomley
@ 2011-04-28 21:12 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 21:12 UTC (permalink / raw)
To: Mel Gorman
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote:
> Actually, talking to Chris, I think I can get the system up using
> init=/bin/bash without systemd, so I can try the no cgroup config.
OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back
runs of untar without locking or getting kswapd pegged, so I'm pretty
certain this is cgroups related. The next steps are to turn cgroups
back on but try disabling the memory and IO controllers.
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 21:12 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 21:12 UTC (permalink / raw)
To: Mel Gorman
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote:
> Actually, talking to Chris, I think I can get the system up using
> init=/bin/bash without systemd, so I can try the no cgroup config.
OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back
runs of untar without locking or getting kswapd pegged, so I'm pretty
certain this is cgroups related. The next steps are to turn cgroups
back on but try disabling the memory and IO controllers.
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 14:58 ` Colin Ian King
@ 2011-04-28 22:40 ` Jan Kara
0 siblings, 0 replies; 138+ messages in thread
From: Jan Kara @ 2011-04-28 22:40 UTC (permalink / raw)
To: Colin Ian King
Cc: Jan Kara, Chris Mason, James Bottomley, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu 28-04-11 15:58:21, Colin Ian King wrote:
> On Thu, 2011-04-28 at 16:33 +0200, Jan Kara wrote:
> > On Thu 28-04-11 16:25:51, Jan Kara wrote:
> > > On Thu 28-04-11 15:01:22, Colin Ian King wrote:
> > > >
> > > > > Could you post the soft lockups you're seeing?
> > > >
> > > > As requested, attached
> > > Hum, what keeps puzzling me is that in all the cases of hangs I've seen
> > > so far, we are stuck waiting for IO to finish for a long time - e.g. in the
> > > traces below kjournald waits for PageWriteback bit to get cleared. Also we
> > > are stuck waiting for page locks which might be because those pages are
> > > being read in? All in all it seems that the IO is just incredibly slow.
> > >
> > > But it's not clear to me what pushes us into that situation (especially
> > > since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the
> > > underlying blocks are not already allocated.
> > Hmm, maybe because the system is under memory pressure (and kswapd is not
> > able to get rid of dirty pages), we page out clean pages. Thus also pages
> > of executables which need to be paged in soon anyway thus putting heavy
> > read load on the system which makes writes crawl? I'm not sure why
> > compaction should make this any worse but maybe it can.
> >
> > James, Colin, can you capture output of 'vmstat 1' while you do the
> > copying? Thanks.
>
> Attached.
Thanks. So I there are a few interesting points in the vmstat output:
For first 30 seconds, we are happily copying data - relatively steady read
throughput (about 20-40 MB/s) and occasional peak from flusher thread
flushing dirty data. During this time free memory drops from about 1.4 GB
to about 22!!! MB - mm seems to like to really use the machine ;). Then
things get interesting:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 1 0 83372 5228 1450776 0 0 39684 90132 450 918 0 4 74 22
0 1 0 22416 5228 1509864 0 0 29452 48492 403 869 1 2 80 18
2 0 0 20056 5384 1513996 0 0 2248 2116 434 1191 4 4 71 21
0 1 0 19800 5932 1514600 0 0 644 104 454 1166 8 3 64 24
1 0 0 21848 5940 1515244 0 0 292 380 468 1775 15 6 75 3
1 0 0 20936 5940 1515876 0 0 296 296 496 1846 18 8 74 0
1 0 0 17792 5940 1516580 0 0 356 356 484 1862 23 8 69 0
1 0 0 17544 5940 1517364 0 0 412 412 482 1812 16 7 77 0
4 0 0 18148 5948 1517968 0 0 288 344 436 1749 19 9 69 3
0 2 220 137528 1616 1402468 0 220 74708 2164 849 1806 3 6 63 28
0 3 224 36184 1628 1499648 0 4 50820 86204 532 1272 0 4 64 32
0 2 19680 53688 1628 1484388 32 19456 6080 62972 242 287 0 2 63 34
0 2 36928 1407432 1356 150980 0 17252 1564 17276 368 764 1 3 73 22
So when free memory reached about 20 MB, both read and write activity
almost stalled for 7 s (probably everybody waits for free memory). Then
mm manages to free 100 MB from page cache, things move on for two seconds,
then we swap out! about 36 MB and page reclaim also finally decides it
maybe has too much of page cache and reaps most of it (1.3 GB in one go).
Then things get going again, although there are still occasional stalls
such as this (about 30s later):
1 3 36688 753192 1208 792344 0 0 35948 32768 435 6625 0 6 61 33
0 2 36668 754996 1344 792760 0 0 252 58736 2832 16239 0 1 60 39
0 2 36668 750132 1388 796688 0 0 2508 1524 325 959 1 3 68 28
1 0 36668 751160 1400 797968 0 0 620 620 460 1470 6 6 50 38
1 0 36668 750516 1400 798520 0 0 300 300 412 1764 17 8 75 1
1 0 36668 750648 1408 799108 0 0 280 340 423 1816 18 6 73 3
1 0 36668 748856 1408 799752 0 0 336 328 409 1788 18 8 75 0
1 0 36668 748120 1416 800604 0 0 428 452 407 1723 14 10 75 2
1 0 36668 750048 1416 801176 0 0 296 296 405 1779 18 7 75 1
0 1 36668 650428 1420 897252 0 0 48100 556 658 1718 10 3 71 15
0 2 36668 505444 1424 1037012 0 0 69888 90272 686 1491 1 4 68 27
0 1 36668 479264 1428 1063372 0 0 12984 40896 324 674 1 1 76 23
...
I'm not sure what we were blocked on here since there is still plenty of
free memory (750 MB). These stalls repeat once in a while but things go on.
Then at about 350s, things just stop:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 1 75280 73564 1844 1503848 0 0 43396 81976 627 1061 0 25 42 32
3 3 75280 73344 1852 1504256 0 0 256 20 240 149 0 26 25 49
3 3 75280 73344 1852 1504268 0 0 0 0 265 140 0 29 13 58
3 3 75280 73468 1852 1504232 0 0 0 0 265 132 0 22 34 44
3 3 75280 73468 1852 1504232 0 0 0 0 339 283 0 25 26 49
3 3 75280 73468 1852 1504232 0 0 0 0 362 327 0 25 25 50
3 3 75280 73468 1852 1504232 0 0 0 0 317 320 0 26 25 49
3 3 75280 73468 1852 1504232 0 0 0 0 361 343 0 26 25 50
and nothing really happens for 150 s, except more and more tasks blocking
in D state (second column).
3 6 75272 73416 1872 1503872 0 0 0 0 445 700 0 25 25 50
0 7 75264 67940 1884 1509008 64 0 5056 16 481 876 0 22 23 55
Then suddently things get going again:
0 2 75104 76808 1892 1502552 352 0 14292 40456 459 14865 0 2 39 59
0 1 75104 75704 1900 1503412 0 0 820 32 405 788 1 1 72 27
1 0 75104 76512 1904 1505576 0 0 1068 1072 454 1586 8 7 74 11
I guess this 150 s stall is when kernel barfs the "task blocked for more
than 30 seconds" messages. And from the traces we know that everyone is
waiting for PageWriteback or page lock during this time. Also James's vmstat
report shows that IO is stalled when kswapd is spinning. Really strange.
James in the meantime identified that cgroups are somehow involved. Are you
using systemd by any chance? Maybe cgroup IO throttling screws us?
Honza
> > > [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds.
> > > [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000
> > > [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000
> > > [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00
> > > [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00
> > > [ 287.088400] Call Trace:
> > > [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170
> > > [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0
> > > [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> > > [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> > > [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> > > [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> > > [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230
> > > [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds.
> > > [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000
> > > [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000
> > > [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00
> > > [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00
> > > [ 287.088484] Call Trace:
> > > [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60
> > > [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60
> > > [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> > > [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> > > [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20
> > > [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0
> > > [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90
> > > [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds.
> > > [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000
> > > [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000
> > > [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00
> > > [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00
> > > [ 287.088577] Call Trace:
> > > [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> > > [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> > > [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> > > [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds.
> > > [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000
> > > [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000
> > > [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00
> > > [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00
> > > [ 287.088647] Call Trace:
> > > [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190
> > > [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30
> > > [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170
> > > [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190
> > > [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0
> > > [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220
> > > [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
> > > [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220
> > > [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0
> > > [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
> > > [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0
> > > [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
> > > [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds.
> > > [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000
> > > [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000
> > > [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00
> > > [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00
> > > [ 287.088723] Call Trace:
> > > [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> > > [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> > > [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> > > [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500
> > > [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0
> > > [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70
> > > [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70
> > > [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40
> > > [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40
> > > [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00
> > > [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300
> > > [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00
> > > [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0
> > > [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80
> > > [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0
> > > --
> > > Jan Kara <jack@suse.cz>
> > > SUSE Labs, CR
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 22:40 ` Jan Kara
0 siblings, 0 replies; 138+ messages in thread
From: Jan Kara @ 2011-04-28 22:40 UTC (permalink / raw)
To: Colin Ian King
Cc: Jan Kara, Chris Mason, James Bottomley, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu 28-04-11 15:58:21, Colin Ian King wrote:
> On Thu, 2011-04-28 at 16:33 +0200, Jan Kara wrote:
> > On Thu 28-04-11 16:25:51, Jan Kara wrote:
> > > On Thu 28-04-11 15:01:22, Colin Ian King wrote:
> > > >
> > > > > Could you post the soft lockups you're seeing?
> > > >
> > > > As requested, attached
> > > Hum, what keeps puzzling me is that in all the cases of hangs I've seen
> > > so far, we are stuck waiting for IO to finish for a long time - e.g. in the
> > > traces below kjournald waits for PageWriteback bit to get cleared. Also we
> > > are stuck waiting for page locks which might be because those pages are
> > > being read in? All in all it seems that the IO is just incredibly slow.
> > >
> > > But it's not clear to me what pushes us into that situation (especially
> > > since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the
> > > underlying blocks are not already allocated.
> > Hmm, maybe because the system is under memory pressure (and kswapd is not
> > able to get rid of dirty pages), we page out clean pages. Thus also pages
> > of executables which need to be paged in soon anyway thus putting heavy
> > read load on the system which makes writes crawl? I'm not sure why
> > compaction should make this any worse but maybe it can.
> >
> > James, Colin, can you capture output of 'vmstat 1' while you do the
> > copying? Thanks.
>
> Attached.
Thanks. So I there are a few interesting points in the vmstat output:
For first 30 seconds, we are happily copying data - relatively steady read
throughput (about 20-40 MB/s) and occasional peak from flusher thread
flushing dirty data. During this time free memory drops from about 1.4 GB
to about 22!!! MB - mm seems to like to really use the machine ;). Then
things get interesting:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 1 0 83372 5228 1450776 0 0 39684 90132 450 918 0 4 74 22
0 1 0 22416 5228 1509864 0 0 29452 48492 403 869 1 2 80 18
2 0 0 20056 5384 1513996 0 0 2248 2116 434 1191 4 4 71 21
0 1 0 19800 5932 1514600 0 0 644 104 454 1166 8 3 64 24
1 0 0 21848 5940 1515244 0 0 292 380 468 1775 15 6 75 3
1 0 0 20936 5940 1515876 0 0 296 296 496 1846 18 8 74 0
1 0 0 17792 5940 1516580 0 0 356 356 484 1862 23 8 69 0
1 0 0 17544 5940 1517364 0 0 412 412 482 1812 16 7 77 0
4 0 0 18148 5948 1517968 0 0 288 344 436 1749 19 9 69 3
0 2 220 137528 1616 1402468 0 220 74708 2164 849 1806 3 6 63 28
0 3 224 36184 1628 1499648 0 4 50820 86204 532 1272 0 4 64 32
0 2 19680 53688 1628 1484388 32 19456 6080 62972 242 287 0 2 63 34
0 2 36928 1407432 1356 150980 0 17252 1564 17276 368 764 1 3 73 22
So when free memory reached about 20 MB, both read and write activity
almost stalled for 7 s (probably everybody waits for free memory). Then
mm manages to free 100 MB from page cache, things move on for two seconds,
then we swap out! about 36 MB and page reclaim also finally decides it
maybe has too much of page cache and reaps most of it (1.3 GB in one go).
Then things get going again, although there are still occasional stalls
such as this (about 30s later):
1 3 36688 753192 1208 792344 0 0 35948 32768 435 6625 0 6 61 33
0 2 36668 754996 1344 792760 0 0 252 58736 2832 16239 0 1 60 39
0 2 36668 750132 1388 796688 0 0 2508 1524 325 959 1 3 68 28
1 0 36668 751160 1400 797968 0 0 620 620 460 1470 6 6 50 38
1 0 36668 750516 1400 798520 0 0 300 300 412 1764 17 8 75 1
1 0 36668 750648 1408 799108 0 0 280 340 423 1816 18 6 73 3
1 0 36668 748856 1408 799752 0 0 336 328 409 1788 18 8 75 0
1 0 36668 748120 1416 800604 0 0 428 452 407 1723 14 10 75 2
1 0 36668 750048 1416 801176 0 0 296 296 405 1779 18 7 75 1
0 1 36668 650428 1420 897252 0 0 48100 556 658 1718 10 3 71 15
0 2 36668 505444 1424 1037012 0 0 69888 90272 686 1491 1 4 68 27
0 1 36668 479264 1428 1063372 0 0 12984 40896 324 674 1 1 76 23
...
I'm not sure what we were blocked on here since there is still plenty of
free memory (750 MB). These stalls repeat once in a while but things go on.
Then at about 350s, things just stop:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 1 75280 73564 1844 1503848 0 0 43396 81976 627 1061 0 25 42 32
3 3 75280 73344 1852 1504256 0 0 256 20 240 149 0 26 25 49
3 3 75280 73344 1852 1504268 0 0 0 0 265 140 0 29 13 58
3 3 75280 73468 1852 1504232 0 0 0 0 265 132 0 22 34 44
3 3 75280 73468 1852 1504232 0 0 0 0 339 283 0 25 26 49
3 3 75280 73468 1852 1504232 0 0 0 0 362 327 0 25 25 50
3 3 75280 73468 1852 1504232 0 0 0 0 317 320 0 26 25 49
3 3 75280 73468 1852 1504232 0 0 0 0 361 343 0 26 25 50
and nothing really happens for 150 s, except more and more tasks blocking
in D state (second column).
3 6 75272 73416 1872 1503872 0 0 0 0 445 700 0 25 25 50
0 7 75264 67940 1884 1509008 64 0 5056 16 481 876 0 22 23 55
Then suddently things get going again:
0 2 75104 76808 1892 1502552 352 0 14292 40456 459 14865 0 2 39 59
0 1 75104 75704 1900 1503412 0 0 820 32 405 788 1 1 72 27
1 0 75104 76512 1904 1505576 0 0 1068 1072 454 1586 8 7 74 11
I guess this 150 s stall is when kernel barfs the "task blocked for more
than 30 seconds" messages. And from the traces we know that everyone is
waiting for PageWriteback or page lock during this time. Also James's vmstat
report shows that IO is stalled when kswapd is spinning. Really strange.
James in the meantime identified that cgroups are somehow involved. Are you
using systemd by any chance? Maybe cgroup IO throttling screws us?
Honza
> > > [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds.
> > > [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000
> > > [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000
> > > [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00
> > > [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00
> > > [ 287.088400] Call Trace:
> > > [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170
> > > [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0
> > > [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> > > [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> > > [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> > > [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> > > [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230
> > > [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds.
> > > [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000
> > > [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000
> > > [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00
> > > [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00
> > > [ 287.088484] Call Trace:
> > > [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60
> > > [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60
> > > [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> > > [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> > > [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20
> > > [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0
> > > [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90
> > > [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds.
> > > [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000
> > > [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000
> > > [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00
> > > [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00
> > > [ 287.088577] Call Trace:
> > > [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> > > [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> > > [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> > > [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds.
> > > [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000
> > > [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000
> > > [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00
> > > [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00
> > > [ 287.088647] Call Trace:
> > > [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190
> > > [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30
> > > [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170
> > > [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190
> > > [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0
> > > [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220
> > > [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
> > > [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220
> > > [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0
> > > [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
> > > [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0
> > > [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
> > > [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds.
> > > [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000
> > > [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000
> > > [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00
> > > [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00
> > > [ 287.088723] Call Trace:
> > > [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> > > [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> > > [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> > > [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500
> > > [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0
> > > [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70
> > > [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70
> > > [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40
> > > [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40
> > > [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00
> > > [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300
> > > [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00
> > > [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0
> > > [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80
> > > [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0
> > > --
> > > Jan Kara <jack@suse.cz>
> > > SUSE Labs, CR
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 21:12 ` James Bottomley
@ 2011-04-28 22:43 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 22:43 UTC (permalink / raw)
To: Mel Gorman
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 16:12 -0500, James Bottomley wrote:
> On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote:
> > Actually, talking to Chris, I think I can get the system up using
> > init=/bin/bash without systemd, so I can try the no cgroup config.
>
> OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back
> runs of untar without locking or getting kswapd pegged, so I'm pretty
> certain this is cgroups related. The next steps are to turn cgroups
> back on but try disabling the memory and IO controllers.
I tried non-PREEMPT CGROUP but disabled GROUP_MEM_RES_CTLR.
The results are curious: the tar does complete (I've done three back to
back). However, I did get one soft lockup in kswapd (below). But the
system recovers instead of halting I/O and hanging like it did
previously.
The soft lockup is in shrink_slab, so perhaps it's a combination of slab
shrinker and cgroup memory controller issues?
James
---
[ 670.823843] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46]
[ 670.825472] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel btusb snd_hda_codec snd_hwdep iwlagn snd_seq mac80211 bluetooth snd_seq_device uvcvideo snd_pcm cfg80211 wmi microcode e1000e videodev xhci_hcd rfkill snd_timer iTCO_wdt v4l2_compat_ioctl32 iTCO_vendor_support pcspkr i2c_i801 snd soundcore snd_page_alloc joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 670.830864] CPU 2
[ 670.830881] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel btusb snd_hda_codec snd_hwdep iwlagn snd_seq mac80211 bluetooth snd_seq_device uvcvideo snd_pcm cfg80211 wmi microcode e1000e videodev xhci_hcd rfkill snd_timer iTCO_wdt v4l2_compat_ioctl32 iTCO_vendor_support pcspkr i2c_i801 snd soundcore snd_page_alloc joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 670.838385]
[ 670.840289] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #3 LENOVO 4170CTO/4170CTO
[ 670.842193] RIP: 0010:[<ffffffff810e07cb>] [<ffffffff810e07cb>] shrink_slab+0x86/0x166
[ 670.844063] RSP: 0018:ffff88006eea5da0 EFLAGS: 00000206
[ 670.845881] RAX: 0000000000000000 RBX: ffff88006eea5de0 RCX: 0000000000000002
[ 670.847652] RDX: 0000000000000000 RSI: ffff88006eea5d60 RDI: ffff88006eea5d60
[ 670.849394] RBP: ffff88006eea5de0 R08: 000000000000000c R09: 0000000000000000
[ 670.851091] R10: 0000000000000001 R11: 000000000000005f R12: ffffffff8147b50e
[ 670.852733] R13: ffff8801005e6e00 R14: 0000000000000010 R15: 0000000000017fb6
[ 670.854351] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 670.855968] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 670.857555] CR2: 00000037d90ae040 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 670.859138] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 670.860720] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 670.862320] Process kswapd0 (pid: 46, threadinfo ffff88006eea4000, task ffff88006eeb0000)
[ 670.863932] Stack:
[ 670.865477] 0000000000000001 0000000000000080 ffff880000000002 ffff8801005e6e00
[ 670.867023] ffff8801005e6000 0000000000000002 0000000000000000 000000000000000c
[ 670.868558] ffff88006eea5ee0 ffffffff810e308c 0000000000000003 ffff88006eeb0000
[ 670.870120] Call Trace:
[ 670.871652] [<ffffffff810e308c>] kswapd+0x4f0/0x774
[ 670.873218] [<ffffffff810e2b9c>] ? try_to_free_pages+0xe5/0xe5
[ 670.874786] [<ffffffff8106ce57>] kthread+0x84/0x8c
[ 670.876327] [<ffffffff8147bc64>] kernel_thread_helper+0x4/0x10
[ 670.877871] [<ffffffff8106cdd3>] ? kthread_worker_fn+0x148/0x148
[ 670.879403] [<ffffffff8147bc60>] ? gs_change+0x13/0x13
[ 670.880932] Code: 83 eb 10 e9 ce 00 00 00 44 89 f2 31 f6 48 89 df ff 13 48 63 4b 08 4c 63 e8 48 8b 45 c8 31 d2 48 f7 f1 31 d2 49 0f af c5 49 f7 f7
[ 670.881086] 03 43 20 48 85 c0 48 89 43 20 79 18 48 8b 33 48 89 c2 48 c7
[ 670.884285] Call Trace:
[ 670.885884] [<ffffffff810e308c>] kswapd+0x4f0/0x774
[ 670.887462] [<ffffffff810e2b9c>] ? try_to_free_pages+0xe5/0xe5
[ 670.889031] [<ffffffff8106ce57>] kthread+0x84/0x8c
[ 670.890578] [<ffffffff8147bc64>] kernel_thread_helper+0x4/0x10
[ 670.892130] [<ffffffff8106cdd3>] ? kthread_worker_fn+0x148/0x148
[ 670.893653] [<ffffffff8147bc60>] ? gs_change+0x13/0x13
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 22:43 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 22:43 UTC (permalink / raw)
To: Mel Gorman
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 16:12 -0500, James Bottomley wrote:
> On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote:
> > Actually, talking to Chris, I think I can get the system up using
> > init=/bin/bash without systemd, so I can try the no cgroup config.
>
> OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back
> runs of untar without locking or getting kswapd pegged, so I'm pretty
> certain this is cgroups related. The next steps are to turn cgroups
> back on but try disabling the memory and IO controllers.
I tried non-PREEMPT CGROUP but disabled GROUP_MEM_RES_CTLR.
The results are curious: the tar does complete (I've done three back to
back). However, I did get one soft lockup in kswapd (below). But the
system recovers instead of halting I/O and hanging like it did
previously.
The soft lockup is in shrink_slab, so perhaps it's a combination of slab
shrinker and cgroup memory controller issues?
James
---
[ 670.823843] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46]
[ 670.825472] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel btusb snd_hda_codec snd_hwdep iwlagn snd_seq mac80211 bluetooth snd_seq_device uvcvideo snd_pcm cfg80211 wmi microcode e1000e videodev xhci_hcd rfkill snd_timer iTCO_wdt v4l2_compat_ioctl32 iTCO_vendor_support pcspkr i2c_i801 snd soundcore snd_page_alloc joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 670.830864] CPU 2
[ 670.830881] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel btusb snd_hda_codec snd_hwdep iwlagn snd_seq mac80211 bluetooth snd_seq_device uvcvideo snd_pcm cfg80211 wmi microcode e1000e videodev xhci_hcd rfkill snd_timer iTCO_wdt v4l2_compat_ioctl32 iTCO_vendor_support pcspkr i2c_i801 snd soundcore snd_page_alloc joydev uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 670.838385]
[ 670.840289] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #3 LENOVO 4170CTO/4170CTO
[ 670.842193] RIP: 0010:[<ffffffff810e07cb>] [<ffffffff810e07cb>] shrink_slab+0x86/0x166
[ 670.844063] RSP: 0018:ffff88006eea5da0 EFLAGS: 00000206
[ 670.845881] RAX: 0000000000000000 RBX: ffff88006eea5de0 RCX: 0000000000000002
[ 670.847652] RDX: 0000000000000000 RSI: ffff88006eea5d60 RDI: ffff88006eea5d60
[ 670.849394] RBP: ffff88006eea5de0 R08: 000000000000000c R09: 0000000000000000
[ 670.851091] R10: 0000000000000001 R11: 000000000000005f R12: ffffffff8147b50e
[ 670.852733] R13: ffff8801005e6e00 R14: 0000000000000010 R15: 0000000000017fb6
[ 670.854351] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 670.855968] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 670.857555] CR2: 00000037d90ae040 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 670.859138] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 670.860720] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 670.862320] Process kswapd0 (pid: 46, threadinfo ffff88006eea4000, task ffff88006eeb0000)
[ 670.863932] Stack:
[ 670.865477] 0000000000000001 0000000000000080 ffff880000000002 ffff8801005e6e00
[ 670.867023] ffff8801005e6000 0000000000000002 0000000000000000 000000000000000c
[ 670.868558] ffff88006eea5ee0 ffffffff810e308c 0000000000000003 ffff88006eeb0000
[ 670.870120] Call Trace:
[ 670.871652] [<ffffffff810e308c>] kswapd+0x4f0/0x774
[ 670.873218] [<ffffffff810e2b9c>] ? try_to_free_pages+0xe5/0xe5
[ 670.874786] [<ffffffff8106ce57>] kthread+0x84/0x8c
[ 670.876327] [<ffffffff8147bc64>] kernel_thread_helper+0x4/0x10
[ 670.877871] [<ffffffff8106cdd3>] ? kthread_worker_fn+0x148/0x148
[ 670.879403] [<ffffffff8147bc60>] ? gs_change+0x13/0x13
[ 670.880932] Code: 83 eb 10 e9 ce 00 00 00 44 89 f2 31 f6 48 89 df ff 13 48 63 4b 08 4c 63 e8 48 8b 45 c8 31 d2 48 f7 f1 31 d2 49 0f af c5 49 f7 f7
[ 670.881086] 03 43 20 48 85 c0 48 89 43 20 79 18 48 8b 33 48 89 c2 48 c7
[ 670.884285] Call Trace:
[ 670.885884] [<ffffffff810e308c>] kswapd+0x4f0/0x774
[ 670.887462] [<ffffffff810e2b9c>] ? try_to_free_pages+0xe5/0xe5
[ 670.889031] [<ffffffff8106ce57>] kthread+0x84/0x8c
[ 670.890578] [<ffffffff8147bc64>] kernel_thread_helper+0x4/0x10
[ 670.892130] [<ffffffff8106cdd3>] ? kthread_worker_fn+0x148/0x148
[ 670.893653] [<ffffffff8147bc60>] ? gs_change+0x13/0x13
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 22:40 ` Jan Kara
@ 2011-04-28 22:44 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 22:44 UTC (permalink / raw)
To: Jan Kara
Cc: Colin Ian King, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Fri, 2011-04-29 at 00:40 +0200, Jan Kara wrote:
> James in the meantime identified that cgroups are somehow involved.
> Are you
> using systemd by any chance? Maybe cgroup IO throttling screws us?
Yes, this is a FC15 system with systemd as the default.
See related post: it looks like disabling the memory controller is what
makes this go away.
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-28 22:44 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-28 22:44 UTC (permalink / raw)
To: Jan Kara
Cc: Colin Ian King, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Fri, 2011-04-29 at 00:40 +0200, Jan Kara wrote:
> James in the meantime identified that cgroups are somehow involved.
> Are you
> using systemd by any chance? Maybe cgroup IO throttling screws us?
Yes, this is a FC15 system with systemd as the default.
See related post: it looks like disabling the memory controller is what
makes this go away.
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-27 17:21 ` James Bottomley
@ 2011-04-29 10:23 ` Sedat Dilek
-1 siblings, 0 replies; 138+ messages in thread
From: Sedat Dilek @ 2011-04-29 10:23 UTC (permalink / raw)
To: James Bottomley; +Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel
Sorry for the top-posting!
But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try.
First, I have seen negative effects on my UP-system was when playing
with linux-next [2].
It was not clear what the origin was and the the side-effects were
somehow "bizarre".
The issue could be easily reproduced by tar-ing the kernel build-dir
to an external USB-hdd.
The issue kept RCU and TIP folks really busy.
Before stepping 4 weeks in the dark, give it a try and let me know in
case of success.
For the systemd/cgroups part of the discussion (yes, I followed this
thread in parallel):
The patch from [4] might be interesting (untested here).
Hope this helps you.
Have fun!
- Sedat -
[1] http://git.us.kernel.org/?p=linux/kernel/git/tip/linux-2.6-tip.git;a=commit;h=ce31332d3c77532d6ea97ddcb475a2b02dd358b4
[2] http://lkml.org/lkml/2011/3/25/97
[3] http://lkml.org/lkml/2011/4/28/444
[4] https://patchwork.kernel.org/patch/738921/
On Wed, Apr 27, 2011 at 7:21 PM, James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
> On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
>> Ok, I'd try turning it on so we catch the sleeping with a spinlock held
>> case better.
>
> Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
>
> This is the trace with sysrq-l and sysrq-w
>
> The repro this time doesn't have a soft lockup, just the tar is hung and
> one of my CPUs is in 99% system.
>
> James
>
> ---
>
>
> [ 453.351255] SysRq : Show backtrace of all active CPUs
> [ 453.352601] sending NMI to all CPUs:
> [ 453.353849] NMI backtrace for cpu 3
> [ 453.355545] CPU 3
> [ 453.355560] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 453.363188]
> [ 453.365162] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 453.367133] RIP: 0010:[<ffffffff8147af8c>] [<ffffffff8147af8c>] mutex_trylock+0x16/0x38
> [ 453.369122] RSP: 0018:ffff88006dfc1d40 EFLAGS: 00000246
> [ 453.371098] RAX: 0000000000000001 RBX: ffff880037de15f0 RCX: 0000000000000001
> [ 453.373099] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880037875820
> [ 453.375097] RBP: ffff88006dfc1d40 R08: 0000000000000000 R09: 00000000000074ad
> [ 453.377079] R10: 0000000000000002 R11: ffffffff81a44e50 R12: 0000000000000000
> [ 453.379052] R13: 0000000000000000 R14: ffff880037875800 R15: ffff880037875820
> [ 453.381015] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000
> [ 453.382985] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 453.384944] CR2: 00007fbf8ea8d090 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ 453.386920] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 453.388887] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 453.390789] Process kswapd0 (pid: 46, threadinfo ffff88006dfc0000, task ffff88006dfb8000)
> [ 453.392649] Stack:
> [ 453.394473] ffff88006dfc1d90 ffffffffa007ff52 ffff88006dfc1d90 ffffffff811613b5
> [ 453.396337] ffff88006dfc1d60 ffff880037de15f0 0000000000000000 0000000000000000
> [ 453.398174] 00000000000000d0 000000000006366c ffff88006dfc1de0 ffffffff810e1f89
> [ 453.400015] Call Trace:
> [ 453.401845] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915]
> [ 453.403702] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0
> [ 453.405499] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
> [ 453.407234] [<ffffffff810e4bcc>] kswapd+0x533/0x798
> [ 453.408952] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
> [ 453.410690] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 453.412432] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 453.414187] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 453.415945] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 453.417671] Code: 00 48 c7 47 18 00 00 00 00 f0 ff 07 7f 05 e8 ed 02 00 00 5d c3 55 48 89 e5 0f 1f 44 00 00 b9 01 00 00 00 31 d2 89 c8 f0 0f b1 17
> [ 453.417905] c1 31 c0 ff c9 75 18 65 48 8b 04 25 c8 cc 00 00 48 2d d8 1f
> [ 453.421534] Call Trace:
> [ 453.423337] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915]
> [ 453.425172] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0
> [ 453.426997] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
> [ 453.428818] [<ffffffff810e4bcc>] kswapd+0x533/0x798
> [ 453.430639] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
> [ 453.432485] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 453.434298] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 453.436112] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 453.437894] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 453.439654] NMI backtrace for cpu 2
> [ 453.441508] CPU 2
> [ 453.441525] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 453.449517]
> [ 453.451516] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 453.453704] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
> [ 453.455772] RSP: 0018:ffff8800715c9e68 EFLAGS: 00000046
> [ 453.457827] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001
> [ 453.459903] RDX: 0000000000000000 RSI: ffff8800715c9fd8 RDI: ffffffff81a0e640
> [ 453.461999] RBP: ffff8800715c9eb8 R08: 000000000000006d R09: 00000000000003e5
> [ 453.464039] R10: ffffffff00000002 R11: ffff880100293b40 R12: 0000000000000030
> [ 453.466013] R13: 12187898d4512537 R14: 0000000000000004 R15: 0000000000000002
> [ 453.467918] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
> [ 453.469797] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 453.471644] CR2: 0000000000452630 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ 453.473512] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 453.475356] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 453.477185] Process kworker/0:1 (pid: 0, threadinfo ffff8800715c8000, task ffff8800715a1700)
> [ 453.479024] Stack:
> [ 453.480855] ffff8800715c9e88 ffffffff810731c0 ffff880100291290 0000000000011290
> [ 453.482739] ffff8800715c9eb8 000000028139c97a ffffe8ffffc80170 ffffe8ffffc80170
> [ 453.484631] ffffe8ffffc80300 0000000000000000 ffff8800715c9ef8 ffffffff8139b868
> [ 453.486518] Call Trace:
> [ 453.488365] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 453.490226] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 453.492096] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 453.493947] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 453.495791] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
> [ 453.499958] Call Trace:
> [ 453.501895] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 453.503837] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 453.505775] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 453.507687] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 453.509598] NMI backtrace for cpu 1
> [ 453.511390] CPU 1
> [ 453.511405] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 453.519288]
> [ 453.521289] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 453.523353] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
> [ 453.525377] RSP: 0018:ffff880071587e68 EFLAGS: 00000046
> [ 453.527353] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
> [ 453.529332] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640
> [ 453.531303] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5
> [ 453.533276] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010
> [ 453.535249] R13: 12187898d4512ee3 R14: 0000000000000002 R15: 0000000000000001
> [ 453.537229] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000
> [ 453.539224] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 453.541210] CR2: 00000037d9071fa0 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ 453.543220] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 453.545233] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 453.547239] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700)
> [ 453.549274] Stack:
> [ 453.551289] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290
> [ 453.553301] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170
> [ 453.555257] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868
> [ 453.557156] Call Trace:
> [ 453.558980] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 453.560817] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 453.562635] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 453.564438] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 453.566227] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
> [ 453.570292] Call Trace:
> [ 453.572196] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 453.574117] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 453.576031] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 453.577924] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 453.579811] NMI backtrace for cpu 0
> [ 453.581279] CPU 0
> [ 453.581289] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 453.587576]
> [ 453.589160] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 453.590777] RIP: 0010:[<ffffffff8100f0fa>] [<ffffffff8100f0fa>] native_read_tsc+0x1/0x14
> [ 453.592390] RSP: 0018:ffff880100203b98 EFLAGS: 00000883
> [ 453.594001] RAX: 00000000f9ab6980 RBX: 0000000000002710 RCX: 0000000000000040
> [ 453.595624] RDX: 000000000026066c RSI: 0000000000000100 RDI: 000000000026066d
> [ 453.597250] RBP: ffff880100203ba8 R08: 000000008b000052 R09: 0000000000000000
> [ 453.598872] R10: 0000000000000000 R11: 0000000000000003 R12: 000000000026066d
> [ 453.600437] R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000001
> [ 453.601924] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000
> [ 453.603434] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 453.604942] CR2: 0000000000440360 CR3: 0000000001a03000 CR4: 00000000000406f0
> [ 453.606460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 453.607981] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 453.609597] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020)
> [ 453.611134] Stack:
> [ 453.612655] ffff880100203ba8 ffffffff81232dfe ffff880100203bd8 ffffffff81232ecb
> [ 453.614176] 0000000000002710 0000000000000008 000000000000006c 0000000000000002
> [ 453.615656] ffff880100203be8 ffffffff81232e49 ffff880100203bf8 ffffffff81232e77
> [ 453.617101] Call Trace:
> [ 453.618492] <IRQ>
> [ 453.619874] [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12
> [ 453.621251] [<ffffffff81232ecb>] delay_tsc+0x27/0x74
> [ 453.622613] [<ffffffff81232e49>] __delay+0xf/0x11
> [ 453.623973] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e
> [ 453.625322] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
> [ 453.626673] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
> [ 453.628027] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
> [ 453.629372] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
> [ 453.630706] [<ffffffff81365764>] input_pass_event+0x94/0xcc
> [ 453.632028] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
> [ 453.633342] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
> [ 453.634655] [<ffffffff81366cf2>] input_event+0x69/0x87
> [ 453.635978] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
> [ 453.637283] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
> [ 453.638563] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
> [ 453.639830] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
> [ 453.641092] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
> [ 453.642354] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
> [ 453.643598] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
> [ 453.644832] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
> [ 453.646069] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
> [ 453.647299] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
> [ 453.648527] [<ffffffff8147c253>] common_interrupt+0x13/0x13
> [ 453.649695] <EOI>
> [ 453.650794] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
> [ 453.651911] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
> [ 453.653025] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
> [ 453.654129] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 453.655231] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 453.656329] [<ffffffff8145a91e>] rest_init+0x72/0x74
> [ 453.657417] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
> [ 453.658514] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
> [ 453.659576] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
> [ 453.660589] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
> [ 453.661559] Code: 21 00 00 e8 74 3d 22 00 5d c3 90 90 90 55 40 88 f8 48 89 e5 e6 70 e4 71 5d c3 55 40 88 f0 48 89 e5 e6 70 40 88 f8 e6 71 5d c3 55
> [ 453.661721] 89 e5 0f 31 89 c1 48 89 d0 48 c1 e0 20 48 09 c8 5d c3 55 b2
> [ 453.663728] Call Trace:
> [ 453.664713] <IRQ> [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12
> [ 453.665710] [<ffffffff81232ecb>] delay_tsc+0x27/0x74
> [ 453.666698] [<ffffffff81232e49>] __delay+0xf/0x11
> [ 453.667686] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e
> [ 453.668675] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
> [ 453.669676] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
> [ 453.670681] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
> [ 453.671676] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
> [ 453.672662] [<ffffffff81365764>] input_pass_event+0x94/0xcc
> [ 453.673646] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
> [ 453.674627] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
> [ 453.675614] [<ffffffff81366cf2>] input_event+0x69/0x87
> [ 453.676587] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
> [ 453.677550] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
> [ 453.678498] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
> [ 453.679431] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
> [ 453.680351] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
> [ 453.681270] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
> [ 453.682186] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
> [ 453.683100] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
> [ 453.684014] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
> [ 453.684924] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
> [ 453.685831] [<ffffffff8147c253>] common_interrupt+0x13/0x13
> [ 453.686740] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
> [ 453.687655] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
> [ 453.688573] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
> [ 453.689480] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 453.690387] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 453.691293] [<ffffffff8145a91e>] rest_init+0x72/0x74
> [ 453.692195] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
> [ 453.693095] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
> [ 453.693996] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
> [ 453.694905] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
> [ 454.680802] SysRq : Show Blocked State
> [ 454.683427] task PC stack pid father
> [ 454.686058] systemd D 0000000000000000 0 1 0 0x00000000
> [ 454.688752] ffff8801003bdcd8 0000000000000082 ffff8801003bdc88 ffffffff00000000
> [ 454.691491] ffff880100370000 ffff8801003bdfd8 ffff8801003bdfd8 0000000000013b40
> [ 454.694228] ffff8800715e1700 ffff880100370000 ffff8801003bdcb8 00000001811329db
> [ 454.696969] Call Trace:
> [ 454.699683] [<ffffffff8147ac4b>] schedule_timeout+0x34/0xde
> [ 454.702447] [<ffffffff810ad618>] ? __call_rcu+0x123/0x12c
> [ 454.705184] [<ffffffff810ad64d>] ? call_rcu_sched+0x15/0x17
> [ 454.707906] [<ffffffff8147aa14>] wait_for_common+0xac/0x101
> [ 454.710656] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226
> [ 454.713445] [<ffffffff8147ab1d>] wait_for_completion+0x1d/0x1f
> [ 454.716199] [<ffffffff810ada5a>] synchronize_sched+0x5a/0x5c
> [ 454.718972] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43
> [ 454.721766] [<ffffffff81090d05>] cgroup_diput+0x37/0xe3
> [ 454.724562] [<ffffffff81090cce>] ? parse_cgroupfs_options+0x353/0x353
> [ 454.727328] [<ffffffff8112fc79>] dentry_kill+0xfa/0x121
> [ 454.730103] [<ffffffff81130189>] dput+0xdd/0xea
> [ 454.732866] [<ffffffff8112aa68>] do_rmdir+0xc6/0xfe
> [ 454.735481] [<ffffffff8111dc78>] ? filp_close+0x6e/0x7a
> [ 454.737990] [<ffffffff8112b32f>] sys_rmdir+0x16/0x18
> [ 454.740469] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000
> [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
> [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
> [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
> [ 454.753048] Call Trace:
> [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
> [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
> [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
> [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
> [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
> [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
> [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
> [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
> [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
> [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
> [ 454.791703] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
> [ 454.794334] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
> [ 454.796962] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
> [ 454.799582] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
> [ 454.802204] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
> [ 454.804803] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
> [ 454.807427] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d
> [ 454.810077] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde
> [ 454.812776] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
> [ 454.815464] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
> [ 454.818129] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 454.820808] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 454.823452] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 454.826103] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000
> [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
> [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
> [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
> [ 454.839453] Call Trace:
> [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12
> [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
> [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
> [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
> [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
> [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
> [ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
> [ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
> [ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
> [ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000
> [ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282
> [ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40
> [ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3
> [ 454.899530] Call Trace:
> [ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
> [ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> [ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
> [ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
> [ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
> [ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
> [ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
> [ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e
> [ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204
> [ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a
> [ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
> [ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
> [ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
> [ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff
> [ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
> [ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
> [ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3
> [ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e
> [ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 454.964732] dhclient-script D 0000000000000000 0 2856 2855 0x00000000
> [ 454.967360] ffff88006e1f5b18 0000000000000082 ffff8800378da880 0000000000000000
> [ 454.970056] ffff88006deb1700 ffff88006e1f5fd8 ffff88006e1f5fd8 0000000000013b40
> [ 454.972706] ffff880071589700 ffff88006deb1700 ffff88006e1f5ad8 000000016e1f5ad8
> [ 454.975323] Call Trace:
> [ 454.977882] [<ffffffff810d5916>] ? sleep_on_page+0x12/0x12
> [ 454.980477] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 454.983063] [<ffffffff810d5924>] sleep_on_page_killable+0xe/0x3b
> [ 454.985622] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
> [ 454.988182] [<ffffffff810d5819>] __lock_page_killable+0x66/0x68
> [ 454.990785] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.993436] [<ffffffff810d5859>] lock_page_killable+0x3e/0x43
> [ 454.996099] [<ffffffff810d71ea>] generic_file_aio_read+0x463/0x640
> [ 454.998730] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
> [ 455.001383] [<ffffffff811ebc34>] ? avc_has_perm+0x51/0x63
> [ 455.004012] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 455.006644] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 455.009289] [<ffffffff8111f511>] vfs_read+0xa9/0xf0
> [ 455.011917] [<ffffffff811233eb>] kernel_read+0x41/0x4f
> [ 455.014511] [<ffffffff811234dd>] prepare_binprm+0xe4/0xe8
> [ 455.017094] [<ffffffff81124d40>] do_execve+0x114/0x277
> [ 455.019678] [<ffffffff8100ff91>] sys_execve+0x43/0x5a
> [ 455.022281] [<ffffffff81482a9c>] stub_execve+0x6c/0xc0
> [ 455.024888] Sched Debug Version: v0.10, 2.6.39-rc4+ #1
> [ 455.027487] ktime : 455879.226285
> [ 455.030142] sched_clk : 455024.886257
> [ 455.032786] cpu_clk : 455024.886397
> [ 455.035352] jiffies : 4295123185
> [ 455.037904] sched_clock_stable : 1
> [ 455.040413]
> [ 455.042892] sysctl_sched
> [ 455.045306] .sysctl_sched_latency : 18.000000
> [ 455.047775] .sysctl_sched_min_granularity : 2.250000
> [ 455.050206] .sysctl_sched_wakeup_granularity : 3.000000
> [ 455.052643] .sysctl_sched_child_runs_first : 0
> [ 455.055034] .sysctl_sched_features : 7279
> [ 455.057423] .sysctl_sched_tunable_scaling : 1 (logaritmic)
> [ 455.059829]
> [ 455.059830] cpu#0, 2491.994 MHz
> [ 455.064443] .nr_running : 0
> [ 455.066757] .load : 0
> [ 455.069054] .nr_switches : 146510
> [ 455.071353] .nr_load_updates : 233084
> [ 455.073642] .nr_uninterruptible : 2
> [ 455.075894] .next_balance : 4295.122831
> [ 455.078152] .curr->pid : 0
> [ 455.080396] .clock : 454680.481348
> [ 455.082634] .cpu_load[0] : 0
> [ 455.084867] .cpu_load[1] : 0
> [ 455.087065] .cpu_load[2] : 0
> [ 455.089233] .cpu_load[3] : 0
> [ 455.091390] .cpu_load[4] : 0
> [ 455.093499] .yld_count : 0
> [ 455.095605] .sched_switch : 0
> [ 455.097667] .sched_count : 149062
> [ 455.099765] .sched_goidle : 62756
> [ 455.101781] .avg_idle : 1000000
> [ 455.103807] .ttwu_count : 77219
> [ 455.105958] .ttwu_local : 74144
> [ 455.107957] .bkl_count : 0
> [ 455.109914]
> [ 455.109915] cfs_rq[0]:/
> [ 455.113642] .exec_clock : 20017.048374
> [ 455.115515] .MIN_vruntime : 0.000001
> [ 455.117353] .min_vruntime : 28900.800090
> [ 455.119185] .max_vruntime : 0.000001
> [ 455.121028] .spread : 0.000000
> [ 455.122820] .spread0 : 0.000000
> [ 455.124581] .nr_spread_over : 54
> [ 455.126318] .nr_running : 0
> [ 455.128045] .load : 0
> [ 455.129743] .load_avg : 0.000000
> [ 455.131414] .load_period : 0.000000
> [ 455.133052] .load_contrib : 0
> [ 455.134692] .load_tg : 0
> [ 455.136305]
> [ 455.136306] runnable tasks:
> [ 455.136307] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 455.136310] ----------------------------------------------------------------------------------------------------------
> [ 455.142856]
> [ 455.142857] cpu#1, 2491.994 MHz
> [ 455.146215] .nr_running : 0
> [ 455.147931] .load : 0
> [ 455.149648] .nr_switches : 62175
> [ 455.151348] .nr_load_updates : 217324
> [ 455.153045] .nr_uninterruptible : 2
> [ 455.154747] .next_balance : 4295.123306
> [ 455.156457] .curr->pid : 0
> [ 455.158149] .clock : 455157.364642
> [ 455.159867] .cpu_load[0] : 0
> [ 455.161594] .cpu_load[1] : 0
> [ 455.163283] .cpu_load[2] : 0
> [ 455.164954] .cpu_load[3] : 0
> [ 455.166575] .cpu_load[4] : 0
> [ 455.168185] .yld_count : 60
> [ 455.169791] .sched_switch : 0
> [ 455.171401] .sched_count : 62899
> [ 455.172984] .sched_goidle : 27394
> [ 455.174580] .avg_idle : 1000000
> [ 455.176171] .ttwu_count : 30510
> [ 455.177739] .ttwu_local : 25277
> [ 455.179292] .bkl_count : 0
> [ 455.180882]
> [ 455.180883] cfs_rq[1]:/
> [ 455.183954] .exec_clock : 10655.021809
> [ 455.185550] .MIN_vruntime : 0.000001
> [ 455.187141] .min_vruntime : 19718.135550
> [ 455.188771] .max_vruntime : 0.000001
> [ 455.190407] .spread : 0.000000
> [ 455.192016] .spread0 : -9182.664540
> [ 455.193634] .nr_spread_over : 80
> [ 455.195242] .nr_running : 0
> [ 455.196848] .load : 0
> [ 455.198441] .load_avg : 0.000000
> [ 455.200059] .load_period : 0.000000
> [ 455.201654] .load_contrib : 0
> [ 455.203240] .load_tg : 0
> [ 455.204836]
> [ 455.204837] runnable tasks:
> [ 455.204838] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 455.204841] ----------------------------------------------------------------------------------------------------------
> [ 455.211403]
> [ 455.211404] cpu#2, 2491.994 MHz
> [ 455.214958] .nr_running : 0
> [ 455.216771] .load : 0
> [ 455.218564] .nr_switches : 37444
> [ 455.220394] .nr_load_updates : 219042
> [ 455.222202] .nr_uninterruptible : 0
> [ 455.224039] .next_balance : 4295.123375
> [ 455.225897] .curr->pid : 0
> [ 455.227769] .clock : 455227.204963
> [ 455.229681] .cpu_load[0] : 0
> [ 455.231548] .cpu_load[1] : 0
> [ 455.233353] .cpu_load[2] : 0
> [ 455.235059] .cpu_load[3] : 0
> [ 455.236676] .cpu_load[4] : 0
> [ 455.238280] .yld_count : 0
> [ 455.239879] .sched_switch : 0
> [ 455.241471] .sched_count : 37815
> [ 455.243075] .sched_goidle : 16831
> [ 455.244674] .avg_idle : 1000000
> [ 455.246270] .ttwu_count : 18348
> [ 455.247849] .ttwu_local : 16899
> [ 455.249430] .bkl_count : 0
> [ 455.250992]
> [ 455.250993] cfs_rq[2]:/
> [ 455.254040] .exec_clock : 6758.351942
> [ 455.255630] .MIN_vruntime : 0.000001
> [ 455.257236] .min_vruntime : 13719.942866
> [ 455.258861] .max_vruntime : 0.000001
> [ 455.260497] .spread : 0.000000
> [ 455.262122] .spread0 : -15180.857224
> [ 455.263753] .nr_spread_over : 21
> [ 455.265389] .nr_running : 0
> [ 455.267018] .load : 0
> [ 455.268637] .load_avg : 0.000000
> [ 455.270292] .load_period : 0.000000
> [ 455.271911] .load_contrib : 0
> [ 455.273530] .load_tg : 0
> [ 455.275163]
> [ 455.275165] runnable tasks:
> [ 455.275166] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 455.275169] ----------------------------------------------------------------------------------------------------------
> [ 455.281810]
> [ 455.281811] cpu#3, 2491.994 MHz
> [ 455.285349] .nr_running : 4
> [ 455.287149] .load : 2048
> [ 455.288952] .nr_switches : 30358
> [ 455.290787] .nr_load_updates : 223302
> [ 455.292599] .nr_uninterruptible : 1
> [ 455.294419] .next_balance : 4295.123506
> [ 455.296250] .curr->pid : 46
> [ 455.298090] .clock : 431792.394314
> [ 455.299953] .cpu_load[0] : 2048
> [ 455.301810] .cpu_load[1] : 2048
> [ 455.303610] .cpu_load[2] : 2048
> [ 455.305311] .cpu_load[3] : 2048
> [ 455.306926] .cpu_load[4] : 2048
> [ 455.308531] .yld_count : 63
> [ 455.310130] .sched_switch : 0
> [ 455.311723] .sched_count : 30824
> [ 455.313315] .sched_goidle : 13491
> [ 455.314904] .avg_idle : 1000000
> [ 455.316481] .ttwu_count : 13858
> [ 455.318057] .ttwu_local : 12507
> [ 455.319629] .bkl_count : 0
> [ 455.321187]
> [ 455.321188] cfs_rq[3]:/
> [ 455.324213] .exec_clock : 5334.144946
> [ 455.325795] .MIN_vruntime : 13295.262803
> [ 455.327401] .min_vruntime : 13302.523317
> [ 455.329046] .max_vruntime : 13295.262803
> [ 455.330656] .spread : 0.000000
> [ 455.332297] .spread0 : -15598.276773
> [ 455.333925] .nr_spread_over : 117
> [ 455.335577] .nr_running : 2
> [ 455.337204] .load : 2048
> [ 455.338823] .load_avg : 0.000000
> [ 455.340476] .load_period : 0.000000
> [ 455.342094] .load_contrib : 0
> [ 455.343702] .load_tg : 0
> [ 455.345302]
> [ 455.345303] rt_rq[3]:/
> [ 455.348416] .rt_nr_running : 1
> [ 455.350006] .rt_throttled : 0
> [ 455.351587] .rt_time : 0.000000
> [ 455.353173] .rt_runtime : 950.000000
> [ 455.354759]
> [ 455.354760] runnable tasks:
> [ 455.354761] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 455.354764] ----------------------------------------------------------------------------------------------------------
> [ 455.361357] migration/3 17 0.000000 1084 0 0.000000 0.000933 0.000000 /
> [ 455.363278] watchdog/3 20 0.000000 31 0 0.000000 0.780405 0.002351 /
> [ 455.365212] R kswapd0 46 13302.523317 714 120 13302.523317 1148.767369 245855.504389 /
> [ 455.367220] kworker/3:1 74 13295.262803 10488 120 13295.262803 324.924994 315669.659686 /
> [ 455.369317]
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-29 10:23 ` Sedat Dilek
0 siblings, 0 replies; 138+ messages in thread
From: Sedat Dilek @ 2011-04-29 10:23 UTC (permalink / raw)
To: James Bottomley; +Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel
Sorry for the top-posting!
But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try.
First, I have seen negative effects on my UP-system was when playing
with linux-next [2].
It was not clear what the origin was and the the side-effects were
somehow "bizarre".
The issue could be easily reproduced by tar-ing the kernel build-dir
to an external USB-hdd.
The issue kept RCU and TIP folks really busy.
Before stepping 4 weeks in the dark, give it a try and let me know in
case of success.
For the systemd/cgroups part of the discussion (yes, I followed this
thread in parallel):
The patch from [4] might be interesting (untested here).
Hope this helps you.
Have fun!
- Sedat -
[1] http://git.us.kernel.org/?p=linux/kernel/git/tip/linux-2.6-tip.git;a=commit;h=ce31332d3c77532d6ea97ddcb475a2b02dd358b4
[2] http://lkml.org/lkml/2011/3/25/97
[3] http://lkml.org/lkml/2011/4/28/444
[4] https://patchwork.kernel.org/patch/738921/
On Wed, Apr 27, 2011 at 7:21 PM, James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
> On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote:
>> Ok, I'd try turning it on so we catch the sleeping with a spinlock held
>> case better.
>
> Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)?
>
> This is the trace with sysrq-l and sysrq-w
>
> The repro this time doesn't have a soft lockup, just the tar is hung and
> one of my CPUs is in 99% system.
>
> James
>
> ---
>
>
> [ 453.351255] SysRq : Show backtrace of all active CPUs
> [ 453.352601] sending NMI to all CPUs:
> [ 453.353849] NMI backtrace for cpu 3
> [ 453.355545] CPU 3
> [ 453.355560] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 453.363188]
> [ 453.365162] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 453.367133] RIP: 0010:[<ffffffff8147af8c>] [<ffffffff8147af8c>] mutex_trylock+0x16/0x38
> [ 453.369122] RSP: 0018:ffff88006dfc1d40 EFLAGS: 00000246
> [ 453.371098] RAX: 0000000000000001 RBX: ffff880037de15f0 RCX: 0000000000000001
> [ 453.373099] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880037875820
> [ 453.375097] RBP: ffff88006dfc1d40 R08: 0000000000000000 R09: 00000000000074ad
> [ 453.377079] R10: 0000000000000002 R11: ffffffff81a44e50 R12: 0000000000000000
> [ 453.379052] R13: 0000000000000000 R14: ffff880037875800 R15: ffff880037875820
> [ 453.381015] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000
> [ 453.382985] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 453.384944] CR2: 00007fbf8ea8d090 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ 453.386920] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 453.388887] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 453.390789] Process kswapd0 (pid: 46, threadinfo ffff88006dfc0000, task ffff88006dfb8000)
> [ 453.392649] Stack:
> [ 453.394473] ffff88006dfc1d90 ffffffffa007ff52 ffff88006dfc1d90 ffffffff811613b5
> [ 453.396337] ffff88006dfc1d60 ffff880037de15f0 0000000000000000 0000000000000000
> [ 453.398174] 00000000000000d0 000000000006366c ffff88006dfc1de0 ffffffff810e1f89
> [ 453.400015] Call Trace:
> [ 453.401845] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915]
> [ 453.403702] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0
> [ 453.405499] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
> [ 453.407234] [<ffffffff810e4bcc>] kswapd+0x533/0x798
> [ 453.408952] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
> [ 453.410690] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 453.412432] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 453.414187] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 453.415945] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 453.417671] Code: 00 48 c7 47 18 00 00 00 00 f0 ff 07 7f 05 e8 ed 02 00 00 5d c3 55 48 89 e5 0f 1f 44 00 00 b9 01 00 00 00 31 d2 89 c8 f0 0f b1 17
> [ 453.417905] c1 31 c0 ff c9 75 18 65 48 8b 04 25 c8 cc 00 00 48 2d d8 1f
> [ 453.421534] Call Trace:
> [ 453.423337] [<ffffffffa007ff52>] i915_gem_inactive_shrink+0x2f/0x194 [i915]
> [ 453.425172] [<ffffffff811613b5>] ? mb_cache_shrink_fn+0x32/0xd0
> [ 453.426997] [<ffffffff810e1f89>] shrink_slab+0x6d/0x166
> [ 453.428818] [<ffffffff810e4bcc>] kswapd+0x533/0x798
> [ 453.430639] [<ffffffff810e4699>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
> [ 453.432485] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 453.434298] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 453.436112] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 453.437894] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 453.439654] NMI backtrace for cpu 2
> [ 453.441508] CPU 2
> [ 453.441525] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 453.449517]
> [ 453.451516] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 453.453704] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
> [ 453.455772] RSP: 0018:ffff8800715c9e68 EFLAGS: 00000046
> [ 453.457827] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001
> [ 453.459903] RDX: 0000000000000000 RSI: ffff8800715c9fd8 RDI: ffffffff81a0e640
> [ 453.461999] RBP: ffff8800715c9eb8 R08: 000000000000006d R09: 00000000000003e5
> [ 453.464039] R10: ffffffff00000002 R11: ffff880100293b40 R12: 0000000000000030
> [ 453.466013] R13: 12187898d4512537 R14: 0000000000000004 R15: 0000000000000002
> [ 453.467918] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
> [ 453.469797] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 453.471644] CR2: 0000000000452630 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ 453.473512] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 453.475356] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 453.477185] Process kworker/0:1 (pid: 0, threadinfo ffff8800715c8000, task ffff8800715a1700)
> [ 453.479024] Stack:
> [ 453.480855] ffff8800715c9e88 ffffffff810731c0 ffff880100291290 0000000000011290
> [ 453.482739] ffff8800715c9eb8 000000028139c97a ffffe8ffffc80170 ffffe8ffffc80170
> [ 453.484631] ffffe8ffffc80300 0000000000000000 ffff8800715c9ef8 ffffffff8139b868
> [ 453.486518] Call Trace:
> [ 453.488365] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 453.490226] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 453.492096] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 453.493947] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 453.495791] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
> [ 453.499958] Call Trace:
> [ 453.501895] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 453.503837] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 453.505775] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 453.507687] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 453.509598] NMI backtrace for cpu 1
> [ 453.511390] CPU 1
> [ 453.511405] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 453.519288]
> [ 453.521289] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 453.523353] RIP: 0010:[<ffffffff81275d36>] [<ffffffff81275d36>] intel_idle+0xaa/0x100
> [ 453.525377] RSP: 0018:ffff880071587e68 EFLAGS: 00000046
> [ 453.527353] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
> [ 453.529332] RDX: 0000000000000000 RSI: ffff880071587fd8 RDI: ffffffff81a0e640
> [ 453.531303] RBP: ffff880071587eb8 R08: 00000000000004af R09: 00000000000003e5
> [ 453.533276] R10: ffffffff00000001 R11: ffff880100253b40 R12: 0000000000000010
> [ 453.535249] R13: 12187898d4512ee3 R14: 0000000000000002 R15: 0000000000000001
> [ 453.537229] FS: 0000000000000000(0000) GS:ffff880100240000(0000) knlGS:0000000000000000
> [ 453.539224] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 453.541210] CR2: 00000037d9071fa0 CR3: 0000000001a03000 CR4: 00000000000406e0
> [ 453.543220] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 453.545233] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 453.547239] Process kworker/0:0 (pid: 0, threadinfo ffff880071586000, task ffff880071589700)
> [ 453.549274] Stack:
> [ 453.551289] ffff880071587e88 ffffffff810731c0 ffff880100251290 0000000000011290
> [ 453.553301] ffff880071587eb8 000000018139c97a ffffe8ffffc40170 ffffe8ffffc40170
> [ 453.555257] ffffe8ffffc40240 0000000000000000 ffff880071587ef8 ffffffff8139b868
> [ 453.557156] Call Trace:
> [ 453.558980] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 453.560817] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 453.562635] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 453.564438] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 453.566227] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 09 e0 ff 4c 29 e8 48 89 c7 e8 ab 29 de ff 4c 69 e0 40 42
> [ 453.570292] Call Trace:
> [ 453.572196] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
> [ 453.574117] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 453.576031] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 453.577924] [<ffffffff8146ae57>] start_secondary+0x223/0x225
> [ 453.579811] NMI backtrace for cpu 0
> [ 453.581279] CPU 0
> [ 453.581289] Modules linked in: netconsole configfs cpufreq_ondemand acpi_cpufreq freq_table mperf snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec iwlagn snd_hwdep snd_seq mac80211 snd_seq_device uvcvideo btusb cfg80211 bluetooth wmi e1000e snd_pcm videodev rfkill i2c_i801 microcode iTCO_wdt iTCO_vendor_support xhci_hcd snd_timer v4l2_compat_ioctl32 joydev pcspkr snd soundcore snd_page_alloc uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
> [ 453.587576]
> [ 453.589160] Pid: 0, comm: swapper Not tainted 2.6.39-rc4+ #1 LENOVO 4170CTO/4170CTO
> [ 453.590777] RIP: 0010:[<ffffffff8100f0fa>] [<ffffffff8100f0fa>] native_read_tsc+0x1/0x14
> [ 453.592390] RSP: 0018:ffff880100203b98 EFLAGS: 00000883
> [ 453.594001] RAX: 00000000f9ab6980 RBX: 0000000000002710 RCX: 0000000000000040
> [ 453.595624] RDX: 000000000026066c RSI: 0000000000000100 RDI: 000000000026066d
> [ 453.597250] RBP: ffff880100203ba8 R08: 000000008b000052 R09: 0000000000000000
> [ 453.598872] R10: 0000000000000000 R11: 0000000000000003 R12: 000000000026066d
> [ 453.600437] R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000001
> [ 453.601924] FS: 0000000000000000(0000) GS:ffff880100200000(0000) knlGS:0000000000000000
> [ 453.603434] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 453.604942] CR2: 0000000000440360 CR3: 0000000001a03000 CR4: 00000000000406f0
> [ 453.606460] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 453.607981] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 453.609597] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a0b020)
> [ 453.611134] Stack:
> [ 453.612655] ffff880100203ba8 ffffffff81232dfe ffff880100203bd8 ffffffff81232ecb
> [ 453.614176] 0000000000002710 0000000000000008 000000000000006c 0000000000000002
> [ 453.615656] ffff880100203be8 ffffffff81232e49 ffff880100203bf8 ffffffff81232e77
> [ 453.617101] Call Trace:
> [ 453.618492] <IRQ>
> [ 453.619874] [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12
> [ 453.621251] [<ffffffff81232ecb>] delay_tsc+0x27/0x74
> [ 453.622613] [<ffffffff81232e49>] __delay+0xf/0x11
> [ 453.623973] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e
> [ 453.625322] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
> [ 453.626673] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
> [ 453.628027] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
> [ 453.629372] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
> [ 453.630706] [<ffffffff81365764>] input_pass_event+0x94/0xcc
> [ 453.632028] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
> [ 453.633342] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
> [ 453.634655] [<ffffffff81366cf2>] input_event+0x69/0x87
> [ 453.635978] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
> [ 453.637283] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
> [ 453.638563] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
> [ 453.639830] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
> [ 453.641092] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
> [ 453.642354] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
> [ 453.643598] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
> [ 453.644832] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
> [ 453.646069] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
> [ 453.647299] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
> [ 453.648527] [<ffffffff8147c253>] common_interrupt+0x13/0x13
> [ 453.649695] <EOI>
> [ 453.650794] [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
> [ 453.651911] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
> [ 453.653025] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
> [ 453.654129] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 453.655231] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 453.656329] [<ffffffff8145a91e>] rest_init+0x72/0x74
> [ 453.657417] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
> [ 453.658514] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
> [ 453.659576] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
> [ 453.660589] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
> [ 453.661559] Code: 21 00 00 e8 74 3d 22 00 5d c3 90 90 90 55 40 88 f8 48 89 e5 e6 70 e4 71 5d c3 55 40 88 f0 48 89 e5 e6 70 40 88 f8 e6 71 5d c3 55
> [ 453.661721] 89 e5 0f 31 89 c1 48 89 d0 48 c1 e0 20 48 09 c8 5d c3 55 b2
> [ 453.663728] Call Trace:
> [ 453.664713] <IRQ> [<ffffffff81232dfe>] ? paravirt_read_tsc+0xe/0x12
> [ 453.665710] [<ffffffff81232ecb>] delay_tsc+0x27/0x74
> [ 453.666698] [<ffffffff81232e49>] __delay+0xf/0x11
> [ 453.667686] [<ffffffff81232e77>] __const_udelay+0x2c/0x2e
> [ 453.668675] [<ffffffff8102166e>] arch_trigger_all_cpu_backtrace+0x76/0x88
> [ 453.669676] [<ffffffff812be0ad>] sysrq_handle_showallcpus+0xe/0x10
> [ 453.670681] [<ffffffff812be310>] __handle_sysrq+0xa2/0x13c
> [ 453.671676] [<ffffffff812be514>] sysrq_filter+0x112/0x16e
> [ 453.672662] [<ffffffff81365764>] input_pass_event+0x94/0xcc
> [ 453.673646] [<ffffffff81366bf1>] input_handle_event+0x480/0x48f
> [ 453.674627] [<ffffffff810483af>] ? walk_tg_tree.constprop.71+0x28/0x94
> [ 453.675614] [<ffffffff81366cf2>] input_event+0x69/0x87
> [ 453.676587] [<ffffffff8136c17b>] atkbd_interrupt+0x4c1/0x58e
> [ 453.677550] [<ffffffff81361b2e>] serio_interrupt+0x45/0x7f
> [ 453.678498] [<ffffffff81362870>] i8042_interrupt+0x299/0x2ab
> [ 453.679431] [<ffffffff8100eb79>] ? native_sched_clock+0x34/0x36
> [ 453.680351] [<ffffffff810a95d5>] handle_irq_event_percpu+0x5f/0x198
> [ 453.681270] [<ffffffff810a9746>] handle_irq_event+0x38/0x56
> [ 453.682186] [<ffffffff81022e0e>] ? ack_apic_edge+0x25/0x29
> [ 453.683100] [<ffffffff810ab71a>] handle_edge_irq+0x9d/0xc0
> [ 453.684014] [<ffffffff8100ab9d>] handle_irq+0x88/0x8e
> [ 453.684924] [<ffffffff8148409d>] do_IRQ+0x4d/0xa5
> [ 453.685831] [<ffffffff8147c253>] common_interrupt+0x13/0x13
> [ 453.686740] <EOI> [<ffffffff8100e6cd>] ? paravirt_read_tsc+0x9/0xd
> [ 453.687655] [<ffffffff81275d67>] ? intel_idle+0xdb/0x100
> [ 453.688573] [<ffffffff81275d46>] ? intel_idle+0xba/0x100
> [ 453.689480] [<ffffffff8139b868>] cpuidle_idle_call+0xe7/0x166
> [ 453.690387] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
> [ 453.691293] [<ffffffff8145a91e>] rest_init+0x72/0x74
> [ 453.692195] [<ffffffff81b59b9f>] start_kernel+0x3de/0x3e9
> [ 453.693095] [<ffffffff81b592c4>] x86_64_start_reservations+0xaf/0xb3
> [ 453.693996] [<ffffffff81b59140>] ? early_idt_handlers+0x140/0x140
> [ 453.694905] [<ffffffff81b593ca>] x86_64_start_kernel+0x102/0x111
> [ 454.680802] SysRq : Show Blocked State
> [ 454.683427] task PC stack pid father
> [ 454.686058] systemd D 0000000000000000 0 1 0 0x00000000
> [ 454.688752] ffff8801003bdcd8 0000000000000082 ffff8801003bdc88 ffffffff00000000
> [ 454.691491] ffff880100370000 ffff8801003bdfd8 ffff8801003bdfd8 0000000000013b40
> [ 454.694228] ffff8800715e1700 ffff880100370000 ffff8801003bdcb8 00000001811329db
> [ 454.696969] Call Trace:
> [ 454.699683] [<ffffffff8147ac4b>] schedule_timeout+0x34/0xde
> [ 454.702447] [<ffffffff810ad618>] ? __call_rcu+0x123/0x12c
> [ 454.705184] [<ffffffff810ad64d>] ? call_rcu_sched+0x15/0x17
> [ 454.707906] [<ffffffff8147aa14>] wait_for_common+0xac/0x101
> [ 454.710656] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226
> [ 454.713445] [<ffffffff8147ab1d>] wait_for_completion+0x1d/0x1f
> [ 454.716199] [<ffffffff810ada5a>] synchronize_sched+0x5a/0x5c
> [ 454.718972] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43
> [ 454.721766] [<ffffffff81090d05>] cgroup_diput+0x37/0xe3
> [ 454.724562] [<ffffffff81090cce>] ? parse_cgroupfs_options+0x353/0x353
> [ 454.727328] [<ffffffff8112fc79>] dentry_kill+0xfa/0x121
> [ 454.730103] [<ffffffff81130189>] dput+0xdd/0xea
> [ 454.732866] [<ffffffff8112aa68>] do_rmdir+0xc6/0xfe
> [ 454.735481] [<ffffffff8111dc78>] ? filp_close+0x6e/0x7a
> [ 454.737990] [<ffffffff8112b32f>] sys_rmdir+0x16/0x18
> [ 454.740469] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x00000000
> [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffffffff00000000
> [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000000000013b40
> [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 000000018106e7c3
> [ 454.753048] Call Trace:
> [ 454.755537] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> [ 454.758071] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.760644] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 454.763206] [<ffffffff8118ea4f>] ? ext4_read_block_bitmap+0x54/0x2d0
> [ 454.765770] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 454.768353] [<ffffffff811b8dbe>] ext4_mb_mark_diskspace_used+0x70/0x2ae
> [ 454.770942] [<ffffffff811bb10e>] ext4_mb_new_blocks+0x1c8/0x3c2
> [ 454.773501] [<ffffffff811b4628>] ext4_ext_map_blocks+0x1961/0x1c04
> [ 454.776082] [<ffffffff8122ed78>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
> [ 454.778711] [<ffffffff810d55f9>] ? find_get_pages_tag+0x3b/0xd6
> [ 454.781323] [<ffffffff811967fa>] ext4_map_blocks+0x112/0x1e7
> [ 454.783894] [<ffffffff811984e8>] mpage_da_map_and_submit+0x93/0x2cd
> [ 454.786491] [<ffffffff81198de5>] ext4_da_writepages+0x2c1/0x44d
> [ 454.789090] [<ffffffff810ddeb4>] do_writepages+0x21/0x2a
> [ 454.791703] [<ffffffff8113cbb7>] writeback_single_inode+0xb2/0x1bc
> [ 454.794334] [<ffffffff8113cf03>] writeback_sb_inodes+0xcd/0x161
> [ 454.796962] [<ffffffff8113d407>] writeback_inodes_wb+0x119/0x12b
> [ 454.799582] [<ffffffff8113d607>] wb_writeback+0x1ee/0x335
> [ 454.802204] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
> [ 454.804803] [<ffffffff8147be3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
> [ 454.807427] [<ffffffff8113d891>] wb_do_writeback+0x143/0x19d
> [ 454.810077] [<ffffffff8147acc7>] ? schedule_timeout+0xb0/0xde
> [ 454.812776] [<ffffffff8113d973>] bdi_writeback_thread+0x88/0x1e5
> [ 454.815464] [<ffffffff8113d8eb>] ? wb_do_writeback+0x19d/0x19d
> [ 454.818129] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 454.820808] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 454.823452] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 454.826103] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x00000000
> [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffffffff00000000
> [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000000000013b40
> [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 000000016d59dad0
> [ 454.839453] Call Trace:
> [ 454.842098] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> [ 454.844738] [<ffffffff810d5904>] ? lock_page+0x3e/0x3e
> [ 454.847303] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 454.849877] [<ffffffff810d5912>] sleep_on_page+0xe/0x12
> [ 454.852469] [<ffffffff8147aea9>] __wait_on_bit+0x48/0x7b
> [ 454.855021] [<ffffffff810d5a8c>] wait_on_page_bit+0x72/0x74
> [ 454.857583] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.860171] [<ffffffff810d5b6b>] filemap_fdatawait_range+0x84/0x163
> [ 454.862744] [<ffffffff810d5c6e>] filemap_fdatawait+0x24/0x26
> [ 454.865299] [<ffffffff811c94a2>] jbd2_journal_commit_transaction+0x922/0x1194
> [ 454.867892] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
> [ 454.870496] [<ffffffff811cd3b6>] kjournald2+0xc9/0x20a
> [ 454.873103] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
> [ 454.875690] [<ffffffff811cd2ed>] ? commit_timeout+0x10/0x10
> [ 454.878327] [<ffffffff8106e157>] kthread+0x84/0x8c
> [ 454.880961] [<ffffffff81483764>] kernel_thread_helper+0x4/0x10
> [ 454.883604] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
> [ 454.886262] [<ffffffff81483760>] ? gs_change+0x13/0x13
> [ 454.888875] tar D ffff88006e573af8 0 991 838 0x00000000
> [ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000000000000282
> [ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000000000013b40
> [ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffffffff8106e7c3
> [ 454.899530] Call Trace:
> [ 454.902118] [<ffffffff8106e7c3>] ? prepare_to_wait+0x6c/0x78
> [ 454.904724] [<ffffffff811c82b8>] do_get_write_access+0x1c6/0x38d
> [ 454.907344] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.909967] [<ffffffff811991cc>] ? ext4_dirty_inode+0x33/0x4c
> [ 454.912574] [<ffffffff811c8588>] jbd2_journal_get_write_access+0x2b/0x42
> [ 454.915192] [<ffffffff811b5888>] __ext4_journal_get_write_access+0x58/0x66
> [ 454.917819] [<ffffffff81195526>] ext4_reserve_inode_write+0x41/0x83
> [ 454.920459] [<ffffffff811955e4>] ext4_mark_inode_dirty+0x7c/0x1f0
> [ 454.923070] [<ffffffff811991cc>] ext4_dirty_inode+0x33/0x4c
> [ 454.925660] [<ffffffff8113c3d6>] __mark_inode_dirty+0x2f/0x175
> [ 454.928247] [<ffffffff81143a0d>] generic_write_end+0x6c/0x7e
> [ 454.930865] [<ffffffff811983f6>] ext4_da_write_end+0x1a5/0x204
> [ 454.933454] [<ffffffff810d5e9d>] generic_file_buffered_write+0x17e/0x23a
> [ 454.936062] [<ffffffff810d6c9d>] __generic_file_aio_write+0x242/0x272
> [ 454.938648] [<ffffffff810d6d2e>] generic_file_aio_write+0x61/0xba
> [ 454.941288] [<ffffffff8118fe00>] ext4_file_write+0x1dc/0x234
> [ 454.943909] [<ffffffff8111edab>] do_sync_write+0xbf/0xff
> [ 454.946501] [<ffffffff8114b9fc>] ? fsnotify+0x1eb/0x217
> [ 454.949114] [<ffffffff811f1866>] ? selinux_file_permission+0x58/0xb4
> [ 454.951736] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 454.954349] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 454.956943] [<ffffffff8111f421>] vfs_write+0xac/0xf3
> [ 454.959530] [<ffffffff8111f610>] sys_write+0x4a/0x6e
> [ 454.962129] [<ffffffff81482642>] system_call_fastpath+0x16/0x1b
> [ 454.964732] dhclient-script D 0000000000000000 0 2856 2855 0x00000000
> [ 454.967360] ffff88006e1f5b18 0000000000000082 ffff8800378da880 0000000000000000
> [ 454.970056] ffff88006deb1700 ffff88006e1f5fd8 ffff88006e1f5fd8 0000000000013b40
> [ 454.972706] ffff880071589700 ffff88006deb1700 ffff88006e1f5ad8 000000016e1f5ad8
> [ 454.975323] Call Trace:
> [ 454.977882] [<ffffffff810d5916>] ? sleep_on_page+0x12/0x12
> [ 454.980477] [<ffffffff8147a7c9>] io_schedule+0x63/0x7e
> [ 454.983063] [<ffffffff810d5924>] sleep_on_page_killable+0xe/0x3b
> [ 454.985622] [<ffffffff8147ad9b>] __wait_on_bit_lock+0x46/0x8f
> [ 454.988182] [<ffffffff810d5819>] __lock_page_killable+0x66/0x68
> [ 454.990785] [<ffffffff8106e88b>] ? autoremove_wake_function+0x3d/0x3d
> [ 454.993436] [<ffffffff810d5859>] lock_page_killable+0x3e/0x43
> [ 454.996099] [<ffffffff810d71ea>] generic_file_aio_read+0x463/0x640
> [ 454.998730] [<ffffffff8111eeaa>] do_sync_read+0xbf/0xff
> [ 455.001383] [<ffffffff811ebc34>] ? avc_has_perm+0x51/0x63
> [ 455.004012] [<ffffffff811e9cfe>] ? security_file_permission+0x2e/0x33
> [ 455.006644] [<ffffffff8111f196>] ? rw_verify_area+0xb0/0xcd
> [ 455.009289] [<ffffffff8111f511>] vfs_read+0xa9/0xf0
> [ 455.011917] [<ffffffff811233eb>] kernel_read+0x41/0x4f
> [ 455.014511] [<ffffffff811234dd>] prepare_binprm+0xe4/0xe8
> [ 455.017094] [<ffffffff81124d40>] do_execve+0x114/0x277
> [ 455.019678] [<ffffffff8100ff91>] sys_execve+0x43/0x5a
> [ 455.022281] [<ffffffff81482a9c>] stub_execve+0x6c/0xc0
> [ 455.024888] Sched Debug Version: v0.10, 2.6.39-rc4+ #1
> [ 455.027487] ktime : 455879.226285
> [ 455.030142] sched_clk : 455024.886257
> [ 455.032786] cpu_clk : 455024.886397
> [ 455.035352] jiffies : 4295123185
> [ 455.037904] sched_clock_stable : 1
> [ 455.040413]
> [ 455.042892] sysctl_sched
> [ 455.045306] .sysctl_sched_latency : 18.000000
> [ 455.047775] .sysctl_sched_min_granularity : 2.250000
> [ 455.050206] .sysctl_sched_wakeup_granularity : 3.000000
> [ 455.052643] .sysctl_sched_child_runs_first : 0
> [ 455.055034] .sysctl_sched_features : 7279
> [ 455.057423] .sysctl_sched_tunable_scaling : 1 (logaritmic)
> [ 455.059829]
> [ 455.059830] cpu#0, 2491.994 MHz
> [ 455.064443] .nr_running : 0
> [ 455.066757] .load : 0
> [ 455.069054] .nr_switches : 146510
> [ 455.071353] .nr_load_updates : 233084
> [ 455.073642] .nr_uninterruptible : 2
> [ 455.075894] .next_balance : 4295.122831
> [ 455.078152] .curr->pid : 0
> [ 455.080396] .clock : 454680.481348
> [ 455.082634] .cpu_load[0] : 0
> [ 455.084867] .cpu_load[1] : 0
> [ 455.087065] .cpu_load[2] : 0
> [ 455.089233] .cpu_load[3] : 0
> [ 455.091390] .cpu_load[4] : 0
> [ 455.093499] .yld_count : 0
> [ 455.095605] .sched_switch : 0
> [ 455.097667] .sched_count : 149062
> [ 455.099765] .sched_goidle : 62756
> [ 455.101781] .avg_idle : 1000000
> [ 455.103807] .ttwu_count : 77219
> [ 455.105958] .ttwu_local : 74144
> [ 455.107957] .bkl_count : 0
> [ 455.109914]
> [ 455.109915] cfs_rq[0]:/
> [ 455.113642] .exec_clock : 20017.048374
> [ 455.115515] .MIN_vruntime : 0.000001
> [ 455.117353] .min_vruntime : 28900.800090
> [ 455.119185] .max_vruntime : 0.000001
> [ 455.121028] .spread : 0.000000
> [ 455.122820] .spread0 : 0.000000
> [ 455.124581] .nr_spread_over : 54
> [ 455.126318] .nr_running : 0
> [ 455.128045] .load : 0
> [ 455.129743] .load_avg : 0.000000
> [ 455.131414] .load_period : 0.000000
> [ 455.133052] .load_contrib : 0
> [ 455.134692] .load_tg : 0
> [ 455.136305]
> [ 455.136306] runnable tasks:
> [ 455.136307] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 455.136310] ----------------------------------------------------------------------------------------------------------
> [ 455.142856]
> [ 455.142857] cpu#1, 2491.994 MHz
> [ 455.146215] .nr_running : 0
> [ 455.147931] .load : 0
> [ 455.149648] .nr_switches : 62175
> [ 455.151348] .nr_load_updates : 217324
> [ 455.153045] .nr_uninterruptible : 2
> [ 455.154747] .next_balance : 4295.123306
> [ 455.156457] .curr->pid : 0
> [ 455.158149] .clock : 455157.364642
> [ 455.159867] .cpu_load[0] : 0
> [ 455.161594] .cpu_load[1] : 0
> [ 455.163283] .cpu_load[2] : 0
> [ 455.164954] .cpu_load[3] : 0
> [ 455.166575] .cpu_load[4] : 0
> [ 455.168185] .yld_count : 60
> [ 455.169791] .sched_switch : 0
> [ 455.171401] .sched_count : 62899
> [ 455.172984] .sched_goidle : 27394
> [ 455.174580] .avg_idle : 1000000
> [ 455.176171] .ttwu_count : 30510
> [ 455.177739] .ttwu_local : 25277
> [ 455.179292] .bkl_count : 0
> [ 455.180882]
> [ 455.180883] cfs_rq[1]:/
> [ 455.183954] .exec_clock : 10655.021809
> [ 455.185550] .MIN_vruntime : 0.000001
> [ 455.187141] .min_vruntime : 19718.135550
> [ 455.188771] .max_vruntime : 0.000001
> [ 455.190407] .spread : 0.000000
> [ 455.192016] .spread0 : -9182.664540
> [ 455.193634] .nr_spread_over : 80
> [ 455.195242] .nr_running : 0
> [ 455.196848] .load : 0
> [ 455.198441] .load_avg : 0.000000
> [ 455.200059] .load_period : 0.000000
> [ 455.201654] .load_contrib : 0
> [ 455.203240] .load_tg : 0
> [ 455.204836]
> [ 455.204837] runnable tasks:
> [ 455.204838] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 455.204841] ----------------------------------------------------------------------------------------------------------
> [ 455.211403]
> [ 455.211404] cpu#2, 2491.994 MHz
> [ 455.214958] .nr_running : 0
> [ 455.216771] .load : 0
> [ 455.218564] .nr_switches : 37444
> [ 455.220394] .nr_load_updates : 219042
> [ 455.222202] .nr_uninterruptible : 0
> [ 455.224039] .next_balance : 4295.123375
> [ 455.225897] .curr->pid : 0
> [ 455.227769] .clock : 455227.204963
> [ 455.229681] .cpu_load[0] : 0
> [ 455.231548] .cpu_load[1] : 0
> [ 455.233353] .cpu_load[2] : 0
> [ 455.235059] .cpu_load[3] : 0
> [ 455.236676] .cpu_load[4] : 0
> [ 455.238280] .yld_count : 0
> [ 455.239879] .sched_switch : 0
> [ 455.241471] .sched_count : 37815
> [ 455.243075] .sched_goidle : 16831
> [ 455.244674] .avg_idle : 1000000
> [ 455.246270] .ttwu_count : 18348
> [ 455.247849] .ttwu_local : 16899
> [ 455.249430] .bkl_count : 0
> [ 455.250992]
> [ 455.250993] cfs_rq[2]:/
> [ 455.254040] .exec_clock : 6758.351942
> [ 455.255630] .MIN_vruntime : 0.000001
> [ 455.257236] .min_vruntime : 13719.942866
> [ 455.258861] .max_vruntime : 0.000001
> [ 455.260497] .spread : 0.000000
> [ 455.262122] .spread0 : -15180.857224
> [ 455.263753] .nr_spread_over : 21
> [ 455.265389] .nr_running : 0
> [ 455.267018] .load : 0
> [ 455.268637] .load_avg : 0.000000
> [ 455.270292] .load_period : 0.000000
> [ 455.271911] .load_contrib : 0
> [ 455.273530] .load_tg : 0
> [ 455.275163]
> [ 455.275165] runnable tasks:
> [ 455.275166] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 455.275169] ----------------------------------------------------------------------------------------------------------
> [ 455.281810]
> [ 455.281811] cpu#3, 2491.994 MHz
> [ 455.285349] .nr_running : 4
> [ 455.287149] .load : 2048
> [ 455.288952] .nr_switches : 30358
> [ 455.290787] .nr_load_updates : 223302
> [ 455.292599] .nr_uninterruptible : 1
> [ 455.294419] .next_balance : 4295.123506
> [ 455.296250] .curr->pid : 46
> [ 455.298090] .clock : 431792.394314
> [ 455.299953] .cpu_load[0] : 2048
> [ 455.301810] .cpu_load[1] : 2048
> [ 455.303610] .cpu_load[2] : 2048
> [ 455.305311] .cpu_load[3] : 2048
> [ 455.306926] .cpu_load[4] : 2048
> [ 455.308531] .yld_count : 63
> [ 455.310130] .sched_switch : 0
> [ 455.311723] .sched_count : 30824
> [ 455.313315] .sched_goidle : 13491
> [ 455.314904] .avg_idle : 1000000
> [ 455.316481] .ttwu_count : 13858
> [ 455.318057] .ttwu_local : 12507
> [ 455.319629] .bkl_count : 0
> [ 455.321187]
> [ 455.321188] cfs_rq[3]:/
> [ 455.324213] .exec_clock : 5334.144946
> [ 455.325795] .MIN_vruntime : 13295.262803
> [ 455.327401] .min_vruntime : 13302.523317
> [ 455.329046] .max_vruntime : 13295.262803
> [ 455.330656] .spread : 0.000000
> [ 455.332297] .spread0 : -15598.276773
> [ 455.333925] .nr_spread_over : 117
> [ 455.335577] .nr_running : 2
> [ 455.337204] .load : 2048
> [ 455.338823] .load_avg : 0.000000
> [ 455.340476] .load_period : 0.000000
> [ 455.342094] .load_contrib : 0
> [ 455.343702] .load_tg : 0
> [ 455.345302]
> [ 455.345303] rt_rq[3]:/
> [ 455.348416] .rt_nr_running : 1
> [ 455.350006] .rt_throttled : 0
> [ 455.351587] .rt_time : 0.000000
> [ 455.353173] .rt_runtime : 950.000000
> [ 455.354759]
> [ 455.354760] runnable tasks:
> [ 455.354761] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> [ 455.354764] ----------------------------------------------------------------------------------------------------------
> [ 455.361357] migration/3 17 0.000000 1084 0 0.000000 0.000933 0.000000 /
> [ 455.363278] watchdog/3 20 0.000000 31 0 0.000000 0.780405 0.002351 /
> [ 455.365212] R kswapd0 46 13302.523317 714 120 13302.523317 1148.767369 245855.504389 /
> [ 455.367220] kworker/3:1 74 13295.262803 10488 120 13295.262803 324.924994 315669.659686 /
> [ 455.369317]
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 20:27 ` Mel Gorman
@ 2011-04-29 15:02 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-29 15:02 UTC (permalink / raw)
To: Mel Gorman
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 21:27 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 02:59:27PM -0500, James Bottomley wrote:
> > On Thu, 2011-04-28 at 20:21 +0100, Mel Gorman wrote:
> > > On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote:
> > > > > Way hey, cgroups are also in the mix. How jolly.
> > > > >
> > > > > Is systemd a common element of the machines hitting this bug by any
> > > > > chance?
> > > >
> > > > Well, yes, the bug report is against FC15, which needs cgroups for
> > > > systemd.
> > > >
> > >
> > > Ok although we do not have direct evidence that it's the problem yet. A
> > > broken shrinker could just mean we are also trying to aggressively
> > > reclaim in cgroups.
> > >
> > > > > The remaining traces seem to be follow-on damage related to the three
> > > > > issues of "shrinkers are bust in some manner" causing "we are not
> > > > > getting over the min watermark" and as a side-show "we are spending lots
> > > > > of time doing something unspecified but unhelpful in cgroups".
> > > >
> > > > Heh, well find a way for me to verify this: I can't turn off cgroups
> > > > because systemd then won't work and the machine won't boot ...
> > > >
> > >
> > > Same testcase, same kernel but a distro that is not using systemd to
> > > verify if cgroups are the problem. Not ideal I know. When I'm back
> > > online Tuesday, I'll try reproducing this on a !Fedora distribution. In
> > > the meantime, the following untested hatchet job might spit out
> > > which shrinker we are getting stuck in. It is also breaking out of
> > > the shrink_slab loop so it'd even be interesting to see if the bug
> > > is mitigated in any way.
> >
> > Actually, talking to Chris, I think I can get the system up using
> > init=/bin/bash without systemd, so I can try the no cgroup config.
> >
> > > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > > index c74a501..ed99104 100644
> >
> > In the mean time, this patch produces:
> >
> > (that's nothing ... apparently the trace doesn't activate when kswapd
> > goes mad).
> >
>
> Or is looping there for shorter than we expect. HZ/10?
Still doesn't print anything, even with HZ/10.
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-29 15:02 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-29 15:02 UTC (permalink / raw)
To: Mel Gorman
Cc: Jan Kara, colin.king, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 21:27 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 02:59:27PM -0500, James Bottomley wrote:
> > On Thu, 2011-04-28 at 20:21 +0100, Mel Gorman wrote:
> > > On Thu, Apr 28, 2011 at 01:30:36PM -0500, James Bottomley wrote:
> > > > > Way hey, cgroups are also in the mix. How jolly.
> > > > >
> > > > > Is systemd a common element of the machines hitting this bug by any
> > > > > chance?
> > > >
> > > > Well, yes, the bug report is against FC15, which needs cgroups for
> > > > systemd.
> > > >
> > >
> > > Ok although we do not have direct evidence that it's the problem yet. A
> > > broken shrinker could just mean we are also trying to aggressively
> > > reclaim in cgroups.
> > >
> > > > > The remaining traces seem to be follow-on damage related to the three
> > > > > issues of "shrinkers are bust in some manner" causing "we are not
> > > > > getting over the min watermark" and as a side-show "we are spending lots
> > > > > of time doing something unspecified but unhelpful in cgroups".
> > > >
> > > > Heh, well find a way for me to verify this: I can't turn off cgroups
> > > > because systemd then won't work and the machine won't boot ...
> > > >
> > >
> > > Same testcase, same kernel but a distro that is not using systemd to
> > > verify if cgroups are the problem. Not ideal I know. When I'm back
> > > online Tuesday, I'll try reproducing this on a !Fedora distribution. In
> > > the meantime, the following untested hatchet job might spit out
> > > which shrinker we are getting stuck in. It is also breaking out of
> > > the shrink_slab loop so it'd even be interesting to see if the bug
> > > is mitigated in any way.
> >
> > Actually, talking to Chris, I think I can get the system up using
> > init=/bin/bash without systemd, so I can try the no cgroup config.
> >
> > > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > > index c74a501..ed99104 100644
> >
> > In the mean time, this patch produces:
> >
> > (that's nothing ... apparently the trace doesn't activate when kswapd
> > goes mad).
> >
>
> Or is looping there for shorter than we expect. HZ/10?
Still doesn't print anything, even with HZ/10.
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-29 10:23 ` Sedat Dilek
@ 2011-04-29 15:37 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-29 15:37 UTC (permalink / raw)
To: sedat.dilek; +Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel
On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote:
> But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try.
> First, I have seen negative effects on my UP-system was when playing
> with linux-next [2].
> It was not clear what the origin was and the the side-effects were
> somehow "bizarre".
> The issue could be easily reproduced by tar-ing the kernel build-dir
> to an external USB-hdd.
> The issue kept RCU and TIP folks really busy.
> Before stepping 4 weeks in the dark, give it a try and let me know in
> case of success.
Well, it's highly unlikely because that's a 2.6.39 artifact and the bug
showed up in 2.6.38 ... I tried it just in case with no effect, so we
know it isn't the cause.
> For the systemd/cgroups part of the discussion (yes, I followed this
> thread in parallel):
> The patch from [4] might be interesting (untested here).
What makes you think that? This patch is about rcu problems in cgroup
manipulation through the mounted cgroup control filesystems. The bug in
question manifests when all of that is quiescent, so it's not likely to
be related.
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-29 15:37 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-29 15:37 UTC (permalink / raw)
To: sedat.dilek; +Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel
On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote:
> But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try.
> First, I have seen negative effects on my UP-system was when playing
> with linux-next [2].
> It was not clear what the origin was and the the side-effects were
> somehow "bizarre".
> The issue could be easily reproduced by tar-ing the kernel build-dir
> to an external USB-hdd.
> The issue kept RCU and TIP folks really busy.
> Before stepping 4 weeks in the dark, give it a try and let me know in
> case of success.
Well, it's highly unlikely because that's a 2.6.39 artifact and the bug
showed up in 2.6.38 ... I tried it just in case with no effect, so we
know it isn't the cause.
> For the systemd/cgroups part of the discussion (yes, I followed this
> thread in parallel):
> The patch from [4] might be interesting (untested here).
What makes you think that? This patch is about rcu problems in cgroup
manipulation through the mounted cgroup control filesystems. The bug in
question manifests when all of that is quiescent, so it's not likely to
be related.
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-29 15:37 ` James Bottomley
@ 2011-04-29 16:31 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-29 16:31 UTC (permalink / raw)
To: sedat.dilek
Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, Thomas Gleixner
On Fri, 2011-04-29 at 10:37 -0500, James Bottomley wrote:
> On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote:
> > But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try.
> > First, I have seen negative effects on my UP-system was when playing
> > with linux-next [2].
> > It was not clear what the origin was and the the side-effects were
> > somehow "bizarre".
> > The issue could be easily reproduced by tar-ing the kernel build-dir
> > to an external USB-hdd.
> > The issue kept RCU and TIP folks really busy.
> > Before stepping 4 weeks in the dark, give it a try and let me know in
> > case of success.
>
> Well, it's highly unlikely because that's a 2.6.39 artifact and the bug
> showed up in 2.6.38 ... I tried it just in case with no effect, so we
> know it isn't the cause.
Actually, I tell a lie: it does't stop kswapd spinning on PREEMPT, but
it does seem to prevent non-PREEMPT from locking up totally (at least it
survives three back to back untar runs).
It's probable it alters the memory pin conditions that cause the spin,
so it's masking the problem rather than fixing it.
The question now is, what causes it in 2.6.38 stable?
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-29 16:31 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-29 16:31 UTC (permalink / raw)
To: sedat.dilek
Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, Thomas Gleixner
On Fri, 2011-04-29 at 10:37 -0500, James Bottomley wrote:
> On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote:
> > But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try.
> > First, I have seen negative effects on my UP-system was when playing
> > with linux-next [2].
> > It was not clear what the origin was and the the side-effects were
> > somehow "bizarre".
> > The issue could be easily reproduced by tar-ing the kernel build-dir
> > to an external USB-hdd.
> > The issue kept RCU and TIP folks really busy.
> > Before stepping 4 weeks in the dark, give it a try and let me know in
> > case of success.
>
> Well, it's highly unlikely because that's a 2.6.39 artifact and the bug
> showed up in 2.6.38 ... I tried it just in case with no effect, so we
> know it isn't the cause.
Actually, I tell a lie: it does't stop kswapd spinning on PREEMPT, but
it does seem to prevent non-PREEMPT from locking up totally (at least it
survives three back to back untar runs).
It's probable it alters the memory pin conditions that cause the spin,
so it's masking the problem rather than fixing it.
The question now is, what causes it in 2.6.38 stable?
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-29 16:31 ` James Bottomley
@ 2011-04-29 18:02 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-29 18:02 UTC (permalink / raw)
To: sedat.dilek
Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, Thomas Gleixner
On Fri, 2011-04-29 at 11:31 -0500, James Bottomley wrote:
> On Fri, 2011-04-29 at 10:37 -0500, James Bottomley wrote:
> > On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote:
> > > But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try.
> > > First, I have seen negative effects on my UP-system was when playing
> > > with linux-next [2].
> > > It was not clear what the origin was and the the side-effects were
> > > somehow "bizarre".
> > > The issue could be easily reproduced by tar-ing the kernel build-dir
> > > to an external USB-hdd.
> > > The issue kept RCU and TIP folks really busy.
> > > Before stepping 4 weeks in the dark, give it a try and let me know in
> > > case of success.
> >
> > Well, it's highly unlikely because that's a 2.6.39 artifact and the bug
> > showed up in 2.6.38 ... I tried it just in case with no effect, so we
> > know it isn't the cause.
>
> Actually, I tell a lie: it does't stop kswapd spinning on PREEMPT, but
> it does seem to prevent non-PREEMPT from locking up totally (at least it
> survives three back to back untar runs).
>
> It's probable it alters the memory pin conditions that cause the spin,
> so it's masking the problem rather than fixing it.
Confirmed ... it's just harder to reproduce with the hrtimers init fix.
The problem definitely still exists (I had to load up the system more
before doing the tar).
This time I've caught kswapd in mem_cgroup_shrink_node_zone. sysrq-w
doesn't complete for an unknown reason
James
---
[ 575.083025] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46]
[ 575.083043] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 575.083251] CPU 2
[ 575.083255] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 575.083430]
[ 575.083437] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO
[ 575.083455] RIP: 0010:[<ffffffffa007ff8f>] [<ffffffffa007ff8f>] i915_gem_inactive_shrink+0x6c/0x194 [i915]
[ 575.083497] RSP: 0018:ffff88006dfb7d50 EFLAGS: 00000206
[ 575.083504] RAX: ffff88005f018200 RBX: 00000000000000c0 RCX: 0000000000000000
[ 575.083512] RDX: ffff88005f01bab0 RSI: 0000000000000000 RDI: ffff880037885820
[ 575.083529] RBP: ffff88006dfb7d90 R08: 0000000000000000 R09: 0000000000017131
[ 575.083535] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff81482e0e
[ 575.083541] R13: ffff88006dfb7cf0 R14: ffff88006dfb7cf8 R15: ffffffff810dd44d
[ 575.083548] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 575.083555] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 575.083572] CR2: 00007f3934e8a000 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 575.083579] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 575.083587] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 575.083596] Process kswapd0 (pid: 46, threadinfo ffff88006dfb6000, task ffff88006dfb8000)
[ 575.083602] Stack:
[ 575.083608] ffff88006dfb7d90 ffff880037709638 ffff88006dfb7d60 ffff8800377095f0
[ 575.083626] 0000000000000000 0000000000000000 00000000000000d0 000000000005103b
[ 575.083643] ffff88006dfb7de0 ffffffff810e1d89 000000000000003d 0000000000000080
[ 575.083660] Call Trace:
[ 575.083676] [<ffffffff810e1d89>] shrink_slab+0x6d/0x166
[ 575.083686] [<ffffffff810e49cc>] kswapd+0x533/0x798
[ 575.083698] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 575.083712] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 575.083726] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 575.083738] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 575.083750] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 575.083756] Code: e4 48 89 45 c8 75 37 48 8b 43 48 45 31 ed 48 83 c3 48 48 2d b0 00 00 00 eb 0a 48 8d 82 50 ff ff ff 41 ff c5 48 8b 90 b0 00 00 00
[ 575.083887] 05 b0 00 00 00 48 39 d8 0f 18 0a 75 e1 e9 da 00 00 00 4c 89
[ 575.083949] Call Trace:
[ 575.083957] [<ffffffff810e1d89>] shrink_slab+0x6d/0x166
[ 575.083966] [<ffffffff810e49cc>] kswapd+0x533/0x798
[ 575.083976] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 575.083986] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 575.083998] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 575.084010] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 575.084030] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 658.883363] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46]
[ 658.883382] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 658.883574] CPU 2
[ 658.883579] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 658.883749]
[ 658.883757] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO
[ 658.883776] RIP: 0010:[<ffffffff81080bc0>] [<ffffffff81080bc0>] arch_local_irq_restore+0xc/0xd
[ 658.883797] RSP: 0018:ffff88006dfb7d98 EFLAGS: 00000246
[ 658.883805] RAX: ffff8801005f9e68 RBX: ffff8801005e6e00 RCX: 0000000000004ae4
[ 658.883813] RDX: ffff88006dfb8000 RSI: 0000000000000246 RDI: 0000000000000246
[ 658.883821] RBP: ffff88006dfb7da0 R08: ffff8801005f9e70 R09: 00000000000007eb
[ 658.883828] R10: 0000000000000000 R11: 00000000000000cd R12: ffffffff81482e0e
[ 658.883835] R13: 0000000000016fd3 R14: 000000000000a731 R15: ffffffff810ddb31
[ 658.883844] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 658.883853] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 658.883860] CR2: 00007f3934e8a000 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 658.883867] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 658.883875] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 658.883883] Process kswapd0 (pid: 46, threadinfo ffff88006dfb6000, task ffff88006dfb8000)
[ 658.883890] Stack:
[ 658.883895] ffffffff8147bc8c ffff88006dfb7de0 ffffffff8106e7c3 ffff880000000002
[ 658.883913] 0000000000000000 ffff8801005e6000 ffff8801005f9e68 0000000000000002
[ 658.883929] 0000000000000000 ffff88006dfb7ee0 ffffffff810e45f0 0000000000000003
[ 658.883947] Call Trace:
[ 658.883961] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 658.883976] [<ffffffff8106e7c3>] prepare_to_wait+0x6c/0x78
[ 658.883988] [<ffffffff810e45f0>] kswapd+0x157/0x798
[ 658.884000] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 658.884009] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 658.884020] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 658.884033] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 658.884045] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 658.884057] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 658.884062] Code: 1f 44 00 00 66 ff 05 30 16 98 00 fb 66 0f 1f 44 00 00 5d c3 55 48 89 e5 0f 1f 44 00 00 5d c3 55 48 89 e5 57 9d 0f 1f 44 00 00 5d <c3> 55 48 89 e5 fa 66 0f 1f 44 00 00 5d c3 55 48 89 e5 50 9c 58
[ 658.884238] Call Trace:
[ 658.884248] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 658.884260] [<ffffffff8106e7c3>] prepare_to_wait+0x6c/0x78
[ 658.884269] [<ffffffff810e45f0>] kswapd+0x157/0x798
[ 658.884280] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 658.884289] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 658.884299] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 658.884311] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 658.884322] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 658.884333] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 690.767137] SysRq : Show backtrace of all active CPUs
[ 690.767158] sending NMI to all CPUs:
[ 690.767171] NMI backtrace for cpu 2
[ 690.767188] CPU 2
[ 690.767196] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 690.767457]
[ 690.767468] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO
[ 690.767493] RIP: 0010:[<ffffffff810e3980>] [<ffffffff810e3980>] shrink_zone+0x221/0x4b3
[ 690.767519] RSP: 0018:ffff88006dfb7d30 EFLAGS: 00000206
[ 690.767529] RAX: 0000000000000000 RBX: ffff88006dfb7e58 RCX: 000000000000000c
[ 690.767539] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 690.767550] RBP: ffff88006dfb7de0 R08: 0000000000000001 R09: 00000000000007eb
[ 690.767561] R10: 0000000000000002 R11: 00000000000000f5 R12: ffff8801005e6000
[ 690.767572] R13: ffff8801005e64d8 R14: 0000000000001572 R15: 00000000ffffffff
[ 690.767584] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 690.767597] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 690.767607] CR2: 00007f3934e8a000 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 690.767618] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 690.767628] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 690.767640] Process kswapd0 (pid: 46, threadinfo ffff88006dfb6000, task ffff88006dfb8000)
[ 690.767650] Stack:
[ 690.767657] 0000000000000001 000000000000008c 000000000000003c 0000000000000000
[ 690.767683] ffffffffffffffff ffff8801005e6480 000000000000000c 0000000c81118591
[ 690.767717] 0000000000000000 0000000000000000 ffff880100530380 0000000000000000
[ 690.767742] Call Trace:
[ 690.767764] [<ffffffff810d9967>] ? zone_watermark_ok_safe+0xa1/0xae
[ 690.767779] [<ffffffff810e49a6>] kswapd+0x50d/0x798
[ 690.767794] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 690.767811] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 690.767830] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 690.767846] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 690.767861] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 690.767868] Code: e2 ff ff 83 7d 80 00 44 8b 85 50 ff ff ff 74 1b 8a 4d 8c 31 f6 41 83 ff 01 40 0f 96 c6 31 d2 48 d3 e8 48 0f af 44 f5 c0 49 f7 f6
[ 690.768028] 89 c1 48 8d 51 04 49 03 44 d5 00 48 83 f8 1f 49 89 44 d5 00
[ 690.768113] Call Trace:
[ 690.768126] [<ffffffff810d9967>] ? zone_watermark_ok_safe+0xa1/0xae
[ 690.768139] [<ffffffff810e49a6>] kswapd+0x50d/0x798
[ 690.768154] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 690.768168] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 690.768183] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 690.768199] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 690.768213] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 690.768223] NMI backtrace for cpu 3
[ 690.768236] CPU 3
[ 690.768244] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 690.768832]
[ 690.768843] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO
[ 690.768867] RIP: 0010:[<ffffffff81275b36>] [<ffffffff81275b36>] intel_idle+0xaa/0x100
[ 690.768890] RSP: 0018:ffff8800715dfe68 EFLAGS: 00000046
[ 690.768900] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001
[ 690.768912] RDX: 0000000000000000 RSI: ffff8800715dffd8 RDI: ffffffff81a0e640
[ 690.768923] RBP: ffff8800715dfeb8 R08: 000000000000006d R09: 00000000000003c2
[ 690.768934] R10: ffffffff00000003 R11: ffff8801002d3b40 R12: 0000000000000030
[ 690.768946] R13: 12191803fa451798 R14: 0000000000000004 R15: 0000000000000003
[ 690.768959] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000
[ 690.768971] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 690.768982] CR2: 0000000000429bf0 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 690.768993] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 690.769005] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 690.769017] Process kworker/0:1 (pid: 0, threadinfo ffff8800715de000, task ffff8800715e1700)
[ 690.769027] Stack:
[ 690.769034] ffff8800715dfe88 ffffffff810731c0 ffff8801002d1280 0000000000011280
[ 690.769063] ffff8800715dfeb8 000000038139c77a ffffe8ffffcc0170 ffffe8ffffcc0170
[ 690.769092] ffffe8ffffcc0300 0000000000000000 ffff8800715dfef8 ffffffff8139b668
[ 690.769121] Call Trace:
[ 690.769137] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 690.769153] [<ffffffff8139b668>] cpuidle_idle_call+0xe7/0x166
[ 690.769173] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 690.769191] [<ffffffff8146ac57>] start_secondary+0x223/0x225
[ 690.769203] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 0b e0 ff 4c 29 e8 48 89 c7 e8 ab 2b de ff 4c 69 e0 40 42
[ 690.769477] Call Trace:
[ 690.769491] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 690.769506] [<ffffffff8139b668>] cpuidle_idle_call+0xe7/0x166
[ 690.769522] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 690.770451] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 690.771380] [<ffffffff81483f73>] ? smp_apic_timer_interrupt+0x7e/0x8c
[ 692.887516] SysRq : Show Blocked State
[ 692.887534] task PC stack pid father
[ 692.887551] fsnotify_mark D 0000000000000000 0 49 2 0x00000000
[ 692.887567] ffff88006dfb1cc0 0000000000000046 0000000000000000 0000000000000000
[ 692.887585] ffff88006dfbc500 ffff88006dfb1fd8 ffff88006dfb1fd8 0000000000013b40
[ 692.887601] ffff880071589700 ffff88006dfbc500 ffff88006dfb1cb0 000000010024f4c0
[ 692.887618] Call Trace:
[ 692.887635] [<ffffffff8147aa4b>] schedule_timeout+0x34/0xde
[ 692.887650] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61
[ 692.887662] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd
[ 692.887673] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 692.887683] [<ffffffff8147a814>] wait_for_common+0xac/0x101
[ 692.887692] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226
[ 692.887702] [<ffffffff810ada0d>] ? synchronize_rcu_bh+0x5c/0x5c
[ 692.887711] [<ffffffff8147a91d>] wait_for_completion+0x1d/0x1f
[ 692.887720] [<ffffffff810ada67>] synchronize_sched+0x5a/0x5c
[ 692.887730] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43
[ 692.887740] [<ffffffff810726fc>] __synchronize_srcu+0x31/0x89
[ 692.887751] [<ffffffff81072780>] synchronize_srcu+0x15/0x17
[ 692.887762] [<ffffffff8114c8be>] fsnotify_mark_destroy+0x90/0x165
[ 692.887774] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.887783] [<ffffffff8114c82e>] ? fsnotify_put_mark+0x1c/0x1c
[ 692.887793] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 692.887806] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 692.887818] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 692.887828] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 692.887840] jbd2/dm-1-8 D 0000000000000000 0 377 2 0x00000000
[ 692.887854] ffff88006d765cd0 0000000000000046 0000000000000000 0000000000000000
[ 692.887871] ffff88006d64dc00 ffff88006d765fd8 ffff88006d765fd8 0000000000013b40
[ 692.887888] ffff880071589700 ffff88006d64dc00 ffff88006d765cd0 000000018106e7c3
[ 692.887905] Call Trace:
[ 692.887917] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194
[ 692.887928] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61
[ 692.887939] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd
[ 692.887949] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 692.887960] [<ffffffff8100804e>] ? load_TLS+0x10/0x14
[ 692.887971] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
[ 692.887982] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.887992] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
[ 692.888003] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 692.888012] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81
[ 692.888023] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a
[ 692.888033] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.888043] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10
[ 692.888053] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 692.888065] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 692.888076] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 692.888087] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 692.888095] flush-253:1 D 0000000000000000 0 439 2 0x00000000
[ 692.888117] ffff88006d70f730 0000000000000046 ffff88006d70f6c0 ffffffff00000000
[ 692.888135] ffff880037b7ae00 ffff88006d70ffd8 ffff88006d70ffd8 0000000000013b40
[ 692.888153] ffff880071589700 ffff880037b7ae00 ffff8801002543c0 000000016d70f798
[ 692.888170] Call Trace:
[ 692.888180] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e
[ 692.888190] [<ffffffff81217900>] get_request_wait+0x102/0x18b
[ 692.888200] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.888209] [<ffffffff81218641>] __make_request+0x18a/0x2b8
[ 692.888219] [<ffffffff81217338>] generic_make_request+0x2a9/0x323
[ 692.888227] [<ffffffff81217490>] submit_bio+0xde/0xfd
[ 692.888238] [<ffffffff81199bd9>] ? ext4_bio_write_page+0x30d/0x320
[ 692.888248] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58
[ 692.888257] [<ffffffff81194b65>] mpage_da_submit_io+0x371/0x389
[ 692.888267] [<ffffffff8119527d>] ? ext4_mark_iloc_dirty+0x4db/0x543
[ 692.888276] [<ffffffff8119552d>] ? ext4_mark_inode_dirty+0x1c5/0x1f0
[ 692.888287] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd
[ 692.888298] [<ffffffff811ab72a>] ? ext4_journal_start_sb+0x108/0x120
[ 692.888308] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d
[ 692.888319] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a
[ 692.888331] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc
[ 692.888341] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161
[ 692.888351] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b
[ 692.888361] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335
[ 692.888371] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 692.888381] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1
[ 692.888392] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d
[ 692.888402] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5
[ 692.888413] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d
[ 692.888423] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 692.888434] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 692.888445] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 692.888455] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 692.888472] jbd2/dm-2-8 D ffff880071617380 0 791 2 0x00000000
[ 692.888485] ffff88006cfb1cd0 0000000000000046 0000000000000000 ffff880000000001
[ 692.888501] ffff88003769ae00 ffff88006cfb1fd8 ffff88006cfb1fd8 0000000000013b40
[ 692.888517] ffff880058cdc500 ffff88003769ae00 ffff88006cfb1cd0 000000018106e7c3
[ 692.888533] Call Trace:
[ 692.888547] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194
[ 692.888559] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.888568] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
[ 692.888578] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 692.888587] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81
[ 692.888597] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a
[ 692.888607] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.888617] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10
[ 692.888627] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 692.888637] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 692.888648] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 692.888659] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 692.888665] NetworkManager D 0000000000000000 0 809 1 0x00000000
[ 692.888682] ffff88006ff23ad8 0000000000000082 ffff880100293b40 0000000000000000
[ 692.888697] ffff88006cfe2e00 ffff88006ff23fd8 ffff88006ff23fd8 0000000000013b40
[ 692.888713] ffffffff81a0b020 ffff88006cfe2e00 ffff88006ff23ad8 000000018106e7c3
[ 692.888729] Call Trace:
[ 692.888738] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465
[ 692.888753] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.888761] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda
[ 692.888770] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15
[ 692.888780] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120
[ 692.888790] [<ffffffff81044796>] ? delayacct_blkio_end+0x21/0x41
[ 692.888799] [<ffffffff8147a5df>] ? io_schedule+0x79/0x7e
[ 692.888808] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c
[ 692.888818] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175
[ 692.888827] [<ffffffff811320c7>] touch_atime+0x10e/0x131
[ 692.888840] [<ffffffff810d7160>] generic_file_aio_read+0x5d9/0x640
[ 692.888851] [<ffffffff8111ecaa>] do_sync_read+0xbf/0xff
[ 692.888860] [<ffffffff8103fd1f>] ? should_resched+0xe/0x2d
[ 692.888872] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33
[ 692.888884] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd
[ 692.888894] [<ffffffff8111f311>] vfs_read+0xa9/0xf0
[ 692.888902] [<ffffffff8111f3a2>] sys_read+0x4a/0x6e
[ 692.888912] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b
[ 692.888923] rs:main Q:Reg D 0000000000000000 0 886 1 0x00000000
[ 692.888935] ffff88006ee35a38 0000000000000082 ffff88010028fa00 ffff880100000000
[ 692.888950] ffff88006ed5dc00 ffff88006ee35fd8 ffff88006ee35fd8 0000000000013b40
[ 692.888966] ffff880071589700 ffff88006ed5dc00 ffff88006ee35a38 000000018106e7c3
[ 692.888986] Call Trace:
[ 692.888994] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465
[ 692.889005] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.889013] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda
[ 692.889022] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15
[ 692.889031] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120
[ 692.889041] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c
[ 692.889050] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175
[ 692.889059] [<ffffffff81131f95>] file_update_time+0xef/0x113
[ 692.889070] [<ffffffff810d69b8>] __generic_file_aio_write+0x15d/0x272
[ 692.889086] [<ffffffff8107cd0e>] ? futex_wait_queue_me+0xc4/0xe0
[ 692.889099] [<ffffffff810d6b2e>] generic_file_aio_write+0x61/0xba
[ 692.889115] [<ffffffff8118fc00>] ext4_file_write+0x1dc/0x234
[ 692.889125] [<ffffffff8111ebab>] do_sync_write+0xbf/0xff
[ 692.889133] [<ffffffff8114b810>] ? fsnotify+0x1ff/0x217
[ 692.889144] [<ffffffff811f1666>] ? selinux_file_permission+0x58/0xb4
[ 692.889155] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33
[ 692.889164] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd
[ 692.889172] [<ffffffff8111f221>] vfs_write+0xac/0xf3
[ 692.889181] [<ffffffff8111f410>] sys_write+0x4a/0x6e
[ 692.889191] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b
[ 692.889199] flush-253:2 D 0000000000000000 0 883 2 0x00000000
[ 692.889211] ffff88006ee736b0 0000000000000046 ffff88006ee73640 ffffffff00000000
[ 692.889446] ffff88006d5d9700 ffff88006ee73fd8 ffff88006ee73fd8 0000000000013b40
[ 692.889462] ffff880071589700 ffff88006d5d9700 ffff8801002543c0 000000016ee73718
[ 692.889478] Call Trace:
[ 692.889488] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e
[ 692.889496] [<ffffffff81217900>] get_request_wait+0x102/0x18b
[ 692.889507] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.889516] [<ffffffff81218641>] __make_request+0x18a/0x2b8
[ 692.889525] [<ffffffff81217338>] generic_make_request+0x2a9/0x323
[ 692.889536] [<ffffffff8147bd19>] ? _raw_read_unlock_irqrestore+0x17/0x19
[ 692.889547] [<ffffffff8138d520>] ? dm_target_exit+0x19/0x19
[ 692.889555] [<ffffffff81217490>] submit_bio+0xde/0xfd
[ 692.889566] [<ffffffff810ea4e1>] ? inc_zone_page_state+0x27/0x29
[ 692.889576] [<ffffffff810dc9fd>] ? account_page_writeback+0x25/0x29
[ 692.889587] [<ffffffff8122e200>] ? radix_tree_gang_lookup_slot+0x66/0x87
[ 692.889597] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58
[ 692.889606] [<ffffffff81199a48>] ext4_bio_write_page+0x17c/0x320
[ 692.889616] [<ffffffff810ea4a1>] ? mod_state+0x76/0x7d
[ 692.889625] [<ffffffff81194afa>] mpage_da_submit_io+0x306/0x389
[ 692.889636] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd
[ 692.889645] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d
[ 692.889656] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a
[ 692.889666] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc
[ 692.889677] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161
[ 692.889687] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b
[ 692.889697] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335
[ 692.889706] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
[ 692.889717] [<ffffffff8147bc3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[ 692.889727] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1
[ 692.889736] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d
[ 692.889746] [<ffffffff8147aac7>] ? schedule_timeout+0xb0/0xde
[ 692.889756] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5
[ 692.889766] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d
[ 692.889776] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 692.889787] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 692.889798] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 692.889809] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 692.889815] abrt-dump-oops D 0000000000000000 0 903 817 0x00000000
[ 692.889827] ffff88006eda1c78 0000000000000082 0000000000000005 0000000000000000
[ 692.889843] ffff880037698000 ffff88006eda1fd8 ffff88006eda1fd8 0000000000013b40
[ 692.889860] ffffffff81a0b020 ffff880037698000 ffff88006eda1c78 000000018106e7c3
[ 692.889876] Call Trace:
[ 692.889884] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465
[ 692.889895] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.889904] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda
[ 692.889913] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15
[ 692.889923] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120
[ 692.889934] [<ffffffff8119ccaa>] ext4_mkdir+0xc9/0x33e
[ 692.889944] [<ffffffff811297a7>] vfs_mkdir+0x5f/0x9b
[ 692.889954] [<ffffffff8112b0ae>] sys_mkdirat+0x97/0xe8
[ 692.889965] [<ffffffff81122b89>] ? sys_newstat+0x2a/0x33
[ 692.889975] [<ffffffff8112b117>] sys_mkdir+0x18/0x1a
[ 692.889984] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b
[ 692.889998] Xorg D 0000000000000000 0 1187 1183 0x00400084
[ 692.890012] ffff88007171ba38 0000000000000082 ffff8800177db2d8 ffff880000000000
[ 692.890029] ffff88006d581700 ffff88007171bfd8 ffff88007171bfd8 0000000000013b40
[ 692.890046] ffffffff81a0b020 ffff88006d581700 ffff88007171ba38
nc: Write error: No route to host
[ 709.230547] SysRq : Show Blocked State
[ 709.230567] task PC stack pid father
[ 709.230584] fsnotify_mark D 0000000000000000 0 49 2 0x00000000
[ 709.230600] ffff88006dfb1cc0 0000000000000046 0000000000000000 0000000000000000
[ 709.230621] ffff88006dfbc500 ffff88006dfb1fd8 ffff88006dfb1fd8 0000000000013b40
[ 709.230638] ffff880071589700 ffff88006dfbc500 ffff88006dfb1cb0 000000010024f4c0
[ 709.230664] Call Trace:
[ 709.230683] [<ffffffff8147aa4b>] schedule_timeout+0x34/0xde
[ 709.230697] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61
[ 709.230710] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd
[ 709.230721] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 709.230731] [<ffffffff8147a814>] wait_for_common+0xac/0x101
[ 709.230740] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226
[ 709.230751] [<ffffffff810ada0d>] ? synchronize_rcu_bh+0x5c/0x5c
[ 709.230761] [<ffffffff8147a91d>] wait_for_completion+0x1d/0x1f
[ 709.230769] [<ffffffff810ada67>] synchronize_sched+0x5a/0x5c
[ 709.230780] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43
[ 709.230790] [<ffffffff810726fc>] __synchronize_srcu+0x31/0x89
[ 709.230800] [<ffffffff81072780>] synchronize_srcu+0x15/0x17
[ 709.230810] [<ffffffff8114c8be>] fsnotify_mark_destroy+0x90/0x165
[ 709.230822] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.230831] [<ffffffff8114c82e>] ? fsnotify_put_mark+0x1c/0x1c
[ 709.230842] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 709.230854] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 709.230866] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 709.230876] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 709.230889] jbd2/dm-1-8 D 0000000000000000 0 377 2 0x00000000
[ 709.230902] ffff88006d765cd0 0000000000000046 0000000000000000 0000000000000000
[ 709.230920] ffff88006d64dc00 ffff88006d765fd8 ffff88006d765fd8 0000000000013b40
[ 709.230937] ffff880071589700 ffff88006d64dc00 ffff88006d765cd0 000000018106e7c3
[ 709.230955] Call Trace:
[ 709.230967] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194
[ 709.230978] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61
[ 709.230989] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd
[ 709.230999] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 709.231011] [<ffffffff8100804e>] ? load_TLS+0x10/0x14
[ 709.231021] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
[ 709.231033] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.231043] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
[ 709.231055] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 709.231064] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81
[ 709.231075] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a
[ 709.231086] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.231096] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10
[ 709.231106] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 709.231117] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 709.231129] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 709.231139] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 709.231147] flush-253:1 D 0000000000000000 0 439 2 0x00000000
[ 709.231161] ffff88006d70f730 0000000000000046 ffff88006d70f6c0 ffffffff00000000
[ 709.231178] ffff880037b7ae00 ffff88006d70ffd8 ffff88006d70ffd8 0000000000013b40
[ 709.231195] ffff880071589700 ffff880037b7ae00 ffff8801002543c0 000000016d70f798
[ 709.231220] Call Trace:
[ 709.231230] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e
[ 709.231240] [<ffffffff81217900>] get_request_wait+0x102/0x18b
[ 709.231251] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.231260] [<ffffffff81218641>] __make_request+0x18a/0x2b8
[ 709.231270] [<ffffffff81217338>] generic_make_request+0x2a9/0x323
[ 709.231279] [<ffffffff81217490>] submit_bio+0xde/0xfd
[ 709.231290] [<ffffffff81199bd9>] ? ext4_bio_write_page+0x30d/0x320
[ 709.231300] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58
[ 709.231309] [<ffffffff81194b65>] mpage_da_submit_io+0x371/0x389
[ 709.231318] [<ffffffff8119527d>] ? ext4_mark_iloc_dirty+0x4db/0x543
[ 709.231328] [<ffffffff8119552d>] ? ext4_mark_inode_dirty+0x1c5/0x1f0
[ 709.231338] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd
[ 709.231350] [<ffffffff811ab72a>] ? ext4_journal_start_sb+0x108/0x120
[ 709.231360] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d
[ 709.231372] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a
[ 709.231383] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc
[ 709.231393] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161
[ 709.231404] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b
[ 709.231414] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335
[ 709.231424] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 709.231434] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1
[ 709.231445] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d
[ 709.231455] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5
[ 709.231465] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d
[ 709.231475] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 709.231486] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 709.231497] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 709.231513] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 709.231521] jbd2/dm-2-8 D ffff880071617380 0 791 2 0x00000000
[ 709.231534] ffff88006cfb1cd0 0000000000000046 0000000000000000 ffff880000000001
[ 709.231550] ffff88003769ae00 ffff88006cfb1fd8 ffff88006cfb1fd8 0000000000013b40
[ 709.231567] ffff880058cdc500 ffff88003769ae00 ffff88006cfb1cd0 000000018106e7c3
[ 709.231587] Call Trace:
[ 709.231597] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194
[ 709.231609] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.231618] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
[ 709.231628] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 709.231638] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81
[ 709.231652] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a
[ 709.231664] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.231674] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10
[ 709.231684] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 709.231695] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 709.231706] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 709.231716] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 709.231727] NetworkManager D 0000000000000000 0 809 1 0x00000000
[ 709.231739] ffff88006ff23ad8 0000000000000082 ffff880100293b40 0000000000000000
[ 709.231755] ffff88006cfe2e00 ffff88006ff23fd8 ffff88006ff23fd8 0000000000013b40
[ 709.231772] ffffffff81a0b020 ffff88006cfe2e00 ffff88006ff23ad8 000000018106e7c3
[ 709.231787] Call Trace:
[ 709.231801] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465
[ 709.231812] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.231821] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda
[ 709.231829] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15
[ 709.231839] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120
[ 709.231850] [<ffffffff81044796>] ? delayacct_blkio_end+0x21/0x41
[ 709.231859] [<ffffffff8147a5df>] ? io_schedule+0x79/0x7e
[ 709.231868] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c
[ 709.231878] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175
[ 709.231887] [<ffffffff811320c7>] touch_atime+0x10e/0x131
[ 709.231899] [<ffffffff810d7160>] generic_file_aio_read+0x5d9/0x640
[ 709.231911] [<ffffffff8111ecaa>] do_sync_read+0xbf/0xff
[ 709.231919] [<ffffffff8103fd1f>] ? should_resched+0xe/0x2d
[ 709.231935] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33
[ 709.231944] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd
[ 709.231953] [<ffffffff8111f311>] vfs_read+0xa9/0xf0
[ 709.231962] [<ffffffff8111f3a2>] sys_read+0x4a/0x6e
[ 709.231972] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b
[ 709.231983] rs:main Q:Reg D 0000000000000000 0 886 1 0x00000000
[ 709.231995] ffff88006ee35a38 0000000000000082 ffff88010028fa00 ffff880100000000
[ 709.232011] ffff88006ed5dc00 ffff88006ee35fd8 ffff88006ee35fd8 0000000000013b40
[ 709.232027] ffff880071589700 ffff88006ed5dc00 ffff88006ee35a38 000000018106e7c3
[ 709.232047] Call Trace:
[ 709.232055] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465
[ 709.232066] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.232075] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda
[ 709.232083] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15
[ 709.232093] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120
[ 709.232102] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c
[ 709.232112] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175
[ 709.232121] [<ffffffff81131f95>] file_update_time+0xef/0x113
[ 709.232136] [<ffffffff810d69b8>] __generic_file_aio_write+0x15d/0x272
[ 709.232148] [<ffffffff8107cd0e>] ? futex_wait_queue_me+0xc4/0xe0
[ 709.232159] [<ffffffff810d6b2e>] generic_file_aio_write+0x61/0xba
[ 709.232171] [<ffffffff8118fc00>] ext4_file_write+0x1dc/0x234
[ 709.232181] [<ffffffff8111ebab>] do_sync_write+0xbf/0xff
[ 709.232189] [<ffffffff8114b810>] ? fsnotify+0x1ff/0x217
[ 709.232200] [<ffffffff811f1666>] ? selinux_file_permission+0x58/0xb4
[ 709.232210] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33
[ 709.232219] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd
[ 709.232228] [<ffffffff8111f221>] vfs_write+0xac/0xf3
[ 709.232237] [<ffffffff8111f410>] sys_write+0x4a/0x6e
[ 709.232246] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b
[ 709.232255] flush-253:2 D 0000000000000000 0 883 2 0x00000000
[ 709.232267] ffff88006ee736b0 0000000000000046 ffff88006ee73640 ffffffff00000000
[ 709.232500] ffff88006d5d9700 ffff88006ee73fd8 ffff88006ee73fd8 0000000000013b40
[ 709.232516] ffff880071589700 ffff88006d5d9700 ffff8801002543c0 000000016ee73718
[ 709.232532] Call Trace:
[ 709.232542] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e
[ 709.232550] [<ffffffff81217900>] get_request_wait+0x102/0x18b
[ 709.232561] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.232570] [<ffffffff81218641>] __make_request+0x18a/0x2b8
[ 709.232579] [<ffffffff81217338>] generic_make_request+0x2a9/0x323
[ 709.232589] [<ffffffff8147bd19>] ? _raw_read_unlock_irqrestore+0x17/0x19
[ 709.232601] [<ffffffff8138d520>] ? dm_target_exit+0x19/0x19
[ 709.232609] [<ffffffff81217490>] submit_bio+0xde/0xfd
[ 709.232620] [<ffffffff810ea4e1>] ? inc_zone_page_state+0x27/0x29
[ 709.232630] [<ffffffff810dc9fd>] ? account_page_writeback+0x25/0x29
[ 709.232641] [<ffffffff8122e200>] ? radix_tree_gang_lookup_slot+0x66/0x87
[ 709.232656] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58
[ 709.232666] [<ffffffff81199a48>] ext4_bio_write_page+0x17c/0x320
[ 709.232676] [<ffffffff810ea4a1>] ? mod_state+0x76/0x7d
[ 709.232685] [<ffffffff81194afa>] mpage_da_submit_io+0x306/0x389
[ 709.232696] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd
[ 709.232706] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d
[ 709.232716] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a
[ 709.232728] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc
[ 709.232738] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161
[ 709.232748] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b
[ 709.232758] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335
[ 709.232768] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
[ 709.232779] [<ffffffff8147bc3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[ 709.232789] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1
[ 709.232799] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d
[ 709.232808] [<ffffffff8147aac7>] ? schedule_timeout+0xb0/0xde
[ 709.232819] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5
[ 709.232829] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d
[ 709.232839] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 709.232850] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 709.232861] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 709.232872] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 709.232878] abrt-dump-oops D 0000000000000000 0 903 817 0x00000000
[ 709.232891] ffff88006eda1c78 0000000000000082 0000000000000005 0000000000000000
[ 709.232908] ffff880037698000 ffff88006eda1fd8 ffff88006eda1fd8 0000000000013b40
[ 709.232925] ffffffff81a0b020 ffff880037698000 ffff88006eda1c78 000000018106e7c3
[ 709.232942] Call Trace:
[ 709.232951] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465
[ 709.232962] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.232971] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda
[ 709.232979] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15
[ 709.232989] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120
[ 709.233000] [<ffffffff8119ccaa>] ext4_mkdir+0xc9/0x33e
[ 709.233010] [<ffffffff811297a7>] vfs_mkdir+0x5f/0x9b
[ 709.233021] [<ffffffff8112b0ae>] sys_mkdirat+0x97/0xe8
[ 709.233032] [<ffffffff81122b89>] ? sys_newstat+0x2a/0x33
[ 709.233042] [<ffffffff8112b117>] sys_mkdir+0x18/0x1a
[ 709.233051] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b
[ 709.233066] Xorg D 0000000000000000 0 1187 1183 0x00400084
[ 709.233080] ffff88007171ba38 0000000000000082 ffff8800177db2d8 ffff880000000000
[ 709.233097] ffff88006d581700 ffff88007171bfd8 ffff88007171bfd8 0000000000013b40
[ 709.233114] ffffffff81a0b020 ffff88006d581700[ 709.235257] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd
[ 709.235532] .MIN_vruntime : 0.000001
[ 709.235656]
[ 709.236129] .se->load.weight : 2
[ 709.236374] .se->load.weight : 2
[ 709.236431]
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-04-29 18:02 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-04-29 18:02 UTC (permalink / raw)
To: sedat.dilek
Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, Thomas Gleixner
On Fri, 2011-04-29 at 11:31 -0500, James Bottomley wrote:
> On Fri, 2011-04-29 at 10:37 -0500, James Bottomley wrote:
> > On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote:
> > > But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try.
> > > First, I have seen negative effects on my UP-system was when playing
> > > with linux-next [2].
> > > It was not clear what the origin was and the the side-effects were
> > > somehow "bizarre".
> > > The issue could be easily reproduced by tar-ing the kernel build-dir
> > > to an external USB-hdd.
> > > The issue kept RCU and TIP folks really busy.
> > > Before stepping 4 weeks in the dark, give it a try and let me know in
> > > case of success.
> >
> > Well, it's highly unlikely because that's a 2.6.39 artifact and the bug
> > showed up in 2.6.38 ... I tried it just in case with no effect, so we
> > know it isn't the cause.
>
> Actually, I tell a lie: it does't stop kswapd spinning on PREEMPT, but
> it does seem to prevent non-PREEMPT from locking up totally (at least it
> survives three back to back untar runs).
>
> It's probable it alters the memory pin conditions that cause the spin,
> so it's masking the problem rather than fixing it.
Confirmed ... it's just harder to reproduce with the hrtimers init fix.
The problem definitely still exists (I had to load up the system more
before doing the tar).
This time I've caught kswapd in mem_cgroup_shrink_node_zone. sysrq-w
doesn't complete for an unknown reason
James
---
[ 575.083025] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46]
[ 575.083043] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 575.083251] CPU 2
[ 575.083255] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 575.083430]
[ 575.083437] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO
[ 575.083455] RIP: 0010:[<ffffffffa007ff8f>] [<ffffffffa007ff8f>] i915_gem_inactive_shrink+0x6c/0x194 [i915]
[ 575.083497] RSP: 0018:ffff88006dfb7d50 EFLAGS: 00000206
[ 575.083504] RAX: ffff88005f018200 RBX: 00000000000000c0 RCX: 0000000000000000
[ 575.083512] RDX: ffff88005f01bab0 RSI: 0000000000000000 RDI: ffff880037885820
[ 575.083529] RBP: ffff88006dfb7d90 R08: 0000000000000000 R09: 0000000000017131
[ 575.083535] R10: 0000000000000002 R11: ffffffff81a44e50 R12: ffffffff81482e0e
[ 575.083541] R13: ffff88006dfb7cf0 R14: ffff88006dfb7cf8 R15: ffffffff810dd44d
[ 575.083548] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 575.083555] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 575.083572] CR2: 00007f3934e8a000 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 575.083579] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 575.083587] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 575.083596] Process kswapd0 (pid: 46, threadinfo ffff88006dfb6000, task ffff88006dfb8000)
[ 575.083602] Stack:
[ 575.083608] ffff88006dfb7d90 ffff880037709638 ffff88006dfb7d60 ffff8800377095f0
[ 575.083626] 0000000000000000 0000000000000000 00000000000000d0 000000000005103b
[ 575.083643] ffff88006dfb7de0 ffffffff810e1d89 000000000000003d 0000000000000080
[ 575.083660] Call Trace:
[ 575.083676] [<ffffffff810e1d89>] shrink_slab+0x6d/0x166
[ 575.083686] [<ffffffff810e49cc>] kswapd+0x533/0x798
[ 575.083698] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 575.083712] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 575.083726] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 575.083738] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 575.083750] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 575.083756] Code: e4 48 89 45 c8 75 37 48 8b 43 48 45 31 ed 48 83 c3 48 48 2d b0 00 00 00 eb 0a 48 8d 82 50 ff ff ff 41 ff c5 48 8b 90 b0 00 00 00
[ 575.083887] 05 b0 00 00 00 48 39 d8 0f 18 0a 75 e1 e9 da 00 00 00 4c 89
[ 575.083949] Call Trace:
[ 575.083957] [<ffffffff810e1d89>] shrink_slab+0x6d/0x166
[ 575.083966] [<ffffffff810e49cc>] kswapd+0x533/0x798
[ 575.083976] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 575.083986] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 575.083998] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 575.084010] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 575.084030] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 658.883363] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:46]
[ 658.883382] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 658.883574] CPU 2
[ 658.883579] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 658.883749]
[ 658.883757] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO
[ 658.883776] RIP: 0010:[<ffffffff81080bc0>] [<ffffffff81080bc0>] arch_local_irq_restore+0xc/0xd
[ 658.883797] RSP: 0018:ffff88006dfb7d98 EFLAGS: 00000246
[ 658.883805] RAX: ffff8801005f9e68 RBX: ffff8801005e6e00 RCX: 0000000000004ae4
[ 658.883813] RDX: ffff88006dfb8000 RSI: 0000000000000246 RDI: 0000000000000246
[ 658.883821] RBP: ffff88006dfb7da0 R08: ffff8801005f9e70 R09: 00000000000007eb
[ 658.883828] R10: 0000000000000000 R11: 00000000000000cd R12: ffffffff81482e0e
[ 658.883835] R13: 0000000000016fd3 R14: 000000000000a731 R15: ffffffff810ddb31
[ 658.883844] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 658.883853] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 658.883860] CR2: 00007f3934e8a000 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 658.883867] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 658.883875] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 658.883883] Process kswapd0 (pid: 46, threadinfo ffff88006dfb6000, task ffff88006dfb8000)
[ 658.883890] Stack:
[ 658.883895] ffffffff8147bc8c ffff88006dfb7de0 ffffffff8106e7c3 ffff880000000002
[ 658.883913] 0000000000000000 ffff8801005e6000 ffff8801005f9e68 0000000000000002
[ 658.883929] 0000000000000000 ffff88006dfb7ee0 ffffffff810e45f0 0000000000000003
[ 658.883947] Call Trace:
[ 658.883961] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 658.883976] [<ffffffff8106e7c3>] prepare_to_wait+0x6c/0x78
[ 658.883988] [<ffffffff810e45f0>] kswapd+0x157/0x798
[ 658.884000] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 658.884009] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 658.884020] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 658.884033] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 658.884045] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 658.884057] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 658.884062] Code: 1f 44 00 00 66 ff 05 30 16 98 00 fb 66 0f 1f 44 00 00 5d c3 55 48 89 e5 0f 1f 44 00 00 5d c3 55 48 89 e5 57 9d 0f 1f 44 00 00 5d <c3> 55 48 89 e5 fa 66 0f 1f 44 00 00 5d c3 55 48 89 e5 50 9c 58
[ 658.884238] Call Trace:
[ 658.884248] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 658.884260] [<ffffffff8106e7c3>] prepare_to_wait+0x6c/0x78
[ 658.884269] [<ffffffff810e45f0>] kswapd+0x157/0x798
[ 658.884280] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 658.884289] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 658.884299] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 658.884311] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 658.884322] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 658.884333] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 690.767137] SysRq : Show backtrace of all active CPUs
[ 690.767158] sending NMI to all CPUs:
[ 690.767171] NMI backtrace for cpu 2
[ 690.767188] CPU 2
[ 690.767196] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 690.767457]
[ 690.767468] Pid: 46, comm: kswapd0 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO
[ 690.767493] RIP: 0010:[<ffffffff810e3980>] [<ffffffff810e3980>] shrink_zone+0x221/0x4b3
[ 690.767519] RSP: 0018:ffff88006dfb7d30 EFLAGS: 00000206
[ 690.767529] RAX: 0000000000000000 RBX: ffff88006dfb7e58 RCX: 000000000000000c
[ 690.767539] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 690.767550] RBP: ffff88006dfb7de0 R08: 0000000000000001 R09: 00000000000007eb
[ 690.767561] R10: 0000000000000002 R11: 00000000000000f5 R12: ffff8801005e6000
[ 690.767572] R13: ffff8801005e64d8 R14: 0000000000001572 R15: 00000000ffffffff
[ 690.767584] FS: 0000000000000000(0000) GS:ffff880100280000(0000) knlGS:0000000000000000
[ 690.767597] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 690.767607] CR2: 00007f3934e8a000 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 690.767618] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 690.767628] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 690.767640] Process kswapd0 (pid: 46, threadinfo ffff88006dfb6000, task ffff88006dfb8000)
[ 690.767650] Stack:
[ 690.767657] 0000000000000001 000000000000008c 000000000000003c 0000000000000000
[ 690.767683] ffffffffffffffff ffff8801005e6480 000000000000000c 0000000c81118591
[ 690.767717] 0000000000000000 0000000000000000 ffff880100530380 0000000000000000
[ 690.767742] Call Trace:
[ 690.767764] [<ffffffff810d9967>] ? zone_watermark_ok_safe+0xa1/0xae
[ 690.767779] [<ffffffff810e49a6>] kswapd+0x50d/0x798
[ 690.767794] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 690.767811] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 690.767830] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 690.767846] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 690.767861] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 690.767868] Code: e2 ff ff 83 7d 80 00 44 8b 85 50 ff ff ff 74 1b 8a 4d 8c 31 f6 41 83 ff 01 40 0f 96 c6 31 d2 48 d3 e8 48 0f af 44 f5 c0 49 f7 f6
[ 690.768028] 89 c1 48 8d 51 04 49 03 44 d5 00 48 83 f8 1f 49 89 44 d5 00
[ 690.768113] Call Trace:
[ 690.768126] [<ffffffff810d9967>] ? zone_watermark_ok_safe+0xa1/0xae
[ 690.768139] [<ffffffff810e49a6>] kswapd+0x50d/0x798
[ 690.768154] [<ffffffff810e4499>] ? mem_cgroup_shrink_node_zone+0xe3/0xe3
[ 690.768168] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 690.768183] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 690.768199] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 690.768213] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 690.768223] NMI backtrace for cpu 3
[ 690.768236] CPU 3
[ 690.768244] Modules linked in: netconsole configfs fuse sunrpc bluetooth cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_conexant arc4 snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device iwlagn snd_pcm uvcvideo videodev v4l2_compat_ioctl32 snd_timer iTCO_wdt mac80211 snd xhci_hcd i2c_i801 cfg80211 soundcore snd_page_alloc e1000e microcode iTCO_vendor_support wmi pcspkr joydev rfkill uinput ipv6 sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 690.768832]
[ 690.768843] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-rc4+ #6 LENOVO 4170CTO/4170CTO
[ 690.768867] RIP: 0010:[<ffffffff81275b36>] [<ffffffff81275b36>] intel_idle+0xaa/0x100
[ 690.768890] RSP: 0018:ffff8800715dfe68 EFLAGS: 00000046
[ 690.768900] RAX: 0000000000000030 RBX: 0000000000000010 RCX: 0000000000000001
[ 690.768912] RDX: 0000000000000000 RSI: ffff8800715dffd8 RDI: ffffffff81a0e640
[ 690.768923] RBP: ffff8800715dfeb8 R08: 000000000000006d R09: 00000000000003c2
[ 690.768934] R10: ffffffff00000003 R11: ffff8801002d3b40 R12: 0000000000000030
[ 690.768946] R13: 12191803fa451798 R14: 0000000000000004 R15: 0000000000000003
[ 690.768959] FS: 0000000000000000(0000) GS:ffff8801002c0000(0000) knlGS:0000000000000000
[ 690.768971] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 690.768982] CR2: 0000000000429bf0 CR3: 0000000001a03000 CR4: 00000000000406e0
[ 690.768993] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 690.769005] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 690.769017] Process kworker/0:1 (pid: 0, threadinfo ffff8800715de000, task ffff8800715e1700)
[ 690.769027] Stack:
[ 690.769034] ffff8800715dfe88 ffffffff810731c0 ffff8801002d1280 0000000000011280
[ 690.769063] ffff8800715dfeb8 000000038139c77a ffffe8ffffcc0170 ffffe8ffffcc0170
[ 690.769092] ffffe8ffffcc0300 0000000000000000 ffff8800715dfef8 ffffffff8139b668
[ 690.769121] Call Trace:
[ 690.769137] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 690.769153] [<ffffffff8139b668>] cpuidle_idle_call+0xe7/0x166
[ 690.769173] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 690.769191] [<ffffffff8146ac57>] start_secondary+0x223/0x225
[ 690.769203] Code: 28 e0 ff ff 80 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <e8> 23 0b e0 ff 4c 29 e8 48 89 c7 e8 ab 2b de ff 4c 69 e0 40 42
[ 690.769477] Call Trace:
[ 690.769491] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 690.769506] [<ffffffff8139b668>] cpuidle_idle_call+0xe7/0x166
[ 690.769522] [<ffffffff81008321>] cpu_idle+0xa5/0xdf
[ 690.770451] [<ffffffff810731c0>] ? pm_qos_request+0x3e/0x45
[ 690.771380] [<ffffffff81483f73>] ? smp_apic_timer_interrupt+0x7e/0x8c
[ 692.887516] SysRq : Show Blocked State
[ 692.887534] task PC stack pid father
[ 692.887551] fsnotify_mark D 0000000000000000 0 49 2 0x00000000
[ 692.887567] ffff88006dfb1cc0 0000000000000046 0000000000000000 0000000000000000
[ 692.887585] ffff88006dfbc500 ffff88006dfb1fd8 ffff88006dfb1fd8 0000000000013b40
[ 692.887601] ffff880071589700 ffff88006dfbc500 ffff88006dfb1cb0 000000010024f4c0
[ 692.887618] Call Trace:
[ 692.887635] [<ffffffff8147aa4b>] schedule_timeout+0x34/0xde
[ 692.887650] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61
[ 692.887662] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd
[ 692.887673] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 692.887683] [<ffffffff8147a814>] wait_for_common+0xac/0x101
[ 692.887692] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226
[ 692.887702] [<ffffffff810ada0d>] ? synchronize_rcu_bh+0x5c/0x5c
[ 692.887711] [<ffffffff8147a91d>] wait_for_completion+0x1d/0x1f
[ 692.887720] [<ffffffff810ada67>] synchronize_sched+0x5a/0x5c
[ 692.887730] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43
[ 692.887740] [<ffffffff810726fc>] __synchronize_srcu+0x31/0x89
[ 692.887751] [<ffffffff81072780>] synchronize_srcu+0x15/0x17
[ 692.887762] [<ffffffff8114c8be>] fsnotify_mark_destroy+0x90/0x165
[ 692.887774] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.887783] [<ffffffff8114c82e>] ? fsnotify_put_mark+0x1c/0x1c
[ 692.887793] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 692.887806] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 692.887818] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 692.887828] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 692.887840] jbd2/dm-1-8 D 0000000000000000 0 377 2 0x00000000
[ 692.887854] ffff88006d765cd0 0000000000000046 0000000000000000 0000000000000000
[ 692.887871] ffff88006d64dc00 ffff88006d765fd8 ffff88006d765fd8 0000000000013b40
[ 692.887888] ffff880071589700 ffff88006d64dc00 ffff88006d765cd0 000000018106e7c3
[ 692.887905] Call Trace:
[ 692.887917] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194
[ 692.887928] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61
[ 692.887939] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd
[ 692.887949] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 692.887960] [<ffffffff8100804e>] ? load_TLS+0x10/0x14
[ 692.887971] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
[ 692.887982] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.887992] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
[ 692.888003] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 692.888012] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81
[ 692.888023] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a
[ 692.888033] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.888043] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10
[ 692.888053] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 692.888065] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 692.888076] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 692.888087] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 692.888095] flush-253:1 D 0000000000000000 0 439 2 0x00000000
[ 692.888117] ffff88006d70f730 0000000000000046 ffff88006d70f6c0 ffffffff00000000
[ 692.888135] ffff880037b7ae00 ffff88006d70ffd8 ffff88006d70ffd8 0000000000013b40
[ 692.888153] ffff880071589700 ffff880037b7ae00 ffff8801002543c0 000000016d70f798
[ 692.888170] Call Trace:
[ 692.888180] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e
[ 692.888190] [<ffffffff81217900>] get_request_wait+0x102/0x18b
[ 692.888200] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.888209] [<ffffffff81218641>] __make_request+0x18a/0x2b8
[ 692.888219] [<ffffffff81217338>] generic_make_request+0x2a9/0x323
[ 692.888227] [<ffffffff81217490>] submit_bio+0xde/0xfd
[ 692.888238] [<ffffffff81199bd9>] ? ext4_bio_write_page+0x30d/0x320
[ 692.888248] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58
[ 692.888257] [<ffffffff81194b65>] mpage_da_submit_io+0x371/0x389
[ 692.888267] [<ffffffff8119527d>] ? ext4_mark_iloc_dirty+0x4db/0x543
[ 692.888276] [<ffffffff8119552d>] ? ext4_mark_inode_dirty+0x1c5/0x1f0
[ 692.888287] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd
[ 692.888298] [<ffffffff811ab72a>] ? ext4_journal_start_sb+0x108/0x120
[ 692.888308] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d
[ 692.888319] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a
[ 692.888331] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc
[ 692.888341] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161
[ 692.888351] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b
[ 692.888361] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335
[ 692.888371] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 692.888381] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1
[ 692.888392] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d
[ 692.888402] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5
[ 692.888413] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d
[ 692.888423] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 692.888434] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 692.888445] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 692.888455] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 692.888472] jbd2/dm-2-8 D ffff880071617380 0 791 2 0x00000000
[ 692.888485] ffff88006cfb1cd0 0000000000000046 0000000000000000 ffff880000000001
[ 692.888501] ffff88003769ae00 ffff88006cfb1fd8 ffff88006cfb1fd8 0000000000013b40
[ 692.888517] ffff880058cdc500 ffff88003769ae00 ffff88006cfb1cd0 000000018106e7c3
[ 692.888533] Call Trace:
[ 692.888547] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194
[ 692.888559] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.888568] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
[ 692.888578] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 692.888587] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81
[ 692.888597] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a
[ 692.888607] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.888617] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10
[ 692.888627] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 692.888637] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 692.888648] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 692.888659] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 692.888665] NetworkManager D 0000000000000000 0 809 1 0x00000000
[ 692.888682] ffff88006ff23ad8 0000000000000082 ffff880100293b40 0000000000000000
[ 692.888697] ffff88006cfe2e00 ffff88006ff23fd8 ffff88006ff23fd8 0000000000013b40
[ 692.888713] ffffffff81a0b020 ffff88006cfe2e00 ffff88006ff23ad8 000000018106e7c3
[ 692.888729] Call Trace:
[ 692.888738] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465
[ 692.888753] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.888761] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda
[ 692.888770] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15
[ 692.888780] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120
[ 692.888790] [<ffffffff81044796>] ? delayacct_blkio_end+0x21/0x41
[ 692.888799] [<ffffffff8147a5df>] ? io_schedule+0x79/0x7e
[ 692.888808] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c
[ 692.888818] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175
[ 692.888827] [<ffffffff811320c7>] touch_atime+0x10e/0x131
[ 692.888840] [<ffffffff810d7160>] generic_file_aio_read+0x5d9/0x640
[ 692.888851] [<ffffffff8111ecaa>] do_sync_read+0xbf/0xff
[ 692.888860] [<ffffffff8103fd1f>] ? should_resched+0xe/0x2d
[ 692.888872] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33
[ 692.888884] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd
[ 692.888894] [<ffffffff8111f311>] vfs_read+0xa9/0xf0
[ 692.888902] [<ffffffff8111f3a2>] sys_read+0x4a/0x6e
[ 692.888912] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b
[ 692.888923] rs:main Q:Reg D 0000000000000000 0 886 1 0x00000000
[ 692.888935] ffff88006ee35a38 0000000000000082 ffff88010028fa00 ffff880100000000
[ 692.888950] ffff88006ed5dc00 ffff88006ee35fd8 ffff88006ee35fd8 0000000000013b40
[ 692.888966] ffff880071589700 ffff88006ed5dc00 ffff88006ee35a38 000000018106e7c3
[ 692.888986] Call Trace:
[ 692.888994] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465
[ 692.889005] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.889013] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda
[ 692.889022] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15
[ 692.889031] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120
[ 692.889041] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c
[ 692.889050] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175
[ 692.889059] [<ffffffff81131f95>] file_update_time+0xef/0x113
[ 692.889070] [<ffffffff810d69b8>] __generic_file_aio_write+0x15d/0x272
[ 692.889086] [<ffffffff8107cd0e>] ? futex_wait_queue_me+0xc4/0xe0
[ 692.889099] [<ffffffff810d6b2e>] generic_file_aio_write+0x61/0xba
[ 692.889115] [<ffffffff8118fc00>] ext4_file_write+0x1dc/0x234
[ 692.889125] [<ffffffff8111ebab>] do_sync_write+0xbf/0xff
[ 692.889133] [<ffffffff8114b810>] ? fsnotify+0x1ff/0x217
[ 692.889144] [<ffffffff811f1666>] ? selinux_file_permission+0x58/0xb4
[ 692.889155] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33
[ 692.889164] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd
[ 692.889172] [<ffffffff8111f221>] vfs_write+0xac/0xf3
[ 692.889181] [<ffffffff8111f410>] sys_write+0x4a/0x6e
[ 692.889191] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b
[ 692.889199] flush-253:2 D 0000000000000000 0 883 2 0x00000000
[ 692.889211] ffff88006ee736b0 0000000000000046 ffff88006ee73640 ffffffff00000000
[ 692.889446] ffff88006d5d9700 ffff88006ee73fd8 ffff88006ee73fd8 0000000000013b40
[ 692.889462] ffff880071589700 ffff88006d5d9700 ffff8801002543c0 000000016ee73718
[ 692.889478] Call Trace:
[ 692.889488] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e
[ 692.889496] [<ffffffff81217900>] get_request_wait+0x102/0x18b
[ 692.889507] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.889516] [<ffffffff81218641>] __make_request+0x18a/0x2b8
[ 692.889525] [<ffffffff81217338>] generic_make_request+0x2a9/0x323
[ 692.889536] [<ffffffff8147bd19>] ? _raw_read_unlock_irqrestore+0x17/0x19
[ 692.889547] [<ffffffff8138d520>] ? dm_target_exit+0x19/0x19
[ 692.889555] [<ffffffff81217490>] submit_bio+0xde/0xfd
[ 692.889566] [<ffffffff810ea4e1>] ? inc_zone_page_state+0x27/0x29
[ 692.889576] [<ffffffff810dc9fd>] ? account_page_writeback+0x25/0x29
[ 692.889587] [<ffffffff8122e200>] ? radix_tree_gang_lookup_slot+0x66/0x87
[ 692.889597] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58
[ 692.889606] [<ffffffff81199a48>] ext4_bio_write_page+0x17c/0x320
[ 692.889616] [<ffffffff810ea4a1>] ? mod_state+0x76/0x7d
[ 692.889625] [<ffffffff81194afa>] mpage_da_submit_io+0x306/0x389
[ 692.889636] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd
[ 692.889645] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d
[ 692.889656] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a
[ 692.889666] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc
[ 692.889677] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161
[ 692.889687] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b
[ 692.889697] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335
[ 692.889706] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
[ 692.889717] [<ffffffff8147bc3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[ 692.889727] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1
[ 692.889736] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d
[ 692.889746] [<ffffffff8147aac7>] ? schedule_timeout+0xb0/0xde
[ 692.889756] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5
[ 692.889766] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d
[ 692.889776] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 692.889787] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 692.889798] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 692.889809] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 692.889815] abrt-dump-oops D 0000000000000000 0 903 817 0x00000000
[ 692.889827] ffff88006eda1c78 0000000000000082 0000000000000005 0000000000000000
[ 692.889843] ffff880037698000 ffff88006eda1fd8 ffff88006eda1fd8 0000000000013b40
[ 692.889860] ffffffff81a0b020 ffff880037698000 ffff88006eda1c78 000000018106e7c3
[ 692.889876] Call Trace:
[ 692.889884] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465
[ 692.889895] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 692.889904] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda
[ 692.889913] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15
[ 692.889923] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120
[ 692.889934] [<ffffffff8119ccaa>] ext4_mkdir+0xc9/0x33e
[ 692.889944] [<ffffffff811297a7>] vfs_mkdir+0x5f/0x9b
[ 692.889954] [<ffffffff8112b0ae>] sys_mkdirat+0x97/0xe8
[ 692.889965] [<ffffffff81122b89>] ? sys_newstat+0x2a/0x33
[ 692.889975] [<ffffffff8112b117>] sys_mkdir+0x18/0x1a
[ 692.889984] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b
[ 692.889998] Xorg D 0000000000000000 0 1187 1183 0x00400084
[ 692.890012] ffff88007171ba38 0000000000000082 ffff8800177db2d8 ffff880000000000
[ 692.890029] ffff88006d581700 ffff88007171bfd8 ffff88007171bfd8 0000000000013b40
[ 692.890046] ffffffff81a0b020 ffff88006d581700 ffff88007171ba38
nc: Write error: No route to host
[ 709.230547] SysRq : Show Blocked State
[ 709.230567] task PC stack pid father
[ 709.230584] fsnotify_mark D 0000000000000000 0 49 2 0x00000000
[ 709.230600] ffff88006dfb1cc0 0000000000000046 0000000000000000 0000000000000000
[ 709.230621] ffff88006dfbc500 ffff88006dfb1fd8 ffff88006dfb1fd8 0000000000013b40
[ 709.230638] ffff880071589700 ffff88006dfbc500 ffff88006dfb1cb0 000000010024f4c0
[ 709.230664] Call Trace:
[ 709.230683] [<ffffffff8147aa4b>] schedule_timeout+0x34/0xde
[ 709.230697] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61
[ 709.230710] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd
[ 709.230721] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 709.230731] [<ffffffff8147a814>] wait_for_common+0xac/0x101
[ 709.230740] [<ffffffff8104c7af>] ? try_to_wake_up+0x226/0x226
[ 709.230751] [<ffffffff810ada0d>] ? synchronize_rcu_bh+0x5c/0x5c
[ 709.230761] [<ffffffff8147a91d>] wait_for_completion+0x1d/0x1f
[ 709.230769] [<ffffffff810ada67>] synchronize_sched+0x5a/0x5c
[ 709.230780] [<ffffffff8106bdb8>] ? find_ge_pid+0x43/0x43
[ 709.230790] [<ffffffff810726fc>] __synchronize_srcu+0x31/0x89
[ 709.230800] [<ffffffff81072780>] synchronize_srcu+0x15/0x17
[ 709.230810] [<ffffffff8114c8be>] fsnotify_mark_destroy+0x90/0x165
[ 709.230822] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.230831] [<ffffffff8114c82e>] ? fsnotify_put_mark+0x1c/0x1c
[ 709.230842] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 709.230854] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 709.230866] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 709.230876] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 709.230889] jbd2/dm-1-8 D 0000000000000000 0 377 2 0x00000000
[ 709.230902] ffff88006d765cd0 0000000000000046 0000000000000000 0000000000000000
[ 709.230920] ffff88006d64dc00 ffff88006d765fd8 ffff88006d765fd8 0000000000013b40
[ 709.230937] ffff880071589700 ffff88006d64dc00 ffff88006d765cd0 000000018106e7c3
[ 709.230955] Call Trace:
[ 709.230967] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194
[ 709.230978] [<ffffffff8104480b>] ? perf_event_task_sched_out+0x55/0x61
[ 709.230989] [<ffffffff8100eb84>] ? sched_clock+0x9/0xd
[ 709.230999] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 709.231011] [<ffffffff8100804e>] ? load_TLS+0x10/0x14
[ 709.231021] [<ffffffff81008714>] ? __switch_to+0xc6/0x220
[ 709.231033] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.231043] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
[ 709.231055] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 709.231064] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81
[ 709.231075] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a
[ 709.231086] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.231096] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10
[ 709.231106] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 709.231117] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 709.231129] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 709.231139] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 709.231147] flush-253:1 D 0000000000000000 0 439 2 0x00000000
[ 709.231161] ffff88006d70f730 0000000000000046 ffff88006d70f6c0 ffffffff00000000
[ 709.231178] ffff880037b7ae00 ffff88006d70ffd8 ffff88006d70ffd8 0000000000013b40
[ 709.231195] ffff880071589700 ffff880037b7ae00 ffff8801002543c0 000000016d70f798
[ 709.231220] Call Trace:
[ 709.231230] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e
[ 709.231240] [<ffffffff81217900>] get_request_wait+0x102/0x18b
[ 709.231251] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.231260] [<ffffffff81218641>] __make_request+0x18a/0x2b8
[ 709.231270] [<ffffffff81217338>] generic_make_request+0x2a9/0x323
[ 709.231279] [<ffffffff81217490>] submit_bio+0xde/0xfd
[ 709.231290] [<ffffffff81199bd9>] ? ext4_bio_write_page+0x30d/0x320
[ 709.231300] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58
[ 709.231309] [<ffffffff81194b65>] mpage_da_submit_io+0x371/0x389
[ 709.231318] [<ffffffff8119527d>] ? ext4_mark_iloc_dirty+0x4db/0x543
[ 709.231328] [<ffffffff8119552d>] ? ext4_mark_inode_dirty+0x1c5/0x1f0
[ 709.231338] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd
[ 709.231350] [<ffffffff811ab72a>] ? ext4_journal_start_sb+0x108/0x120
[ 709.231360] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d
[ 709.231372] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a
[ 709.231383] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc
[ 709.231393] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161
[ 709.231404] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b
[ 709.231414] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335
[ 709.231424] [<ffffffff810736dc>] ? sched_clock_cpu+0x42/0xc6
[ 709.231434] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1
[ 709.231445] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d
[ 709.231455] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5
[ 709.231465] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d
[ 709.231475] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 709.231486] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 709.231497] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 709.231513] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 709.231521] jbd2/dm-2-8 D ffff880071617380 0 791 2 0x00000000
[ 709.231534] ffff88006cfb1cd0 0000000000000046 0000000000000000 ffff880000000001
[ 709.231550] ffff88003769ae00 ffff88006cfb1fd8 ffff88006cfb1fd8 0000000000013b40
[ 709.231567] ffff880058cdc500 ffff88003769ae00 ffff88006cfb1cd0 000000018106e7c3
[ 709.231587] Call Trace:
[ 709.231597] [<ffffffff811c8b44>] jbd2_journal_commit_transaction+0x1c4/0x1194
[ 709.231609] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.231618] [<ffffffff8105ffef>] ? lock_timer_base+0x2c/0x52
[ 709.231628] [<ffffffff8147bc8c>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[ 709.231638] [<ffffffff81060088>] ? try_to_del_timer_sync+0x73/0x81
[ 709.231652] [<ffffffff811cd1b6>] kjournald2+0xc9/0x20a
[ 709.231664] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.231674] [<ffffffff811cd0ed>] ? commit_timeout+0x10/0x10
[ 709.231684] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 709.231695] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 709.231706] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 709.231716] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 709.231727] NetworkManager D 0000000000000000 0 809 1 0x00000000
[ 709.231739] ffff88006ff23ad8 0000000000000082 ffff880100293b40 0000000000000000
[ 709.231755] ffff88006cfe2e00 ffff88006ff23fd8 ffff88006ff23fd8 0000000000013b40
[ 709.231772] ffffffff81a0b020 ffff88006cfe2e00 ffff88006ff23ad8 000000018106e7c3
[ 709.231787] Call Trace:
[ 709.231801] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465
[ 709.231812] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.231821] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda
[ 709.231829] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15
[ 709.231839] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120
[ 709.231850] [<ffffffff81044796>] ? delayacct_blkio_end+0x21/0x41
[ 709.231859] [<ffffffff8147a5df>] ? io_schedule+0x79/0x7e
[ 709.231868] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c
[ 709.231878] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175
[ 709.231887] [<ffffffff811320c7>] touch_atime+0x10e/0x131
[ 709.231899] [<ffffffff810d7160>] generic_file_aio_read+0x5d9/0x640
[ 709.231911] [<ffffffff8111ecaa>] do_sync_read+0xbf/0xff
[ 709.231919] [<ffffffff8103fd1f>] ? should_resched+0xe/0x2d
[ 709.231935] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33
[ 709.231944] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd
[ 709.231953] [<ffffffff8111f311>] vfs_read+0xa9/0xf0
[ 709.231962] [<ffffffff8111f3a2>] sys_read+0x4a/0x6e
[ 709.231972] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b
[ 709.231983] rs:main Q:Reg D 0000000000000000 0 886 1 0x00000000
[ 709.231995] ffff88006ee35a38 0000000000000082 ffff88010028fa00 ffff880100000000
[ 709.232011] ffff88006ed5dc00 ffff88006ee35fd8 ffff88006ee35fd8 0000000000013b40
[ 709.232027] ffff880071589700 ffff88006ed5dc00 ffff88006ee35a38 000000018106e7c3
[ 709.232047] Call Trace:
[ 709.232055] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465
[ 709.232066] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.232075] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda
[ 709.232083] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15
[ 709.232093] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120
[ 709.232102] [<ffffffff81198fb6>] ext4_dirty_inode+0x1d/0x4c
[ 709.232112] [<ffffffff8113c1d6>] __mark_inode_dirty+0x2f/0x175
[ 709.232121] [<ffffffff81131f95>] file_update_time+0xef/0x113
[ 709.232136] [<ffffffff810d69b8>] __generic_file_aio_write+0x15d/0x272
[ 709.232148] [<ffffffff8107cd0e>] ? futex_wait_queue_me+0xc4/0xe0
[ 709.232159] [<ffffffff810d6b2e>] generic_file_aio_write+0x61/0xba
[ 709.232171] [<ffffffff8118fc00>] ext4_file_write+0x1dc/0x234
[ 709.232181] [<ffffffff8111ebab>] do_sync_write+0xbf/0xff
[ 709.232189] [<ffffffff8114b810>] ? fsnotify+0x1ff/0x217
[ 709.232200] [<ffffffff811f1666>] ? selinux_file_permission+0x58/0xb4
[ 709.232210] [<ffffffff811e9afe>] ? security_file_permission+0x2e/0x33
[ 709.232219] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd
[ 709.232228] [<ffffffff8111f221>] vfs_write+0xac/0xf3
[ 709.232237] [<ffffffff8111f410>] sys_write+0x4a/0x6e
[ 709.232246] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b
[ 709.232255] flush-253:2 D 0000000000000000 0 883 2 0x00000000
[ 709.232267] ffff88006ee736b0 0000000000000046 ffff88006ee73640 ffffffff00000000
[ 709.232500] ffff88006d5d9700 ffff88006ee73fd8 ffff88006ee73fd8 0000000000013b40
[ 709.232516] ffff880071589700 ffff88006d5d9700 ffff8801002543c0 000000016ee73718
[ 709.232532] Call Trace:
[ 709.232542] [<ffffffff8147a5c9>] io_schedule+0x63/0x7e
[ 709.232550] [<ffffffff81217900>] get_request_wait+0x102/0x18b
[ 709.232561] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.232570] [<ffffffff81218641>] __make_request+0x18a/0x2b8
[ 709.232579] [<ffffffff81217338>] generic_make_request+0x2a9/0x323
[ 709.232589] [<ffffffff8147bd19>] ? _raw_read_unlock_irqrestore+0x17/0x19
[ 709.232601] [<ffffffff8138d520>] ? dm_target_exit+0x19/0x19
[ 709.232609] [<ffffffff81217490>] submit_bio+0xde/0xfd
[ 709.232620] [<ffffffff810ea4e1>] ? inc_zone_page_state+0x27/0x29
[ 709.232630] [<ffffffff810dc9fd>] ? account_page_writeback+0x25/0x29
[ 709.232641] [<ffffffff8122e200>] ? radix_tree_gang_lookup_slot+0x66/0x87
[ 709.232656] [<ffffffff811998a0>] ext4_io_submit+0x2c/0x58
[ 709.232666] [<ffffffff81199a48>] ext4_bio_write_page+0x17c/0x320
[ 709.232676] [<ffffffff810ea4a1>] ? mod_state+0x76/0x7d
[ 709.232685] [<ffffffff81194afa>] mpage_da_submit_io+0x306/0x389
[ 709.232696] [<ffffffff8119850c>] mpage_da_map_and_submit+0x2b7/0x2cd
[ 709.232706] [<ffffffff81198be5>] ext4_da_writepages+0x2c1/0x44d
[ 709.232716] [<ffffffff810ddcb4>] do_writepages+0x21/0x2a
[ 709.232728] [<ffffffff8113c9b7>] writeback_single_inode+0xb2/0x1bc
[ 709.232738] [<ffffffff8113cd03>] writeback_sb_inodes+0xcd/0x161
[ 709.232748] [<ffffffff8113d207>] writeback_inodes_wb+0x119/0x12b
[ 709.232758] [<ffffffff8113d407>] wb_writeback+0x1ee/0x335
[ 709.232768] [<ffffffff81080be3>] ? arch_local_irq_save+0x15/0x1b
[ 709.232779] [<ffffffff8147bc3a>] ? _raw_spin_lock_irqsave+0x12/0x2f
[ 709.232789] [<ffffffff810dd59a>] ? global_dirty_limits+0x2b/0xd1
[ 709.232799] [<ffffffff8113d6cd>] wb_do_writeback+0x17f/0x19d
[ 709.232808] [<ffffffff8147aac7>] ? schedule_timeout+0xb0/0xde
[ 709.232819] [<ffffffff8113d773>] bdi_writeback_thread+0x88/0x1e5
[ 709.232829] [<ffffffff8113d6eb>] ? wb_do_writeback+0x19d/0x19d
[ 709.232839] [<ffffffff8106e157>] kthread+0x84/0x8c
[ 709.232850] [<ffffffff81483564>] kernel_thread_helper+0x4/0x10
[ 709.232861] [<ffffffff8106e0d3>] ? kthread_worker_fn+0x148/0x148
[ 709.232872] [<ffffffff81483560>] ? gs_change+0x13/0x13
[ 709.232878] abrt-dump-oops D 0000000000000000 0 903 817 0x00000000
[ 709.232891] ffff88006eda1c78 0000000000000082 0000000000000005 0000000000000000
[ 709.232908] ffff880037698000 ffff88006eda1fd8 ffff88006eda1fd8 0000000000013b40
[ 709.232925] ffffffff81a0b020 ffff880037698000 ffff88006eda1c78 000000018106e7c3
[ 709.232942] Call Trace:
[ 709.232951] [<ffffffff811c6dd5>] start_this_handle+0x2e8/0x465
[ 709.232962] [<ffffffff8106e84e>] ? remove_wait_queue+0x3a/0x3a
[ 709.232971] [<ffffffff811c7218>] jbd2__journal_start+0x94/0xda
[ 709.232979] [<ffffffff811c7271>] jbd2_journal_start+0x13/0x15
[ 709.232989] [<ffffffff811ab72a>] ext4_journal_start_sb+0x108/0x120
[ 709.233000] [<ffffffff8119ccaa>] ext4_mkdir+0xc9/0x33e
[ 709.233010] [<ffffffff811297a7>] vfs_mkdir+0x5f/0x9b
[ 709.233021] [<ffffffff8112b0ae>] sys_mkdirat+0x97/0xe8
[ 709.233032] [<ffffffff81122b89>] ? sys_newstat+0x2a/0x33
[ 709.233042] [<ffffffff8112b117>] sys_mkdir+0x18/0x1a
[ 709.233051] [<ffffffff81482442>] system_call_fastpath+0x16/0x1b
[ 709.233066] Xorg D 0000000000000000 0 1187 1183 0x00400084
[ 709.233080] ffff88007171ba38 0000000000000082 ffff8800177db2d8 ffff880000000000
[ 709.233097] ffff88006d581700 ffff88007171bfd8 ffff88007171bfd8 0000000000013b40
[ 709.233114] ffffffff81a0b020 ffff88006d581700[ 709.235257] [<ffffffff8111ef96>] ? rw_verify_area+0xb0/0xcd
[ 709.235532] .MIN_vruntime : 0.000001
[ 709.235656]
[ 709.236129] .se->load.weight : 2
[ 709.236374] .se->load.weight : 2
[ 709.236431]
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-29 18:02 ` James Bottomley
@ 2011-05-02 20:04 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-02 20:04 UTC (permalink / raw)
To: sedat.dilek
Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, Thomas Gleixner
On Fri, 2011-04-29 at 13:02 -0500, James Bottomley wrote:
> On Fri, 2011-04-29 at 11:31 -0500, James Bottomley wrote:
> > On Fri, 2011-04-29 at 10:37 -0500, James Bottomley wrote:
> > > On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote:
> > > > But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try.
> > > > First, I have seen negative effects on my UP-system was when playing
> > > > with linux-next [2].
> > > > It was not clear what the origin was and the the side-effects were
> > > > somehow "bizarre".
> > > > The issue could be easily reproduced by tar-ing the kernel build-dir
> > > > to an external USB-hdd.
> > > > The issue kept RCU and TIP folks really busy.
> > > > Before stepping 4 weeks in the dark, give it a try and let me know in
> > > > case of success.
> > >
> > > Well, it's highly unlikely because that's a 2.6.39 artifact and the bug
> > > showed up in 2.6.38 ... I tried it just in case with no effect, so we
> > > know it isn't the cause.
> >
> > Actually, I tell a lie: it does't stop kswapd spinning on PREEMPT, but
> > it does seem to prevent non-PREEMPT from locking up totally (at least it
> > survives three back to back untar runs).
> >
> > It's probable it alters the memory pin conditions that cause the spin,
> > so it's masking the problem rather than fixing it.
>
> Confirmed ... it's just harder to reproduce with the hrtimers init fix.
> The problem definitely still exists (I had to load up the system more
> before doing the tar).
>
> This time I've caught kswapd in mem_cgroup_shrink_node_zone. sysrq-w
> doesn't complete for an unknown reason
As a follow on to this, there's a shrink_zone(0, ...) in the cgroup
path. This causes it to scan all memory exhaustively (generating quite
a lot of work). The comment above it implies it's some type of hack for
cgroup accounting, but reducing it to DEF_PRIORITY makes the hang go
away (verified on both 2.6.39-rc4 and 2.6.38.4). Note that I still get
soft lockups in kswapd0, but they no longer hang the box. I'll prepare
a patch for the next round of debate.
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-02 20:04 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-02 20:04 UTC (permalink / raw)
To: sedat.dilek
Cc: Chris Mason, linux-fsdevel, linux-mm, linux-kernel, Thomas Gleixner
On Fri, 2011-04-29 at 13:02 -0500, James Bottomley wrote:
> On Fri, 2011-04-29 at 11:31 -0500, James Bottomley wrote:
> > On Fri, 2011-04-29 at 10:37 -0500, James Bottomley wrote:
> > > On Fri, 2011-04-29 at 12:23 +0200, Sedat Dilek wrote:
> > > > But as I see these RCU (CPU) stalls, the patch from [1] might be worth a try.
> > > > First, I have seen negative effects on my UP-system was when playing
> > > > with linux-next [2].
> > > > It was not clear what the origin was and the the side-effects were
> > > > somehow "bizarre".
> > > > The issue could be easily reproduced by tar-ing the kernel build-dir
> > > > to an external USB-hdd.
> > > > The issue kept RCU and TIP folks really busy.
> > > > Before stepping 4 weeks in the dark, give it a try and let me know in
> > > > case of success.
> > >
> > > Well, it's highly unlikely because that's a 2.6.39 artifact and the bug
> > > showed up in 2.6.38 ... I tried it just in case with no effect, so we
> > > know it isn't the cause.
> >
> > Actually, I tell a lie: it does't stop kswapd spinning on PREEMPT, but
> > it does seem to prevent non-PREEMPT from locking up totally (at least it
> > survives three back to back untar runs).
> >
> > It's probable it alters the memory pin conditions that cause the spin,
> > so it's masking the problem rather than fixing it.
>
> Confirmed ... it's just harder to reproduce with the hrtimers init fix.
> The problem definitely still exists (I had to load up the system more
> before doing the tar).
>
> This time I've caught kswapd in mem_cgroup_shrink_node_zone. sysrq-w
> doesn't complete for an unknown reason
As a follow on to this, there's a shrink_zone(0, ...) in the cgroup
path. This causes it to scan all memory exhaustively (generating quite
a lot of work). The comment above it implies it's some type of hack for
cgroup accounting, but reducing it to DEF_PRIORITY makes the hang go
away (verified on both 2.6.39-rc4 and 2.6.38.4). Note that I still get
soft lockups in kswapd0, but they no longer hang the box. I'll prepare
a patch for the next round of debate.
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 22:43 ` James Bottomley
@ 2011-05-03 9:13 ` Mel Gorman
-1 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-05-03 9:13 UTC (permalink / raw)
To: James Bottomley
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Thu, Apr 28, 2011 at 05:43:48PM -0500, James Bottomley wrote:
> On Thu, 2011-04-28 at 16:12 -0500, James Bottomley wrote:
> > On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote:
> > > Actually, talking to Chris, I think I can get the system up using
> > > init=/bin/bash without systemd, so I can try the no cgroup config.
> >
> > OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back
> > runs of untar without locking or getting kswapd pegged, so I'm pretty
> > certain this is cgroups related. The next steps are to turn cgroups
> > back on but try disabling the memory and IO controllers.
>
> I tried non-PREEMPT CGROUP but disabled GROUP_MEM_RES_CTLR.
>
> The results are curious: the tar does complete (I've done three back to
> back). However, I did get one soft lockup in kswapd (below). But the
> system recovers instead of halting I/O and hanging like it did
> previously.
>
> The soft lockup is in shrink_slab, so perhaps it's a combination of slab
> shrinker and cgroup memory controller issues?
>
So, kswapd is still looping in reclaim and spending a lot of time in
shrink_slab but it must not be the shrinker itself or that debug patch
would have triggered. It's curious that cgroups are involved with
systemd considering that one would expect those groups to be fairly
small. I still don't have a new theory but will get hold of a Fedora 15
install CD and see can I reproduce it locally.
One last thing, what is the value of /proc/sys/vm/zone_reclaim_mode? Two
of the reporting machines could be NUMA and if that proc file reads as
1, I'd be interested in hearing the results of a test with it set to 0.
Thanks.
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-03 9:13 ` Mel Gorman
0 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-05-03 9:13 UTC (permalink / raw)
To: James Bottomley
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Thu, Apr 28, 2011 at 05:43:48PM -0500, James Bottomley wrote:
> On Thu, 2011-04-28 at 16:12 -0500, James Bottomley wrote:
> > On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote:
> > > Actually, talking to Chris, I think I can get the system up using
> > > init=/bin/bash without systemd, so I can try the no cgroup config.
> >
> > OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back
> > runs of untar without locking or getting kswapd pegged, so I'm pretty
> > certain this is cgroups related. The next steps are to turn cgroups
> > back on but try disabling the memory and IO controllers.
>
> I tried non-PREEMPT CGROUP but disabled GROUP_MEM_RES_CTLR.
>
> The results are curious: the tar does complete (I've done three back to
> back). However, I did get one soft lockup in kswapd (below). But the
> system recovers instead of halting I/O and hanging like it did
> previously.
>
> The soft lockup is in shrink_slab, so perhaps it's a combination of slab
> shrinker and cgroup memory controller issues?
>
So, kswapd is still looping in reclaim and spending a lot of time in
shrink_slab but it must not be the shrinker itself or that debug patch
would have triggered. It's curious that cgroups are involved with
systemd considering that one would expect those groups to be fairly
small. I still don't have a new theory but will get hold of a Fedora 15
install CD and see can I reproduce it locally.
One last thing, what is the value of /proc/sys/vm/zone_reclaim_mode? Two
of the reporting machines could be NUMA and if that proc file reads as
1, I'd be interested in hearing the results of a test with it set to 0.
Thanks.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 17:18 ` Mel Gorman
@ 2011-05-03 9:54 ` Colin Ian King
-1 siblings, 0 replies; 138+ messages in thread
From: Colin Ian King @ 2011-05-03 9:54 UTC (permalink / raw)
To: Mel Gorman
Cc: James Bottomley, Jan Kara, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 18:18 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote:
> > On Thu, 2011-04-28 at 11:50 -0500, James Bottomley wrote:
> > > This is the output of perf record -g -a -f sleep 5
> > >
> > > (hopefully the list won't choke)
> >
> > Um, this one actually shows kswapd
> >
> > James
> >
> > ---
> >
> > # Events: 6K cycles
> > #
> > # Overhead Command Shared Object Symbol
> > # ........ ........... ................... .......................................
> > #
> > 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab
> > |
> > --- shrink_slab
> > |
> > |--99.91%-- kswapd
> > | kthread
> > | kernel_thread_helper
> > --0.09%-- [...]
> >
>
> Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use
> compaction instead of lumpy reclaim" is related unless we are seeing
> two problems that happen to manifest in a similar manner.
That is a distinct possibility.
>
> However, there were a number of changes made to dcache in particular
> for 2.6.38. Specifically thinks like dentry_kill use trylock and is
> happy to loop around if it fails to acquire anything. See things like
> this for example;
>
[ text deleted ]
> Way hey, cgroups are also in the mix. How jolly.
>
> Is systemd a common element of the machines hitting this bug by any
> chance?
Not in my case, using upstart on my machine.
>
Colin
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-03 9:54 ` Colin Ian King
0 siblings, 0 replies; 138+ messages in thread
From: Colin Ian King @ 2011-05-03 9:54 UTC (permalink / raw)
To: Mel Gorman
Cc: James Bottomley, Jan Kara, Chris Mason, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Thu, 2011-04-28 at 18:18 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 11:56:17AM -0500, James Bottomley wrote:
> > On Thu, 2011-04-28 at 11:50 -0500, James Bottomley wrote:
> > > This is the output of perf record -g -a -f sleep 5
> > >
> > > (hopefully the list won't choke)
> >
> > Um, this one actually shows kswapd
> >
> > James
> >
> > ---
> >
> > # Events: 6K cycles
> > #
> > # Overhead Command Shared Object Symbol
> > # ........ ........... ................... .......................................
> > #
> > 20.41% kswapd0 [kernel.kallsyms] [k] shrink_slab
> > |
> > --- shrink_slab
> > |
> > |--99.91%-- kswapd
> > | kthread
> > | kernel_thread_helper
> > --0.09%-- [...]
> >
>
> Ok. I can't see how the patch "mm: vmscan: reclaim order-0 and use
> compaction instead of lumpy reclaim" is related unless we are seeing
> two problems that happen to manifest in a similar manner.
That is a distinct possibility.
>
> However, there were a number of changes made to dcache in particular
> for 2.6.38. Specifically thinks like dentry_kill use trylock and is
> happy to loop around if it fails to acquire anything. See things like
> this for example;
>
[ text deleted ]
> Way hey, cgroups are also in the mix. How jolly.
>
> Is systemd a common element of the machines hitting this bug by any
> chance?
Not in my case, using upstart on my machine.
>
Colin
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-03 9:13 ` Mel Gorman
@ 2011-05-03 14:13 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-03 14:13 UTC (permalink / raw)
To: Mel Gorman
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Tue, 2011-05-03 at 10:13 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 05:43:48PM -0500, James Bottomley wrote:
> > On Thu, 2011-04-28 at 16:12 -0500, James Bottomley wrote:
> > > On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote:
> > > > Actually, talking to Chris, I think I can get the system up using
> > > > init=/bin/bash without systemd, so I can try the no cgroup config.
> > >
> > > OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back
> > > runs of untar without locking or getting kswapd pegged, so I'm pretty
> > > certain this is cgroups related. The next steps are to turn cgroups
> > > back on but try disabling the memory and IO controllers.
> >
> > I tried non-PREEMPT CGROUP but disabled GROUP_MEM_RES_CTLR.
> >
> > The results are curious: the tar does complete (I've done three back to
> > back). However, I did get one soft lockup in kswapd (below). But the
> > system recovers instead of halting I/O and hanging like it did
> > previously.
> >
> > The soft lockup is in shrink_slab, so perhaps it's a combination of slab
> > shrinker and cgroup memory controller issues?
> >
>
> So, kswapd is still looping in reclaim and spending a lot of time in
> shrink_slab but it must not be the shrinker itself or that debug patch
> would have triggered. It's curious that cgroups are involved with
> systemd considering that one would expect those groups to be fairly
> small. I still don't have a new theory but will get hold of a Fedora 15
> install CD and see can I reproduce it locally.
I've got a ftrace output of kswapd ... it's 500k compressed, so I'll
send under separate cover.
> One last thing, what is the value of /proc/sys/vm/zone_reclaim_mode? Two
> of the reporting machines could be NUMA and if that proc file reads as
> 1, I'd be interested in hearing the results of a test with it set to 0.
> Thanks.
It's zero, I'm afraid
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-03 14:13 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-03 14:13 UTC (permalink / raw)
To: Mel Gorman
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Tue, 2011-05-03 at 10:13 +0100, Mel Gorman wrote:
> On Thu, Apr 28, 2011 at 05:43:48PM -0500, James Bottomley wrote:
> > On Thu, 2011-04-28 at 16:12 -0500, James Bottomley wrote:
> > > On Thu, 2011-04-28 at 14:59 -0500, James Bottomley wrote:
> > > > Actually, talking to Chris, I think I can get the system up using
> > > > init=/bin/bash without systemd, so I can try the no cgroup config.
> > >
> > > OK, so a non-PREEMPT non-CGROUP kernel has survived three back to back
> > > runs of untar without locking or getting kswapd pegged, so I'm pretty
> > > certain this is cgroups related. The next steps are to turn cgroups
> > > back on but try disabling the memory and IO controllers.
> >
> > I tried non-PREEMPT CGROUP but disabled GROUP_MEM_RES_CTLR.
> >
> > The results are curious: the tar does complete (I've done three back to
> > back). However, I did get one soft lockup in kswapd (below). But the
> > system recovers instead of halting I/O and hanging like it did
> > previously.
> >
> > The soft lockup is in shrink_slab, so perhaps it's a combination of slab
> > shrinker and cgroup memory controller issues?
> >
>
> So, kswapd is still looping in reclaim and spending a lot of time in
> shrink_slab but it must not be the shrinker itself or that debug patch
> would have triggered. It's curious that cgroups are involved with
> systemd considering that one would expect those groups to be fairly
> small. I still don't have a new theory but will get hold of a Fedora 15
> install CD and see can I reproduce it locally.
I've got a ftrace output of kswapd ... it's 500k compressed, so I'll
send under separate cover.
> One last thing, what is the value of /proc/sys/vm/zone_reclaim_mode? Two
> of the reporting machines could be NUMA and if that proc file reads as
> 1, I'd be interested in hearing the results of a test with it set to 0.
> Thanks.
It's zero, I'm afraid
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-03 14:13 ` James Bottomley
(?)
@ 2011-05-03 14:22 ` James Bottomley
2011-05-06 7:42 ` Mel Gorman
-1 siblings, 1 reply; 138+ messages in thread
From: James Bottomley @ 2011-05-03 14:22 UTC (permalink / raw)
To: Mel Gorman
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
[-- Attachment #1: Type: text/plain, Size: 238 bytes --]
On Tue, 2011-05-03 at 09:13 -0500, James Bottomley wrote:
> I've got a ftrace output of kswapd ... it's 500k compressed, so I'll
> send under separate cover.
Here it is ... it's under 2.6.38.4 vanilla, but the code is similar.
James
[-- Attachment #2: kswapd-ftrace-2.6.38.4.txt.gz --]
[-- Type: application/x-gzip, Size: 545984 bytes --]
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-04-28 22:40 ` Jan Kara
@ 2011-05-03 18:55 ` Colin Ian King
-1 siblings, 0 replies; 138+ messages in thread
From: Colin Ian King @ 2011-05-03 18:55 UTC (permalink / raw)
To: Jan Kara
Cc: Chris Mason, James Bottomley, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Fri, 2011-04-29 at 00:40 +0200, Jan Kara wrote:
> On Thu 28-04-11 15:58:21, Colin Ian King wrote:
> > On Thu, 2011-04-28 at 16:33 +0200, Jan Kara wrote:
> > > On Thu 28-04-11 16:25:51, Jan Kara wrote:
> > > > On Thu 28-04-11 15:01:22, Colin Ian King wrote:
> > > > >
> > > > > > Could you post the soft lockups you're seeing?
> > > > >
> > > > > As requested, attached
> > > > Hum, what keeps puzzling me is that in all the cases of hangs I've seen
> > > > so far, we are stuck waiting for IO to finish for a long time - e.g. in the
> > > > traces below kjournald waits for PageWriteback bit to get cleared. Also we
> > > > are stuck waiting for page locks which might be because those pages are
> > > > being read in? All in all it seems that the IO is just incredibly slow.
> > > >
> > > > But it's not clear to me what pushes us into that situation (especially
> > > > since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the
> > > > underlying blocks are not already allocated.
> > > Hmm, maybe because the system is under memory pressure (and kswapd is not
> > > able to get rid of dirty pages), we page out clean pages. Thus also pages
> > > of executables which need to be paged in soon anyway thus putting heavy
> > > read load on the system which makes writes crawl? I'm not sure why
> > > compaction should make this any worse but maybe it can.
> > >
> > > James, Colin, can you capture output of 'vmstat 1' while you do the
> > > copying? Thanks.
> >
> > Attached.
> Thanks. So I there are a few interesting points in the vmstat output:
> For first 30 seconds, we are happily copying data - relatively steady read
> throughput (about 20-40 MB/s) and occasional peak from flusher thread
> flushing dirty data. During this time free memory drops from about 1.4 GB
> to about 22!!! MB - mm seems to like to really use the machine ;). Then
> things get interesting:
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 0 1 0 83372 5228 1450776 0 0 39684 90132 450 918 0 4 74 22
> 0 1 0 22416 5228 1509864 0 0 29452 48492 403 869 1 2 80 18
> 2 0 0 20056 5384 1513996 0 0 2248 2116 434 1191 4 4 71 21
> 0 1 0 19800 5932 1514600 0 0 644 104 454 1166 8 3 64 24
> 1 0 0 21848 5940 1515244 0 0 292 380 468 1775 15 6 75 3
> 1 0 0 20936 5940 1515876 0 0 296 296 496 1846 18 8 74 0
> 1 0 0 17792 5940 1516580 0 0 356 356 484 1862 23 8 69 0
> 1 0 0 17544 5940 1517364 0 0 412 412 482 1812 16 7 77 0
> 4 0 0 18148 5948 1517968 0 0 288 344 436 1749 19 9 69 3
> 0 2 220 137528 1616 1402468 0 220 74708 2164 849 1806 3 6 63 28
> 0 3 224 36184 1628 1499648 0 4 50820 86204 532 1272 0 4 64 32
> 0 2 19680 53688 1628 1484388 32 19456 6080 62972 242 287 0 2 63 34
> 0 2 36928 1407432 1356 150980 0 17252 1564 17276 368 764 1 3 73 22
>
> So when free memory reached about 20 MB, both read and write activity
> almost stalled for 7 s (probably everybody waits for free memory). Then
> mm manages to free 100 MB from page cache, things move on for two seconds,
> then we swap out! about 36 MB and page reclaim also finally decides it
> maybe has too much of page cache and reaps most of it (1.3 GB in one go).
> Then things get going again, although there are still occasional stalls
> such as this (about 30s later):
> 1 3 36688 753192 1208 792344 0 0 35948 32768 435 6625 0 6 61 33
> 0 2 36668 754996 1344 792760 0 0 252 58736 2832 16239 0 1 60 39
> 0 2 36668 750132 1388 796688 0 0 2508 1524 325 959 1 3 68 28
> 1 0 36668 751160 1400 797968 0 0 620 620 460 1470 6 6 50 38
> 1 0 36668 750516 1400 798520 0 0 300 300 412 1764 17 8 75 1
> 1 0 36668 750648 1408 799108 0 0 280 340 423 1816 18 6 73 3
> 1 0 36668 748856 1408 799752 0 0 336 328 409 1788 18 8 75 0
> 1 0 36668 748120 1416 800604 0 0 428 452 407 1723 14 10 75 2
> 1 0 36668 750048 1416 801176 0 0 296 296 405 1779 18 7 75 1
> 0 1 36668 650428 1420 897252 0 0 48100 556 658 1718 10 3 71 15
> 0 2 36668 505444 1424 1037012 0 0 69888 90272 686 1491 1 4 68 27
> 0 1 36668 479264 1428 1063372 0 0 12984 40896 324 674 1 1 76 23
> ...
> I'm not sure what we were blocked on here since there is still plenty of
> free memory (750 MB). These stalls repeat once in a while but things go on.
> Then at about 350s, things just stop:
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 3 1 75280 73564 1844 1503848 0 0 43396 81976 627 1061 0 25 42 32
> 3 3 75280 73344 1852 1504256 0 0 256 20 240 149 0 26 25 49
> 3 3 75280 73344 1852 1504268 0 0 0 0 265 140 0 29 13 58
> 3 3 75280 73468 1852 1504232 0 0 0 0 265 132 0 22 34 44
> 3 3 75280 73468 1852 1504232 0 0 0 0 339 283 0 25 26 49
> 3 3 75280 73468 1852 1504232 0 0 0 0 362 327 0 25 25 50
> 3 3 75280 73468 1852 1504232 0 0 0 0 317 320 0 26 25 49
> 3 3 75280 73468 1852 1504232 0 0 0 0 361 343 0 26 25 50
>
> and nothing really happens for 150 s, except more and more tasks blocking
> in D state (second column).
> 3 6 75272 73416 1872 1503872 0 0 0 0 445 700 0 25 25 50
> 0 7 75264 67940 1884 1509008 64 0 5056 16 481 876 0 22 23 55
> Then suddently things get going again:
> 0 2 75104 76808 1892 1502552 352 0 14292 40456 459 14865 0 2 39 59
> 0 1 75104 75704 1900 1503412 0 0 820 32 405 788 1 1 72 27
> 1 0 75104 76512 1904 1505576 0 0 1068 1072 454 1586 8 7 74 11
>
> I guess this 150 s stall is when kernel barfs the "task blocked for more
> than 30 seconds" messages. And from the traces we know that everyone is
> waiting for PageWriteback or page lock during this time. Also James's vmstat
> report shows that IO is stalled when kswapd is spinning. Really strange.
Just to add, this machine has relatively low amount of memory (1GB).
I've re-run the tests today with cgroups disabled and it ran for 47
'copy' cycles, 27 'copy' cycles and then 35 'copy' cycles.
One extra data point, with maxcpus=1 I get a lockup after 2 'copy'
cycles every time, so it's more predictable than the default 4 processor
configuration.
>
> James in the meantime identified that cgroups are somehow involved. Are you
> using systemd by any chance?
No, I'm using upstart.
> Maybe cgroup IO throttling screws us?
>
> Honza
>
> > > > [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds.
> > > > [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000
> > > > [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000
> > > > [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00
> > > > [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00
> > > > [ 287.088400] Call Trace:
> > > > [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > > [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > > [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > > [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > > [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > > [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > > [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > > [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > > [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > > [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170
> > > > [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0
> > > > [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > > [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> > > > [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> > > > [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> > > > [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > > [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > > [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> > > > [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230
> > > > [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > > [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds.
> > > > [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000
> > > > [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000
> > > > [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00
> > > > [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00
> > > > [ 287.088484] Call Trace:
> > > > [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > > [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > > [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > > [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > > [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > > [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > > [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > > [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > > [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60
> > > > [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > > [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60
> > > > [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > > [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> > > > [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> > > > [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > > [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > > [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20
> > > > [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0
> > > > [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90
> > > > [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > > [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds.
> > > > [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000
> > > > [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000
> > > > [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00
> > > > [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00
> > > > [ 287.088577] Call Trace:
> > > > [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > > [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > > [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > > [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > > [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > > [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > > [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > > [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > > [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > > [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > > [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> > > > [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> > > > [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > > [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > > [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> > > > [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > > [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds.
> > > > [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000
> > > > [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000
> > > > [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00
> > > > [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00
> > > > [ 287.088647] Call Trace:
> > > > [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > > [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > > [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > > [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > > [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > > [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > > [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190
> > > > [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30
> > > > [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170
> > > > [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190
> > > > [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0
> > > > [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220
> > > > [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
> > > > [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220
> > > > [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0
> > > > [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
> > > > [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0
> > > > [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
> > > > [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds.
> > > > [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000
> > > > [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000
> > > > [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00
> > > > [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00
> > > > [ 287.088723] Call Trace:
> > > > [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > > [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > > [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > > [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > > [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > > [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > > [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > > [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > > [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > > [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > > [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> > > > [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> > > > [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> > > > [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > > [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > > [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500
> > > > [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0
> > > > [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > > [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70
> > > > [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70
> > > > [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40
> > > > [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40
> > > > [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00
> > > > [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300
> > > > [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00
> > > > [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0
> > > > [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80
> > > > [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0
> > > > --
> > > > Jan Kara <jack@suse.cz>
> > > > SUSE Labs, CR
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> > > > the body of a message to majordomo@vger.kernel.org
> > > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-03 18:55 ` Colin Ian King
0 siblings, 0 replies; 138+ messages in thread
From: Colin Ian King @ 2011-05-03 18:55 UTC (permalink / raw)
To: Jan Kara
Cc: Chris Mason, James Bottomley, linux-fsdevel, linux-mm,
linux-kernel, linux-ext4, mgorman
On Fri, 2011-04-29 at 00:40 +0200, Jan Kara wrote:
> On Thu 28-04-11 15:58:21, Colin Ian King wrote:
> > On Thu, 2011-04-28 at 16:33 +0200, Jan Kara wrote:
> > > On Thu 28-04-11 16:25:51, Jan Kara wrote:
> > > > On Thu 28-04-11 15:01:22, Colin Ian King wrote:
> > > > >
> > > > > > Could you post the soft lockups you're seeing?
> > > > >
> > > > > As requested, attached
> > > > Hum, what keeps puzzling me is that in all the cases of hangs I've seen
> > > > so far, we are stuck waiting for IO to finish for a long time - e.g. in the
> > > > traces below kjournald waits for PageWriteback bit to get cleared. Also we
> > > > are stuck waiting for page locks which might be because those pages are
> > > > being read in? All in all it seems that the IO is just incredibly slow.
> > > >
> > > > But it's not clear to me what pushes us into that situation (especially
> > > > since ext4 refuses to do any IO from ->writepage (i.e. kswapd) when the
> > > > underlying blocks are not already allocated.
> > > Hmm, maybe because the system is under memory pressure (and kswapd is not
> > > able to get rid of dirty pages), we page out clean pages. Thus also pages
> > > of executables which need to be paged in soon anyway thus putting heavy
> > > read load on the system which makes writes crawl? I'm not sure why
> > > compaction should make this any worse but maybe it can.
> > >
> > > James, Colin, can you capture output of 'vmstat 1' while you do the
> > > copying? Thanks.
> >
> > Attached.
> Thanks. So I there are a few interesting points in the vmstat output:
> For first 30 seconds, we are happily copying data - relatively steady read
> throughput (about 20-40 MB/s) and occasional peak from flusher thread
> flushing dirty data. During this time free memory drops from about 1.4 GB
> to about 22!!! MB - mm seems to like to really use the machine ;). Then
> things get interesting:
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 0 1 0 83372 5228 1450776 0 0 39684 90132 450 918 0 4 74 22
> 0 1 0 22416 5228 1509864 0 0 29452 48492 403 869 1 2 80 18
> 2 0 0 20056 5384 1513996 0 0 2248 2116 434 1191 4 4 71 21
> 0 1 0 19800 5932 1514600 0 0 644 104 454 1166 8 3 64 24
> 1 0 0 21848 5940 1515244 0 0 292 380 468 1775 15 6 75 3
> 1 0 0 20936 5940 1515876 0 0 296 296 496 1846 18 8 74 0
> 1 0 0 17792 5940 1516580 0 0 356 356 484 1862 23 8 69 0
> 1 0 0 17544 5940 1517364 0 0 412 412 482 1812 16 7 77 0
> 4 0 0 18148 5948 1517968 0 0 288 344 436 1749 19 9 69 3
> 0 2 220 137528 1616 1402468 0 220 74708 2164 849 1806 3 6 63 28
> 0 3 224 36184 1628 1499648 0 4 50820 86204 532 1272 0 4 64 32
> 0 2 19680 53688 1628 1484388 32 19456 6080 62972 242 287 0 2 63 34
> 0 2 36928 1407432 1356 150980 0 17252 1564 17276 368 764 1 3 73 22
>
> So when free memory reached about 20 MB, both read and write activity
> almost stalled for 7 s (probably everybody waits for free memory). Then
> mm manages to free 100 MB from page cache, things move on for two seconds,
> then we swap out! about 36 MB and page reclaim also finally decides it
> maybe has too much of page cache and reaps most of it (1.3 GB in one go).
> Then things get going again, although there are still occasional stalls
> such as this (about 30s later):
> 1 3 36688 753192 1208 792344 0 0 35948 32768 435 6625 0 6 61 33
> 0 2 36668 754996 1344 792760 0 0 252 58736 2832 16239 0 1 60 39
> 0 2 36668 750132 1388 796688 0 0 2508 1524 325 959 1 3 68 28
> 1 0 36668 751160 1400 797968 0 0 620 620 460 1470 6 6 50 38
> 1 0 36668 750516 1400 798520 0 0 300 300 412 1764 17 8 75 1
> 1 0 36668 750648 1408 799108 0 0 280 340 423 1816 18 6 73 3
> 1 0 36668 748856 1408 799752 0 0 336 328 409 1788 18 8 75 0
> 1 0 36668 748120 1416 800604 0 0 428 452 407 1723 14 10 75 2
> 1 0 36668 750048 1416 801176 0 0 296 296 405 1779 18 7 75 1
> 0 1 36668 650428 1420 897252 0 0 48100 556 658 1718 10 3 71 15
> 0 2 36668 505444 1424 1037012 0 0 69888 90272 686 1491 1 4 68 27
> 0 1 36668 479264 1428 1063372 0 0 12984 40896 324 674 1 1 76 23
> ...
> I'm not sure what we were blocked on here since there is still plenty of
> free memory (750 MB). These stalls repeat once in a while but things go on.
> Then at about 350s, things just stop:
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 3 1 75280 73564 1844 1503848 0 0 43396 81976 627 1061 0 25 42 32
> 3 3 75280 73344 1852 1504256 0 0 256 20 240 149 0 26 25 49
> 3 3 75280 73344 1852 1504268 0 0 0 0 265 140 0 29 13 58
> 3 3 75280 73468 1852 1504232 0 0 0 0 265 132 0 22 34 44
> 3 3 75280 73468 1852 1504232 0 0 0 0 339 283 0 25 26 49
> 3 3 75280 73468 1852 1504232 0 0 0 0 362 327 0 25 25 50
> 3 3 75280 73468 1852 1504232 0 0 0 0 317 320 0 26 25 49
> 3 3 75280 73468 1852 1504232 0 0 0 0 361 343 0 26 25 50
>
> and nothing really happens for 150 s, except more and more tasks blocking
> in D state (second column).
> 3 6 75272 73416 1872 1503872 0 0 0 0 445 700 0 25 25 50
> 0 7 75264 67940 1884 1509008 64 0 5056 16 481 876 0 22 23 55
> Then suddently things get going again:
> 0 2 75104 76808 1892 1502552 352 0 14292 40456 459 14865 0 2 39 59
> 0 1 75104 75704 1900 1503412 0 0 820 32 405 788 1 1 72 27
> 1 0 75104 76512 1904 1505576 0 0 1068 1072 454 1586 8 7 74 11
>
> I guess this 150 s stall is when kernel barfs the "task blocked for more
> than 30 seconds" messages. And from the traces we know that everyone is
> waiting for PageWriteback or page lock during this time. Also James's vmstat
> report shows that IO is stalled when kswapd is spinning. Really strange.
Just to add, this machine has relatively low amount of memory (1GB).
I've re-run the tests today with cgroups disabled and it ran for 47
'copy' cycles, 27 'copy' cycles and then 35 'copy' cycles.
One extra data point, with maxcpus=1 I get a lockup after 2 'copy'
cycles every time, so it's more predictable than the default 4 processor
configuration.
>
> James in the meantime identified that cgroups are somehow involved. Are you
> using systemd by any chance?
No, I'm using upstart.
> Maybe cgroup IO throttling screws us?
>
> Honza
>
> > > > [ 287.088371] INFO: task rs:main Q:Reg:749 blocked for more than 30 seconds.
> > > > [ 287.088374] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > [ 287.088376] rs:main Q:Reg D 0000000000000000 0 749 1 0x00000000
> > > > [ 287.088381] ffff880072c17b68 0000000000000082 ffff880072c17fd8 ffff880072c16000
> > > > [ 287.088392] 0000000000013d00 ffff88003591b178 ffff880072c17fd8 0000000000013d00
> > > > [ 287.088396] ffffffff81a0b020 ffff88003591adc0 ffff88001fffc3e8 ffff88001fc13d00
> > > > [ 287.088400] Call Trace:
> > > > [ 287.088404] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > > [ 287.088408] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > > [ 287.088411] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > > [ 287.088414] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > > [ 287.088418] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > > [ 287.088421] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > > [ 287.088425] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > > [ 287.088428] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > > [ 287.088431] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > > [ 287.088434] [<ffffffff81134a43>] ? unmap_region+0x113/0x170
> > > > [ 287.088437] [<ffffffff812ded90>] ? prio_tree_insert+0x150/0x1c0
> > > > [ 287.088440] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > > [ 287.088442] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> > > > [ 287.088446] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> > > > [ 287.088448] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> > > > [ 287.088451] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > > [ 287.088454] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > > [ 287.088457] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> > > > [ 287.088460] [<ffffffff81137127>] ? sys_mmap_pgoff+0x167/0x230
> > > > [ 287.088463] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > > [ 287.088466] INFO: task NetworkManager:764 blocked for more than 30 seconds.
> > > > [ 287.088468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > [ 287.088470] NetworkManager D 0000000000000002 0 764 1 0x00000000
> > > > [ 287.088473] ffff880074ffbb68 0000000000000082 ffff880074ffbfd8 ffff880074ffa000
> > > > [ 287.088477] 0000000000013d00 ffff880036051a98 ffff880074ffbfd8 0000000000013d00
> > > > [ 287.088481] ffff8801005badc0 ffff8800360516e0 ffff88001ffef128 ffff88001fc53d00
> > > > [ 287.088484] Call Trace:
> > > > [ 287.088488] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > > [ 287.088491] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > > [ 287.088494] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > > [ 287.088497] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > > [ 287.088500] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > > [ 287.088503] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > > [ 287.088506] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > > [ 287.088509] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > > [ 287.088513] [<ffffffff81177110>] ? pollwake+0x0/0x60
> > > > [ 287.088516] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > > [ 287.088519] [<ffffffff81177110>] ? pollwake+0x0/0x60
> > > > [ 287.088522] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > > [ 287.088525] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> > > > [ 287.088527] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> > > > [ 287.088530] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > > [ 287.088533] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > > [ 287.088537] [<ffffffff81013859>] ? read_tsc+0x9/0x20
> > > > [ 287.088540] [<ffffffff81092eb1>] ? ktime_get_ts+0xb1/0xf0
> > > > [ 287.088543] [<ffffffff811776d2>] ? poll_select_set_timeout+0x82/0x90
> > > > [ 287.088546] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > > [ 287.088559] INFO: task unity-panel-ser:1521 blocked for more than 30 seconds.
> > > > [ 287.088561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > [ 287.088562] unity-panel-ser D 0000000000000000 0 1521 1 0x00000000
> > > > [ 287.088566] ffff880061f37b68 0000000000000082 ffff880061f37fd8 ffff880061f36000
> > > > [ 287.088570] 0000000000013d00 ffff880068c7c858 ffff880061f37fd8 0000000000013d00
> > > > [ 287.088573] ffff88003591c4a0 ffff880068c7c4a0 ffff88001fff0c88 ffff88001fc13d00
> > > > [ 287.088577] Call Trace:
> > > > [ 287.088581] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > > [ 287.088583] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > > [ 287.088587] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > > [ 287.088589] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > > [ 287.088593] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > > [ 287.088596] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > > [ 287.088599] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > > [ 287.088602] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > > [ 287.088605] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > > [ 287.088608] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > > [ 287.088610] [<ffffffff8111561d>] ? __free_pages+0x2d/0x40
> > > > [ 287.088613] [<ffffffff8112de4f>] ? __pte_alloc+0xdf/0x100
> > > > [ 287.088616] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > > [ 287.088619] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > > [ 287.088622] [<ffffffff81136f85>] ? do_mmap_pgoff+0x335/0x370
> > > > [ 287.088625] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > > [ 287.088629] INFO: task jbd2/sda4-8:1845 blocked for more than 30 seconds.
> > > > [ 287.088630] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > [ 287.088632] jbd2/sda4-8 D 0000000000000000 0 1845 2 0x00000000
> > > > [ 287.088636] ffff880068f6baf0 0000000000000046 ffff880068f6bfd8 ffff880068f6a000
> > > > [ 287.088639] 0000000000013d00 ffff880061d603b8 ffff880068f6bfd8 0000000000013d00
> > > > [ 287.088643] ffff88003591c4a0 ffff880061d60000 ffff88001fff8548 ffff88001fc13d00
> > > > [ 287.088647] Call Trace:
> > > > [ 287.088650] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > > [ 287.088653] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > > [ 287.088656] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > > [ 287.088659] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > > [ 287.088662] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > > [ 287.088665] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > > [ 287.088668] [<ffffffff8110c41d>] filemap_fdatawait_range+0xfd/0x190
> > > > [ 287.088672] [<ffffffff8110c4db>] filemap_fdatawait+0x2b/0x30
> > > > [ 287.088675] [<ffffffff81242a93>] journal_finish_inode_data_buffers+0x63/0x170
> > > > [ 287.088678] [<ffffffff81243284>] jbd2_journal_commit_transaction+0x6e4/0x1190
> > > > [ 287.088682] [<ffffffff81076185>] ? try_to_del_timer_sync+0x85/0xe0
> > > > [ 287.088685] [<ffffffff81247e9b>] kjournald2+0xbb/0x220
> > > > [ 287.088688] [<ffffffff81087f30>] ? autoremove_wake_function+0x0/0x40
> > > > [ 287.088691] [<ffffffff81247de0>] ? kjournald2+0x0/0x220
> > > > [ 287.088694] [<ffffffff810877e6>] kthread+0x96/0xa0
> > > > [ 287.088697] [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
> > > > [ 287.088700] [<ffffffff81087750>] ? kthread+0x0/0xa0
> > > > [ 287.088703] [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
> > > > [ 287.088705] INFO: task dirname:5969 blocked for more than 30 seconds.
> > > > [ 287.088707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > [ 287.088709] dirname D 0000000000000002 0 5969 5214 0x00000000
> > > > [ 287.088712] ffff88005bd9d8b8 0000000000000086 ffff88005bd9dfd8 ffff88005bd9c000
> > > > [ 287.088716] 0000000000013d00 ffff88005d65b178 ffff88005bd9dfd8 0000000000013d00
> > > > [ 287.088720] ffff8801005e5b80 ffff88005d65adc0 ffff88001ffe5228 ffff88001fc53d00
> > > > [ 287.088723] Call Trace:
> > > > [ 287.088726] [<ffffffff8110c070>] ? sync_page+0x0/0x50
> > > > [ 287.088729] [<ffffffff815c0990>] io_schedule+0x70/0xc0
> > > > [ 287.088732] [<ffffffff8110c0b0>] sync_page+0x40/0x50
> > > > [ 287.088735] [<ffffffff815c130f>] __wait_on_bit+0x5f/0x90
> > > > [ 287.088738] [<ffffffff8110c278>] wait_on_page_bit+0x78/0x80
> > > > [ 287.088741] [<ffffffff81087f70>] ? wake_bit_function+0x0/0x50
> > > > [ 287.088744] [<ffffffff8110dffd>] __lock_page_or_retry+0x3d/0x70
> > > > [ 287.088747] [<ffffffff8110e3c7>] filemap_fault+0x397/0x4a0
> > > > [ 287.088750] [<ffffffff8112d144>] __do_fault+0x54/0x520
> > > > [ 287.088753] [<ffffffff811309da>] handle_pte_fault+0xfa/0x210
> > > > [ 287.088756] [<ffffffff810442a7>] ? pte_alloc_one+0x37/0x50
> > > > [ 287.088759] [<ffffffff815c2cce>] ? _raw_spin_lock+0xe/0x20
> > > > [ 287.088761] [<ffffffff8112de25>] ? __pte_alloc+0xb5/0x100
> > > > [ 287.088764] [<ffffffff81131d5d>] handle_mm_fault+0x16d/0x250
> > > > [ 287.088767] [<ffffffff815c6a47>] do_page_fault+0x1a7/0x540
> > > > [ 287.088770] [<ffffffff81136947>] ? mmap_region+0x1f7/0x500
> > > > [ 287.088773] [<ffffffff8112db06>] ? free_pgd_range+0x356/0x4a0
> > > > [ 287.088776] [<ffffffff815c34d5>] page_fault+0x25/0x30
> > > > [ 287.088779] [<ffffffff812e6d5f>] ? __clear_user+0x3f/0x70
> > > > [ 287.088782] [<ffffffff812e6d41>] ? __clear_user+0x21/0x70
> > > > [ 287.088786] [<ffffffff812e6dc6>] clear_user+0x36/0x40
> > > > [ 287.088788] [<ffffffff811b0b6d>] padzero+0x2d/0x40
> > > > [ 287.088791] [<ffffffff811b2c7a>] load_elf_binary+0x95a/0xe00
> > > > [ 287.088794] [<ffffffff8116aa8a>] search_binary_handler+0xda/0x300
> > > > [ 287.088797] [<ffffffff811b2320>] ? load_elf_binary+0x0/0xe00
> > > > [ 287.088800] [<ffffffff8116c49c>] do_execve+0x24c/0x2d0
> > > > [ 287.088802] [<ffffffff8101521a>] sys_execve+0x4a/0x80
> > > > [ 287.088805] [<ffffffff8100c45c>] stub_execve+0x6c/0xc0
> > > > --
> > > > Jan Kara <jack@suse.cz>
> > > > SUSE Labs, CR
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> > > > the body of a message to majordomo@vger.kernel.org
> > > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-03 14:22 ` James Bottomley
@ 2011-05-06 7:42 ` Mel Gorman
0 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-05-06 7:42 UTC (permalink / raw)
To: James Bottomley
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Tue, May 03, 2011 at 09:22:33AM -0500, James Bottomley wrote:
> On Tue, 2011-05-03 at 09:13 -0500, James Bottomley wrote:
> > I've got a ftrace output of kswapd ... it's 500k compressed, so I'll
> > send under separate cover.
>
> Here it is ... it's under 2.6.38.4 vanilla, but the code is similar.
>
I was quiet because I was off trying to reproduce this but not having
much luck. It doesn't seem directly related to filesystems or
cgroups. For example, here is what I see with ext4 without cgroups
2.6.34-vanilla 2.6.37-vanilla 2.6.38-vanilla rc6-vanilla
download tar 70 ( 0.00%) 68 ( 2.94%) 69 ( 1.45%) 70 ( 0.00%)
unpack tar 601 ( 0.00%) 605 (-0.66%) 604 (-0.50%) 605 (-0.66%)
copy source files 319 ( 0.00%) 321 (-0.62%) 320 (-0.31%) 332 (-3.92%)
create tarfile 1368 ( 0.00%) 1372 (-0.29%) 1371 (-0.22%) 1363 ( 0.37%)
delete source dirs 21 ( 0.00%) 21 ( 0.00%) 23 (-8.70%) 22 (-4.55%)
expand tar 263 ( 0.00%) 261 ( 0.77%) 257 ( 2.33%) 259 ( 1.54%)
(all results are in seconds)
When running in cgroups, the results are similar - bit slower but
not remarkably so. ext3 is slower but not enough to count as the bug.
The trace you posted is very short but kswapd is not going to sleep
in it. It's less than a seconds worth on different cpus so it's hard
to draw any conclusion from it other than sleeping_prematurely()
is often deciding that kswapd should not sleep.
So lets consider what keeps it awake.
1. High-order allocations? You machine is using i915 and RPC, something
neither of my test machine uses. i915 is potentially a source for
high-order allocations. I'm attaching a perl script. Please run it as
./watch-highorder.pl --output /tmp/highorders.txt
while you are running tar. When kswapd is running for about 30
seconds, interrupt it with ctrl+c twice in quick succession and
post /tmp/highorders.txt
2. All unreclaimable is not being set or we are not balancing at all.
Can you post the output of sysrq+m while the machine is struggling
please?
3. Slab may not be shrinking for some reason. Can you run a shell
script like this during the whole test and record its output please?
#!/bin/bash
while [ 1 ]; do
echo time: `date +%s`
cat /proc/vmstat
sleep 2
done
Similarly if this is a slab issue, it'd be nice to know who it is so
#!/bin/bash
while [ 1 ]; do
echo time: `date +%s`
cat /proc/slabinfo
sleep $MONITOR_UPDATE_FREQUENCY
done
4. Lets get a better look at what is going on in kswapd
echo 1 > /sys/kernel/debug/tracing/events/vmscan/enable
cat /sys/kernel/debug/tracing/trace_pipe > vmscan-ftrace.txt
Out of curiousity, what's the exact machine you are testing on
because I want to see what sort of hardware combination triggers
this problem? Is the tar local or is it copied over NFS? What is the
configuration of the disk or partition you are copying to?
Thanks
--
Mel Gorman
SUSE Labs
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-06 7:42 ` Mel Gorman
0 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-05-06 7:42 UTC (permalink / raw)
To: James Bottomley
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Tue, May 03, 2011 at 09:22:33AM -0500, James Bottomley wrote:
> On Tue, 2011-05-03 at 09:13 -0500, James Bottomley wrote:
> > I've got a ftrace output of kswapd ... it's 500k compressed, so I'll
> > send under separate cover.
>
> Here it is ... it's under 2.6.38.4 vanilla, but the code is similar.
>
I was quiet because I was off trying to reproduce this but not having
much luck. It doesn't seem directly related to filesystems or
cgroups. For example, here is what I see with ext4 without cgroups
2.6.34-vanilla 2.6.37-vanilla 2.6.38-vanilla rc6-vanilla
download tar 70 ( 0.00%) 68 ( 2.94%) 69 ( 1.45%) 70 ( 0.00%)
unpack tar 601 ( 0.00%) 605 (-0.66%) 604 (-0.50%) 605 (-0.66%)
copy source files 319 ( 0.00%) 321 (-0.62%) 320 (-0.31%) 332 (-3.92%)
create tarfile 1368 ( 0.00%) 1372 (-0.29%) 1371 (-0.22%) 1363 ( 0.37%)
delete source dirs 21 ( 0.00%) 21 ( 0.00%) 23 (-8.70%) 22 (-4.55%)
expand tar 263 ( 0.00%) 261 ( 0.77%) 257 ( 2.33%) 259 ( 1.54%)
(all results are in seconds)
When running in cgroups, the results are similar - bit slower but
not remarkably so. ext3 is slower but not enough to count as the bug.
The trace you posted is very short but kswapd is not going to sleep
in it. It's less than a seconds worth on different cpus so it's hard
to draw any conclusion from it other than sleeping_prematurely()
is often deciding that kswapd should not sleep.
So lets consider what keeps it awake.
1. High-order allocations? You machine is using i915 and RPC, something
neither of my test machine uses. i915 is potentially a source for
high-order allocations. I'm attaching a perl script. Please run it as
./watch-highorder.pl --output /tmp/highorders.txt
while you are running tar. When kswapd is running for about 30
seconds, interrupt it with ctrl+c twice in quick succession and
post /tmp/highorders.txt
2. All unreclaimable is not being set or we are not balancing at all.
Can you post the output of sysrq+m while the machine is struggling
please?
3. Slab may not be shrinking for some reason. Can you run a shell
script like this during the whole test and record its output please?
#!/bin/bash
while [ 1 ]; do
echo time: `date +%s`
cat /proc/vmstat
sleep 2
done
Similarly if this is a slab issue, it'd be nice to know who it is so
#!/bin/bash
while [ 1 ]; do
echo time: `date +%s`
cat /proc/slabinfo
sleep $MONITOR_UPDATE_FREQUENCY
done
4. Lets get a better look at what is going on in kswapd
echo 1 > /sys/kernel/debug/tracing/events/vmscan/enable
cat /sys/kernel/debug/tracing/trace_pipe > vmscan-ftrace.txt
Out of curiousity, what's the exact machine you are testing on
because I want to see what sort of hardware combination triggers
this problem? Is the tar local or is it copied over NFS? What is the
configuration of the disk or partition you are copying to?
Thanks
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-06 7:42 ` Mel Gorman
(?)
@ 2011-05-06 8:07 ` Mel Gorman
2011-05-09 18:16 ` James Bottomley
-1 siblings, 1 reply; 138+ messages in thread
From: Mel Gorman @ 2011-05-06 8:07 UTC (permalink / raw)
To: James Bottomley
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
[-- Attachment #1: Type: text/plain, Size: 554 bytes --]
On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote:
> 1. High-order allocations? You machine is using i915 and RPC, something
> neither of my test machine uses. i915 is potentially a source for
> high-order allocations. I'm attaching a perl script. Please run it as
> ./watch-highorder.pl --output /tmp/highorders.txt
> while you are running tar. When kswapd is running for about 30
> seconds, interrupt it with ctrl+c twice in quick succession and
> post /tmp/highorders.txt
>
Attached this time :/
--
Mel Gorman
SUSE Labs
[-- Attachment #2: watch-highorder.pl --]
[-- Type: application/x-perl, Size: 7303 bytes --]
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-06 7:42 ` Mel Gorman
@ 2011-05-06 11:42 ` Mel Gorman
-1 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-05-06 11:42 UTC (permalink / raw)
To: James Bottomley
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote:
> On Tue, May 03, 2011 at 09:22:33AM -0500, James Bottomley wrote:
> > On Tue, 2011-05-03 at 09:13 -0500, James Bottomley wrote:
> > > I've got a ftrace output of kswapd ... it's 500k compressed, so I'll
> > > send under separate cover.
> >
> > Here it is ... it's under 2.6.38.4 vanilla, but the code is similar.
> >
>
> I was quiet because I was off trying to reproduce this but not having
> much luck. It doesn't seem directly related to filesystems or
> cgroups. For example, here is what I see with ext4 without cgroups
>
> 2.6.34-vanilla 2.6.37-vanilla 2.6.38-vanilla rc6-vanilla
> download tar 70 ( 0.00%) 68 ( 2.94%) 69 ( 1.45%) 70 ( 0.00%)
> unpack tar 601 ( 0.00%) 605 (-0.66%) 604 (-0.50%) 605 (-0.66%)
> copy source files 319 ( 0.00%) 321 (-0.62%) 320 (-0.31%) 332 (-3.92%)
> create tarfile 1368 ( 0.00%) 1372 (-0.29%) 1371 (-0.22%) 1363 ( 0.37%)
> delete source dirs 21 ( 0.00%) 21 ( 0.00%) 23 (-8.70%) 22 (-4.55%)
> expand tar 263 ( 0.00%) 261 ( 0.77%) 257 ( 2.33%) 259 ( 1.54%)
>
> (all results are in seconds)
>
> When running in cgroups, the results are similar - bit slower but
> not remarkably so. ext3 is slower but not enough to count as the bug.
>
> The trace you posted is very short but kswapd is not going to sleep
> in it. It's less than a seconds worth on different cpus so it's hard
> to draw any conclusion from it other than sleeping_prematurely()
> is often deciding that kswapd should not sleep.
>
> So lets consider what keeps it awake.
>
> 1. High-order allocations? You machine is using i915 and RPC, something
> neither of my test machine uses. i915 is potentially a source for
> high-order allocations. I'm attaching a perl script. Please run it as
> ./watch-highorder.pl --output /tmp/highorders.txt
> while you are running tar. When kswapd is running for about 30
> seconds, interrupt it with ctrl+c twice in quick succession and
> post /tmp/highorders.txt
>
> 2. All unreclaimable is not being set or we are not balancing at all.
> Can you post the output of sysrq+m while the machine is struggling
> please?
>
> 3. Slab may not be shrinking for some reason. Can you run a shell
> script like this during the whole test and record its output please?
>
> #!/bin/bash
> while [ 1 ]; do
> echo time: `date +%s`
> cat /proc/vmstat
> sleep 2
> done
>
> Similarly if this is a slab issue, it'd be nice to know who it is so
>
> #!/bin/bash
> while [ 1 ]; do
> echo time: `date +%s`
> cat /proc/slabinfo
> sleep $MONITOR_UPDATE_FREQUENCY
> done
>
> 4. Lets get a better look at what is going on in kswapd
>
> echo 1 > /sys/kernel/debug/tracing/events/vmscan/enable
> cat /sys/kernel/debug/tracing/trace_pipe > vmscan-ftrace.txt
>
Also, could you test the patch at https://lkml.org/lkml/2011/3/5/121
please?
--
Mel Gorman
SUSE Labs
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-06 11:42 ` Mel Gorman
0 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-05-06 11:42 UTC (permalink / raw)
To: James Bottomley
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote:
> On Tue, May 03, 2011 at 09:22:33AM -0500, James Bottomley wrote:
> > On Tue, 2011-05-03 at 09:13 -0500, James Bottomley wrote:
> > > I've got a ftrace output of kswapd ... it's 500k compressed, so I'll
> > > send under separate cover.
> >
> > Here it is ... it's under 2.6.38.4 vanilla, but the code is similar.
> >
>
> I was quiet because I was off trying to reproduce this but not having
> much luck. It doesn't seem directly related to filesystems or
> cgroups. For example, here is what I see with ext4 without cgroups
>
> 2.6.34-vanilla 2.6.37-vanilla 2.6.38-vanilla rc6-vanilla
> download tar 70 ( 0.00%) 68 ( 2.94%) 69 ( 1.45%) 70 ( 0.00%)
> unpack tar 601 ( 0.00%) 605 (-0.66%) 604 (-0.50%) 605 (-0.66%)
> copy source files 319 ( 0.00%) 321 (-0.62%) 320 (-0.31%) 332 (-3.92%)
> create tarfile 1368 ( 0.00%) 1372 (-0.29%) 1371 (-0.22%) 1363 ( 0.37%)
> delete source dirs 21 ( 0.00%) 21 ( 0.00%) 23 (-8.70%) 22 (-4.55%)
> expand tar 263 ( 0.00%) 261 ( 0.77%) 257 ( 2.33%) 259 ( 1.54%)
>
> (all results are in seconds)
>
> When running in cgroups, the results are similar - bit slower but
> not remarkably so. ext3 is slower but not enough to count as the bug.
>
> The trace you posted is very short but kswapd is not going to sleep
> in it. It's less than a seconds worth on different cpus so it's hard
> to draw any conclusion from it other than sleeping_prematurely()
> is often deciding that kswapd should not sleep.
>
> So lets consider what keeps it awake.
>
> 1. High-order allocations? You machine is using i915 and RPC, something
> neither of my test machine uses. i915 is potentially a source for
> high-order allocations. I'm attaching a perl script. Please run it as
> ./watch-highorder.pl --output /tmp/highorders.txt
> while you are running tar. When kswapd is running for about 30
> seconds, interrupt it with ctrl+c twice in quick succession and
> post /tmp/highorders.txt
>
> 2. All unreclaimable is not being set or we are not balancing at all.
> Can you post the output of sysrq+m while the machine is struggling
> please?
>
> 3. Slab may not be shrinking for some reason. Can you run a shell
> script like this during the whole test and record its output please?
>
> #!/bin/bash
> while [ 1 ]; do
> echo time: `date +%s`
> cat /proc/vmstat
> sleep 2
> done
>
> Similarly if this is a slab issue, it'd be nice to know who it is so
>
> #!/bin/bash
> while [ 1 ]; do
> echo time: `date +%s`
> cat /proc/slabinfo
> sleep $MONITOR_UPDATE_FREQUENCY
> done
>
> 4. Lets get a better look at what is going on in kswapd
>
> echo 1 > /sys/kernel/debug/tracing/events/vmscan/enable
> cat /sys/kernel/debug/tracing/trace_pipe > vmscan-ftrace.txt
>
Also, could you test the patch at https://lkml.org/lkml/2011/3/5/121
please?
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-06 7:42 ` Mel Gorman
@ 2011-05-06 15:44 ` Mel Gorman
-1 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-05-06 15:44 UTC (permalink / raw)
To: James Bottomley
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote:
>
> 1. High-order allocations? You machine is using i915 and RPC, something
> neither of my test machine uses. i915 is potentially a source for
> high-order allocations. I'm attaching a perl script. Please run it as
> ./watch-highorder.pl --output /tmp/highorders.txt
> while you are running tar. When kswapd is running for about 30
> seconds, interrupt it with ctrl+c twice in quick succession and
> post /tmp/highorders.txt
>
Colin send me this information for his test case at least and I see
11932 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC
=> alloc_pages_current+0xa5/0x110 <ffffffff81149ef5>
=> new_slab+0x1f5/0x290 <ffffffff81153645>
=> __slab_alloc+0x262/0x390 <ffffffff81155192>
=> kmem_cache_alloc+0x115/0x120 <ffffffff81155ab5>
=> mempool_alloc_slab+0x15/0x20 <ffffffff8110e705>
=> mempool_alloc+0x59/0x140 <ffffffff8110ea49>
=> bio_alloc_bioset+0x3e/0xf0 <ffffffff811976ae>
=> bio_alloc+0x15/0x30 <ffffffff81197805>
Colin and James: Did you happen to switch from SLAB to SLUB between
2.6.37 and 2.6.38? My own tests were against SLAB which might be why I
didn't see the problem. Am restarting the tests with SLUB.
--
Mel Gorman
SUSE Labs
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-06 15:44 ` Mel Gorman
0 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-05-06 15:44 UTC (permalink / raw)
To: James Bottomley
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote:
>
> 1. High-order allocations? You machine is using i915 and RPC, something
> neither of my test machine uses. i915 is potentially a source for
> high-order allocations. I'm attaching a perl script. Please run it as
> ./watch-highorder.pl --output /tmp/highorders.txt
> while you are running tar. When kswapd is running for about 30
> seconds, interrupt it with ctrl+c twice in quick succession and
> post /tmp/highorders.txt
>
Colin send me this information for his test case at least and I see
11932 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC
=> alloc_pages_current+0xa5/0x110 <ffffffff81149ef5>
=> new_slab+0x1f5/0x290 <ffffffff81153645>
=> __slab_alloc+0x262/0x390 <ffffffff81155192>
=> kmem_cache_alloc+0x115/0x120 <ffffffff81155ab5>
=> mempool_alloc_slab+0x15/0x20 <ffffffff8110e705>
=> mempool_alloc+0x59/0x140 <ffffffff8110ea49>
=> bio_alloc_bioset+0x3e/0xf0 <ffffffff811976ae>
=> bio_alloc+0x15/0x30 <ffffffff81197805>
Colin and James: Did you happen to switch from SLAB to SLUB between
2.6.37 and 2.6.38? My own tests were against SLAB which might be why I
didn't see the problem. Am restarting the tests with SLUB.
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-06 7:42 ` Mel Gorman
` (3 preceding siblings ...)
(?)
@ 2011-05-06 15:58 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-06 15:58 UTC (permalink / raw)
To: Mel Gorman
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
[-- Attachment #1: Type: text/plain, Size: 972 bytes --]
On Fri, 2011-05-06 at 08:42 +0100, Mel Gorman wrote:
[...]
Will try tests and post results
> Out of curiousity, what's the exact machine you are testing on
> because I want to see what sort of hardware combination triggers
> this problem? Is the tar local or is it copied over NFS? What is the
> configuration of the disk or partition you are copying to?
It's a Lenovo T420s with 2GB of memory. It's core i5, so it has two
cores each with two threads, so it emulates a 4 cpu system. It's
running Fedora 15 Beta, so the 2.6.38.4 kernel (although I've tried the
default 2.6.38.5 as well, since the fedora i915 updates are a bit
unstable). I'm also booting 2.6.39-rc on it (with and without PREEMPT).
The standard Fedora config is SLUB. Because it's FC15, it has all of
the cgroups stuff enabled for systemd.
Just in case, I've attached the FC15 config ... I modify it slightly for
2.6.39-rc but since the bug shows up in standard FC15, that should be
enough.
James
[-- Attachment #2: config-2.6.38.4-20.fc15.x86_64 --]
[-- Type: text/x-mpsub, Size: 114129 bytes --]
#
# Automatically generated make config: don't edit
# Linux/x86_64 2.6.38.4-20.fc15.x86_64 Kernel Configuration
# Thu Apr 28 23:31:51 2011
#
CONFIG_64BIT=y
# CONFIG_X86_32 is not set
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_GENERIC_GPIO=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_HAVE_INTEL_TXT=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11"
# CONFIG_KTIME_SCALAR is not set
CONFIG_ARCH_CPU_PROBE_RELEASE=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_CONSTRUCTORS=y
CONFIG_HAVE_IRQ_WORK=y
CONFIG_IRQ_WORK=y
#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
# CONFIG_KERNEL_LZO is not set
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_WATCH=y
CONFIG_AUDIT_TREE=y
CONFIG_UTRACE=y
CONFIG_HAVE_GENERIC_HARDIRQS=y
#
# IRQ subsystem
#
CONFIG_GENERIC_HARDIRQS=y
# CONFIG_GENERIC_HARDIRQS_NO_DEPRECATED is not set
CONFIG_HAVE_SPARSE_IRQ=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
# CONFIG_AUTO_IRQ_AFFINITY is not set
# CONFIG_IRQ_PER_CPU is not set
# CONFIG_HARDIRQS_SW_RESEND is not set
CONFIG_SPARSE_IRQ=y
#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_PREEMPT_RCU is not set
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_FANOUT=64
# CONFIG_RCU_FANOUT_EXACT is not set
CONFIG_RCU_FAST_NO_HZ=y
# CONFIG_TREE_RCU_TRACE is not set
# CONFIG_IKCONFIG is not set
CONFIG_LOG_BUF_SHIFT=17
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_CGROUPS=y
# CONFIG_CGROUP_DEBUG is not set
CONFIG_CGROUP_NS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_RESOURCE_COUNTERS=y
CONFIG_CGROUP_MEM_RES_CTLR=y
CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
# CONFIG_CGROUP_MEM_RES_CTLR_SWAP_ENABLED is not set
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_BLK_CGROUP=y
# CONFIG_DEBUG_BLK_CGROUP is not set
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_USER_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_SCHED_AUTOGROUP=y
CONFIG_MM_OWNER=y
# CONFIG_SYSFS_DEPRECATED is not set
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
# CONFIG_EXPERT is not set
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_EXTRA_PASS=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_HAVE_PERF_EVENTS=y
#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
CONFIG_PERF_COUNTERS=y
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
CONFIG_SLUB_DEBUG=y
# CONFIG_COMPAT_BRK is not set
# CONFIG_SLAB is not set
CONFIG_SLUB=y
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y
CONFIG_OPROFILE=m
CONFIG_OPROFILE_EVENT_MULTIPLEX=y
CONFIG_HAVE_OPROFILE=y
CONFIG_KPROBES=y
CONFIG_JUMP_LABEL=y
CONFIG_OPTPROBES=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_KRETPROBES=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
# CONFIG_MODVERSIONS is not set
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLK_DEV_INTEGRITY=y
CONFIG_BLK_DEV_THROTTLING=y
CONFIG_BLOCK_COMPAT=y
#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_CFQ_GROUP_IOSCHED=y
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_PADATA=y
# CONFIG_INLINE_SPIN_TRYLOCK is not set
# CONFIG_INLINE_SPIN_TRYLOCK_BH is not set
# CONFIG_INLINE_SPIN_LOCK is not set
# CONFIG_INLINE_SPIN_LOCK_BH is not set
# CONFIG_INLINE_SPIN_LOCK_IRQ is not set
# CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set
CONFIG_INLINE_SPIN_UNLOCK=y
# CONFIG_INLINE_SPIN_UNLOCK_BH is not set
CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
# CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set
# CONFIG_INLINE_READ_TRYLOCK is not set
# CONFIG_INLINE_READ_LOCK is not set
# CONFIG_INLINE_READ_LOCK_BH is not set
# CONFIG_INLINE_READ_LOCK_IRQ is not set
# CONFIG_INLINE_READ_LOCK_IRQSAVE is not set
CONFIG_INLINE_READ_UNLOCK=y
# CONFIG_INLINE_READ_UNLOCK_BH is not set
CONFIG_INLINE_READ_UNLOCK_IRQ=y
# CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set
# CONFIG_INLINE_WRITE_TRYLOCK is not set
# CONFIG_INLINE_WRITE_LOCK is not set
# CONFIG_INLINE_WRITE_LOCK_BH is not set
# CONFIG_INLINE_WRITE_LOCK_IRQ is not set
# CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set
CONFIG_INLINE_WRITE_UNLOCK=y
# CONFIG_INLINE_WRITE_UNLOCK_BH is not set
CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
# CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_FREEZER=y
#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
CONFIG_X86_X2APIC=y
CONFIG_X86_MPPARSE=y
CONFIG_X86_EXTENDED_PLATFORM=y
# CONFIG_X86_VSMP is not set
# CONFIG_X86_UV is not set
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_PARAVIRT_GUEST=y
CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=128
CONFIG_XEN_SAVE_RESTORE=y
CONFIG_XEN_DEBUG_FS=y
CONFIG_KVM_CLOCK=y
CONFIG_KVM_GUEST=y
CONFIG_PARAVIRT=y
# CONFIG_PARAVIRT_SPINLOCKS is not set
CONFIG_PARAVIRT_CLOCK=y
# CONFIG_PARAVIRT_DEBUG is not set
CONFIG_NO_BOOTMEM=y
# CONFIG_MEMTEST is not set
# CONFIG_MK8 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
CONFIG_GENERIC_CPU=y
CONFIG_X86_CPU=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=7
CONFIG_X86_CMPXCHG=y
CONFIG_CMPXCHG_LOCAL=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
CONFIG_CALGARY_IOMMU=y
CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT=y
CONFIG_AMD_IOMMU=y
CONFIG_AMD_IOMMU_STATS=y
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
CONFIG_IOMMU_API=y
# CONFIG_MAXSMP is not set
CONFIG_NR_CPUS=256
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
CONFIG_IRQ_TIME_ACCOUNTING=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
# CONFIG_X86_MCE_INJECT is not set
CONFIG_X86_THERMAL_VECTOR=y
CONFIG_I8K=m
CONFIG_MICROCODE=m
CONFIG_MICROCODE_INTEL=y
CONFIG_MICROCODE_AMD=y
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
CONFIG_NUMA=y
CONFIG_AMD_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
CONFIG_NODES_SPAN_OTHER_NODES=y
# CONFIG_NUMA_EMU is not set
CONFIG_NODES_SHIFT=9
CONFIG_ARCH_PROC_KCORE_TEXT=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_MEMORY_PROBE=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_NEED_MULTIPLE_NODES=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_SPARSE=y
CONFIG_MEMORY_HOTREMOVE=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
CONFIG_MEMORY_FAILURE=y
CONFIG_HWPOISON_INJECT=m
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
# CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set
# CONFIG_X86_CHECK_BIOS_CORRUPTION is not set
CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_EFI=y
CONFIG_SECCOMP=y
CONFIG_CC_STACKPROTECTOR=y
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_KEXEC_JUMP=y
CONFIG_PHYSICAL_START=0x1000000
CONFIG_RELOCATABLE=y
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
# CONFIG_COMPAT_VDSO is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
CONFIG_HAVE_ARCH_EARLY_PFN_TO_NID=y
CONFIG_USE_PERCPU_NUMA_NODE_ID=y
#
# Power management and ACPI options
#
CONFIG_ARCH_HIBERNATION_HEADER=y
CONFIG_PM=y
CONFIG_PM_DEBUG=y
CONFIG_PM_ADVANCED_DEBUG=y
# CONFIG_PM_VERBOSE is not set
CONFIG_CAN_PM_TRACE=y
CONFIG_PM_TRACE=y
CONFIG_PM_TRACE_RTC=y
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_SLEEP=y
# CONFIG_PM_SLEEP_ADVANCED_DEBUG is not set
CONFIG_SUSPEND=y
# CONFIG_PM_TEST_SUSPEND is not set
CONFIG_SUSPEND_FREEZER=y
CONFIG_HIBERNATION=y
CONFIG_PM_STD_PARTITION=""
CONFIG_PM_RUNTIME=y
CONFIG_PM_OPS=y
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_PROCFS=y
CONFIG_ACPI_PROCFS_POWER=y
CONFIG_ACPI_POWER_METER=m
CONFIG_ACPI_EC_DEBUGFS=m
# CONFIG_ACPI_PROC_EVENT is not set
CONFIG_ACPI_AC=y
CONFIG_ACPI_BATTERY=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_IPMI=m
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_PROCESSOR_AGGREGATOR=m
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_NUMA=y
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_PCI_SLOT=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
CONFIG_ACPI_HOTPLUG_MEMORY=m
CONFIG_ACPI_SBS=m
CONFIG_ACPI_HED=m
CONFIG_ACPI_APEI=y
CONFIG_ACPI_APEI_GHES=m
# CONFIG_ACPI_APEI_EINJ is not set
# CONFIG_ACPI_APEI_ERST_DEBUG is not set
CONFIG_SFI=y
#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=m
CONFIG_CPU_FREQ_DEBUG=y
CONFIG_CPU_FREQ_STAT=m
CONFIG_CPU_FREQ_STAT_DETAILS=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=m
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=m
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=m
#
# CPUFreq processor drivers
#
CONFIG_X86_PCC_CPUFREQ=m
CONFIG_X86_ACPI_CPUFREQ=m
CONFIG_X86_POWERNOW_K8=m
# CONFIG_X86_SPEEDSTEP_CENTRINO is not set
CONFIG_X86_P4_CLOCKMOD=m
#
# shared options
#
CONFIG_X86_SPEEDSTEP_LIB=m
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y
CONFIG_INTEL_IDLE=y
#
# Memory power savings
#
CONFIG_I7300_IDLE_IOAT_CHANNEL=y
CONFIG_I7300_IDLE=m
#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_XEN=y
CONFIG_PCI_DOMAINS=y
# CONFIG_PCI_CNB20LE_QUIRK is not set
CONFIG_DMAR=y
CONFIG_DMAR_DEFAULT_ON=y
CONFIG_DMAR_FLOPPY_WA=y
CONFIG_INTR_REMAP=y
CONFIG_PCIEPORTBUS=y
CONFIG_HOTPLUG_PCI_PCIE=y
CONFIG_PCIEAER=y
CONFIG_PCIE_ECRC=y
CONFIG_PCIEAER_INJECT=m
CONFIG_PCIEASPM=y
# CONFIG_PCIEASPM_DEBUG is not set
CONFIG_PCIE_PME=y
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_PCI_MSI_DEFAULT_ON=y
# CONFIG_PCI_DEBUG is not set
CONFIG_PCI_STUB=y
CONFIG_XEN_PCIDEV_FRONTEND=m
CONFIG_HT_IRQ=y
CONFIG_PCI_IOV=y
CONFIG_PCI_IOAPIC=y
CONFIG_ISA_DMA_API=y
CONFIG_AMD_NB=y
CONFIG_PCCARD=y
CONFIG_PCMCIA=y
CONFIG_PCMCIA_LOAD_CIS=y
CONFIG_CARDBUS=y
#
# PC-card bridges
#
CONFIG_YENTA=m
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
CONFIG_PD6729=m
CONFIG_I82092=m
CONFIG_PCCARD_NONSTATIC=y
CONFIG_HOTPLUG_PCI=y
CONFIG_HOTPLUG_PCI_FAKE=m
CONFIG_HOTPLUG_PCI_ACPI=y
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
# CONFIG_HOTPLUG_PCI_CPCI is not set
CONFIG_HOTPLUG_PCI_SHPC=m
#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
# CONFIG_HAVE_AOUT is not set
CONFIG_BINFMT_MISC=y
CONFIG_IA32_EMULATION=y
# CONFIG_IA32_AOUT is not set
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_HAVE_TEXT_POKE_SMP=y
CONFIG_NET=y
CONFIG_COMPAT_NETLINK_MESSAGES=y
#
# Networking options
#
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_XFRM_USER=y
CONFIG_XFRM_SUB_POLICY=y
CONFIG_XFRM_MIGRATE=y
CONFIG_XFRM_STATISTICS=y
CONFIG_XFRM_IPCOMP=m
CONFIG_NET_KEY=m
CONFIG_NET_KEY_MIGRATE=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_ASK_IP_FIB_HASH=y
# CONFIG_IP_FIB_TRIE is not set
CONFIG_IP_FIB_HASH=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE_DEMUX=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
CONFIG_IP_MROUTE_MULTIPLE_TABLES=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
CONFIG_ARPD=y
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_XFRM_MODE_TRANSPORT=m
CONFIG_INET_XFRM_MODE_TUNNEL=m
CONFIG_INET_XFRM_MODE_BEET=m
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=m
CONFIG_INET_TCP_DIAG=m
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=m
CONFIG_TCP_CONG_CUBIC=y
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_TCP_CONG_HSTCP=m
CONFIG_TCP_CONG_HYBLA=m
CONFIG_TCP_CONG_VEGAS=m
CONFIG_TCP_CONG_SCALABLE=m
CONFIG_TCP_CONG_LP=m
CONFIG_TCP_CONG_VENO=m
CONFIG_TCP_CONG_YEAH=m
CONFIG_TCP_CONG_ILLINOIS=m
CONFIG_DEFAULT_CUBIC=y
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_TCP_MD5SIG=y
CONFIG_IPV6=m
CONFIG_IPV6_PRIVACY=y
CONFIG_IPV6_ROUTER_PREF=y
CONFIG_IPV6_ROUTE_INFO=y
CONFIG_IPV6_OPTIMISTIC_DAD=y
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
CONFIG_INET6_IPCOMP=m
CONFIG_IPV6_MIP6=m
CONFIG_INET6_XFRM_TUNNEL=m
CONFIG_INET6_TUNNEL=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=m
CONFIG_INET6_XFRM_MODE_TUNNEL=m
CONFIG_INET6_XFRM_MODE_BEET=m
CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION=m
CONFIG_IPV6_SIT=m
CONFIG_IPV6_SIT_6RD=y
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_IPV6_TUNNEL=m
CONFIG_IPV6_MULTIPLE_TABLES=y
CONFIG_IPV6_SUBTREES=y
CONFIG_IPV6_MROUTE=y
CONFIG_IPV6_MROUTE_MULTIPLE_TABLES=y
CONFIG_IPV6_PIMSM_V2=y
CONFIG_NETLABEL=y
CONFIG_NETWORK_SECMARK=y
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=y
#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=y
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_ZONES=y
CONFIG_NF_CONNTRACK_EVENTS=y
CONFIG_NF_CT_PROTO_DCCP=m
CONFIG_NF_CT_PROTO_GRE=m
CONFIG_NF_CT_PROTO_SCTP=m
CONFIG_NF_CT_PROTO_UDPLITE=m
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SANE=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_CT_NETLINK=m
CONFIG_NETFILTER_TPROXY=m
CONFIG_NETFILTER_XTABLES=y
#
# Xtables combined modules
#
CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m
#
# Xtables targets
#
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
CONFIG_NETFILTER_XT_TARGET_CT=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m
CONFIG_NETFILTER_XT_TARGET_LED=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
CONFIG_NETFILTER_XT_TARGET_RATEEST=m
CONFIG_NETFILTER_XT_TARGET_TEE=m
CONFIG_NETFILTER_XT_TARGET_TPROXY=m
CONFIG_NETFILTER_XT_TARGET_TRACE=m
CONFIG_NETFILTER_XT_TARGET_SECMARK=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m
#
# Xtables matches
#
CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
CONFIG_NETFILTER_XT_MATCH_CPU=m
CONFIG_NETFILTER_XT_MATCH_DCCP=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_HL=m
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_IPVS=m
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_OSF=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_RATEEST=m
CONFIG_NETFILTER_XT_MATCH_REALM=m
CONFIG_NETFILTER_XT_MATCH_RECENT=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_SOCKET=m
CONFIG_NETFILTER_XT_MATCH_STATE=y
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_TIME=m
CONFIG_NETFILTER_XT_MATCH_U32=m
CONFIG_IP_VS=m
# CONFIG_IP_VS_IPV6 is not set
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12
#
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_AH_ESP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y
CONFIG_IP_VS_PROTO_SCTP=y
#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m
#
# IPVS application helper
#
CONFIG_IP_VS_FTP=m
CONFIG_IP_VS_NFCT=y
CONFIG_IP_VS_PE_SIP=m
#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=y
CONFIG_NF_CONNTRACK_IPV4=y
# CONFIG_NF_CONNTRACK_PROC_COMPAT is not set
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_MATCH_ADDRTYPE=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_NF_NAT_SNMP_BASIC=m
CONFIG_NF_NAT_PROTO_DCCP=m
CONFIG_NF_NAT_PROTO_GRE=m
CONFIG_NF_NAT_PROTO_UDPLITE=m
CONFIG_NF_NAT_PROTO_SCTP=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NF_NAT_AMANDA=m
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_NF_NAT_SIP=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_SECURITY=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
#
# IPv6: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV6=m
CONFIG_NF_CONNTRACK_IPV6=m
CONFIG_IP6_NF_QUEUE=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_AH=m
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_IP6_NF_MATCH_FRAG=m
CONFIG_IP6_NF_MATCH_OPTS=m
CONFIG_IP6_NF_MATCH_HL=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_MATCH_MH=m
CONFIG_IP6_NF_MATCH_RT=m
CONFIG_IP6_NF_TARGET_HL=m
CONFIG_IP6_NF_TARGET_LOG=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_REJECT=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_RAW=m
CONFIG_IP6_NF_SECURITY=m
#
# DECnet: Netfilter Configuration
#
# CONFIG_DECNET_NF_GRABULATOR is not set
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
CONFIG_BRIDGE_EBT_IP6=m
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_BRIDGE_EBT_ULOG=m
CONFIG_BRIDGE_EBT_NFLOG=m
CONFIG_IP_DCCP=m
CONFIG_INET_DCCP_DIAG=m
#
# DCCP CCIDs Configuration (EXPERIMENTAL)
#
# CONFIG_IP_DCCP_CCID2_DEBUG is not set
CONFIG_IP_DCCP_CCID3=y
# CONFIG_IP_DCCP_CCID3_DEBUG is not set
CONFIG_IP_DCCP_TFRC_LIB=y
#
# DCCP Kernel Hacking
#
# CONFIG_IP_DCCP_DEBUG is not set
CONFIG_NET_DCCPPROBE=m
CONFIG_IP_SCTP=m
CONFIG_NET_SCTPPROBE=m
# CONFIG_SCTP_DBG_MSG is not set
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_HMAC_NONE is not set
CONFIG_SCTP_HMAC_SHA1=y
# CONFIG_SCTP_HMAC_MD5 is not set
CONFIG_RDS=m
CONFIG_RDS_RDMA=m
CONFIG_RDS_TCP=m
# CONFIG_RDS_DEBUG is not set
# CONFIG_TIPC is not set
CONFIG_ATM=m
CONFIG_ATM_CLIP=m
# CONFIG_ATM_CLIP_NO_ICMP is not set
CONFIG_ATM_LANE=m
# CONFIG_ATM_MPOA is not set
CONFIG_ATM_BR2684=m
# CONFIG_ATM_BR2684_IPFILTER is not set
CONFIG_L2TP=m
CONFIG_L2TP_DEBUGFS=m
CONFIG_L2TP_V3=y
CONFIG_L2TP_IP=m
CONFIG_L2TP_ETH=m
CONFIG_STP=m
CONFIG_GARP=m
CONFIG_BRIDGE=m
CONFIG_BRIDGE_IGMP_SNOOPING=y
CONFIG_NET_DSA=y
CONFIG_NET_DSA_TAG_DSA=y
CONFIG_NET_DSA_TAG_EDSA=y
CONFIG_NET_DSA_TAG_TRAILER=y
CONFIG_NET_DSA_MV88E6XXX=y
CONFIG_NET_DSA_MV88E6060=y
CONFIG_NET_DSA_MV88E6XXX_NEED_PPU=y
CONFIG_NET_DSA_MV88E6131=y
CONFIG_NET_DSA_MV88E6123_61_65=y
CONFIG_VLAN_8021Q=m
CONFIG_VLAN_8021Q_GVRP=y
CONFIG_DECNET=m
CONFIG_DECNET_ROUTER=y
CONFIG_LLC=m
# CONFIG_LLC2 is not set
CONFIG_IPX=m
# CONFIG_IPX_INTERN is not set
CONFIG_ATALK=m
CONFIG_DEV_APPLETALK=m
CONFIG_IPDDP=m
CONFIG_IPDDP_ENCAP=y
CONFIG_IPDDP_DECAP=y
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
CONFIG_WAN_ROUTER=m
CONFIG_PHONET=m
# CONFIG_PHONET_PIPECTRLR is not set
CONFIG_IEEE802154=m
CONFIG_NET_SCHED=y
#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_ATM=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_MULTIQ=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_DRR=m
CONFIG_NET_SCH_INGRESS=m
#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_CLS_U32_PERF=y
CONFIG_CLS_U32_MARK=y
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
CONFIG_NET_CLS_FLOW=m
CONFIG_NET_CLS_CGROUP=y
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
CONFIG_NET_EMATCH_U32=m
CONFIG_NET_EMATCH_META=m
CONFIG_NET_EMATCH_TEXT=m
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
CONFIG_NET_ACT_IPT=m
CONFIG_NET_ACT_NAT=m
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
CONFIG_NET_ACT_SKBEDIT=m
CONFIG_NET_ACT_CSUM=m
CONFIG_NET_CLS_IND=y
CONFIG_NET_SCH_FIFO=y
CONFIG_DCB=y
CONFIG_DNS_RESOLVER=y
# CONFIG_BATMAN_ADV is not set
CONFIG_RPS=y
CONFIG_XPS=y
#
# Network testing
#
CONFIG_NET_PKTGEN=m
# CONFIG_NET_TCPPROBE is not set
CONFIG_NET_DROP_MONITOR=y
CONFIG_HAMRADIO=y
#
# Packet Radio protocols
#
CONFIG_AX25=m
CONFIG_AX25_DAMA_SLAVE=y
CONFIG_NETROM=m
CONFIG_ROSE=m
#
# AX.25 network device drivers
#
CONFIG_MKISS=m
CONFIG_6PACK=m
CONFIG_BPQETHER=m
CONFIG_BAYCOM_SER_FDX=m
CONFIG_BAYCOM_SER_HDX=m
CONFIG_BAYCOM_PAR=m
CONFIG_YAM=m
CONFIG_CAN=m
CONFIG_CAN_RAW=m
CONFIG_CAN_BCM=m
#
# CAN Device Drivers
#
CONFIG_CAN_VCAN=m
CONFIG_CAN_SLCAN=m
CONFIG_CAN_DEV=m
CONFIG_CAN_CALC_BITTIMING=y
# CONFIG_PCH_CAN is not set
CONFIG_CAN_SJA1000=m
CONFIG_CAN_SJA1000_PLATFORM=m
CONFIG_CAN_EMS_PCI=m
CONFIG_CAN_KVASER_PCI=m
CONFIG_CAN_PLX_PCI=m
#
# CAN USB interfaces
#
CONFIG_CAN_EMS_USB=m
CONFIG_CAN_ESD_USB2=m
CONFIG_CAN_SOFTING=m
CONFIG_CAN_SOFTING_CS=m
# CONFIG_CAN_DEBUG_DEVICES is not set
CONFIG_IRDA=m
#
# IrDA protocols
#
CONFIG_IRLAN=m
CONFIG_IRNET=m
CONFIG_IRCOMM=m
# CONFIG_IRDA_ULTRA is not set
#
# IrDA options
#
CONFIG_IRDA_CACHE_LAST_LSAP=y
CONFIG_IRDA_FAST_RR=y
# CONFIG_IRDA_DEBUG is not set
#
# Infrared-port device drivers
#
#
# SIR device drivers
#
CONFIG_IRTTY_SIR=m
#
# Dongle support
#
CONFIG_DONGLE=y
CONFIG_ESI_DONGLE=m
CONFIG_ACTISYS_DONGLE=m
CONFIG_TEKRAM_DONGLE=m
CONFIG_TOIM3232_DONGLE=m
CONFIG_LITELINK_DONGLE=m
CONFIG_MA600_DONGLE=m
CONFIG_GIRBIL_DONGLE=m
CONFIG_MCP2120_DONGLE=m
CONFIG_OLD_BELKIN_DONGLE=m
CONFIG_ACT200L_DONGLE=m
CONFIG_KINGSUN_DONGLE=m
CONFIG_KSDAZZLE_DONGLE=m
CONFIG_KS959_DONGLE=m
#
# FIR device drivers
#
CONFIG_USB_IRDA=m
CONFIG_SIGMATEL_FIR=m
CONFIG_NSC_FIR=m
CONFIG_WINBOND_FIR=m
CONFIG_SMC_IRCC_FIR=m
CONFIG_ALI_FIR=m
CONFIG_VLSI_FIR=m
CONFIG_VIA_FIR=m
CONFIG_MCS_FIR=m
CONFIG_BT=m
CONFIG_BT_L2CAP=m
CONFIG_BT_SCO=m
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_BNEP_MC_FILTER=y
CONFIG_BT_BNEP_PROTO_FILTER=y
CONFIG_BT_CMTP=m
CONFIG_BT_HIDP=m
#
# Bluetooth device drivers
#
CONFIG_BT_HCIBTUSB=m
CONFIG_BT_HCIBTSDIO=m
CONFIG_BT_HCIUART=m
CONFIG_BT_HCIUART_H4=y
CONFIG_BT_HCIUART_BCSP=y
CONFIG_BT_HCIUART_ATH3K=y
CONFIG_BT_HCIUART_LL=y
CONFIG_BT_HCIBCM203X=m
CONFIG_BT_HCIBPA10X=m
CONFIG_BT_HCIBFUSB=m
CONFIG_BT_HCIDTL1=m
CONFIG_BT_HCIBT3C=m
CONFIG_BT_HCIBLUECARD=m
CONFIG_BT_HCIBTUART=m
CONFIG_BT_HCIVHCI=m
CONFIG_BT_MRVL=m
CONFIG_BT_MRVL_SDIO=m
CONFIG_BT_ATH3K=m
# CONFIG_AF_RXRPC is not set
CONFIG_FIB_RULES=y
CONFIG_WIRELESS=y
CONFIG_WIRELESS_EXT=y
CONFIG_WEXT_CORE=y
CONFIG_WEXT_PROC=y
CONFIG_WEXT_SPY=y
CONFIG_WEXT_PRIV=y
CONFIG_CFG80211=m
# CONFIG_NL80211_TESTMODE is not set
# CONFIG_CFG80211_DEVELOPER_WARNINGS is not set
# CONFIG_CFG80211_REG_DEBUG is not set
CONFIG_CFG80211_DEFAULT_PS=y
CONFIG_CFG80211_DEBUGFS=y
# CONFIG_CFG80211_INTERNAL_REGDB is not set
CONFIG_CFG80211_WEXT=y
CONFIG_WIRELESS_EXT_SYSFS=y
CONFIG_LIB80211=m
CONFIG_LIB80211_CRYPT_WEP=m
CONFIG_LIB80211_CRYPT_CCMP=m
CONFIG_LIB80211_CRYPT_TKIP=m
# CONFIG_LIB80211_DEBUG is not set
CONFIG_MAC80211=m
CONFIG_MAC80211_HAS_RC=y
CONFIG_MAC80211_RC_MINSTREL=y
CONFIG_MAC80211_RC_MINSTREL_HT=y
CONFIG_MAC80211_RC_DEFAULT_MINSTREL=y
CONFIG_MAC80211_RC_DEFAULT="minstrel_ht"
CONFIG_MAC80211_MESH=y
CONFIG_MAC80211_LEDS=y
CONFIG_MAC80211_DEBUGFS=y
# CONFIG_MAC80211_DEBUG_MENU is not set
CONFIG_WIMAX=m
CONFIG_WIMAX_DEBUG_LEVEL=8
CONFIG_RFKILL=m
CONFIG_RFKILL_LEDS=y
CONFIG_RFKILL_INPUT=y
CONFIG_NET_9P=m
CONFIG_NET_9P_VIRTIO=m
CONFIG_NET_9P_RDMA=m
# CONFIG_NET_9P_DEBUG is not set
# CONFIG_CAIF is not set
CONFIG_CEPH_LIB=m
# CONFIG_CEPH_LIB_PRETTYDEBUG is not set
#
# Device Drivers
#
#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH=""
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
# CONFIG_FIRMWARE_IN_KERNEL is not set
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_DEBUG_DRIVER is not set
CONFIG_DEBUG_DEVRES=y
CONFIG_SYS_HYPERVISOR=y
CONFIG_CONNECTOR=y
CONFIG_PROC_EVENTS=y
CONFIG_MTD=m
# CONFIG_MTD_DEBUG is not set
# CONFIG_MTD_TESTS is not set
CONFIG_MTD_CONCAT=m
CONFIG_MTD_PARTITIONS=y
CONFIG_MTD_REDBOOT_PARTS=m
CONFIG_MTD_REDBOOT_DIRECTORY_BLOCK=-1
# CONFIG_MTD_REDBOOT_PARTS_UNALLOCATED is not set
# CONFIG_MTD_REDBOOT_PARTS_READONLY is not set
CONFIG_MTD_AR7_PARTS=m
#
# User Modules And Translation Layers
#
CONFIG_MTD_CHAR=m
CONFIG_MTD_BLKDEVS=m
CONFIG_MTD_BLOCK=m
CONFIG_MTD_BLOCK_RO=m
CONFIG_FTL=m
CONFIG_NFTL=m
CONFIG_NFTL_RW=y
CONFIG_INFTL=m
CONFIG_RFD_FTL=m
CONFIG_SSFDC=m
# CONFIG_SM_FTL is not set
CONFIG_MTD_OOPS=m
#
# RAM/ROM/Flash chip drivers
#
CONFIG_MTD_CFI=m
CONFIG_MTD_JEDECPROBE=m
CONFIG_MTD_GEN_PROBE=m
# CONFIG_MTD_CFI_ADV_OPTIONS is not set
CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
# CONFIG_MTD_MAP_BANK_WIDTH_8 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_16 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_32 is not set
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y
# CONFIG_MTD_CFI_I4 is not set
# CONFIG_MTD_CFI_I8 is not set
CONFIG_MTD_CFI_INTELEXT=m
CONFIG_MTD_CFI_AMDSTD=m
CONFIG_MTD_CFI_STAA=m
CONFIG_MTD_CFI_UTIL=m
CONFIG_MTD_RAM=m
CONFIG_MTD_ROM=m
CONFIG_MTD_ABSENT=m
#
# Mapping drivers for chip access
#
CONFIG_MTD_COMPLEX_MAPPINGS=y
# CONFIG_MTD_PHYSMAP is not set
CONFIG_MTD_SC520CDP=m
CONFIG_MTD_NETSC520=m
CONFIG_MTD_TS5500=m
# CONFIG_MTD_SBC_GXX is not set
# CONFIG_MTD_AMD76XROM is not set
# CONFIG_MTD_ICHXROM is not set
CONFIG_MTD_ESB2ROM=m
CONFIG_MTD_CK804XROM=m
CONFIG_MTD_SCB2_FLASH=m
# CONFIG_MTD_NETtel is not set
# CONFIG_MTD_L440GX is not set
CONFIG_MTD_PCI=m
# CONFIG_MTD_PCMCIA is not set
# CONFIG_MTD_GPIO_ADDR is not set
# CONFIG_MTD_INTEL_VR_NOR is not set
# CONFIG_MTD_PLATRAM is not set
#
# Self-contained MTD device drivers
#
CONFIG_MTD_PMC551=m
# CONFIG_MTD_PMC551_BUGFIX is not set
# CONFIG_MTD_PMC551_DEBUG is not set
# CONFIG_MTD_SLRAM is not set
# CONFIG_MTD_PHRAM is not set
CONFIG_MTD_MTDRAM=m
CONFIG_MTDRAM_TOTAL_SIZE=4096
CONFIG_MTDRAM_ERASE_SIZE=128
CONFIG_MTD_BLOCK2MTD=m
#
# Disk-On-Chip Device Drivers
#
# CONFIG_MTD_DOC2000 is not set
# CONFIG_MTD_DOC2001 is not set
# CONFIG_MTD_DOC2001PLUS is not set
CONFIG_MTD_NAND_ECC=m
CONFIG_MTD_NAND_ECC_SMC=y
CONFIG_MTD_NAND=m
# CONFIG_MTD_NAND_VERIFY_WRITE is not set
CONFIG_MTD_SM_COMMON=m
# CONFIG_MTD_NAND_MUSEUM_IDS is not set
# CONFIG_MTD_NAND_DENALI is not set
CONFIG_MTD_NAND_IDS=m
CONFIG_MTD_NAND_RICOH=m
CONFIG_MTD_NAND_DISKONCHIP=m
# CONFIG_MTD_NAND_DISKONCHIP_PROBE_ADVANCED is not set
CONFIG_MTD_NAND_DISKONCHIP_PROBE_ADDRESS=0
# CONFIG_MTD_NAND_DISKONCHIP_BBTWRITE is not set
# CONFIG_MTD_NAND_CAFE is not set
CONFIG_MTD_NAND_NANDSIM=m
# CONFIG_MTD_NAND_PLATFORM is not set
CONFIG_MTD_ALAUDA=m
# CONFIG_MTD_ONENAND is not set
#
# LPDDR flash memory drivers
#
CONFIG_MTD_LPDDR=m
CONFIG_MTD_QINFO_PROBE=m
CONFIG_MTD_UBI=m
CONFIG_MTD_UBI_WL_THRESHOLD=4096
CONFIG_MTD_UBI_BEB_RESERVE=1
# CONFIG_MTD_UBI_GLUEBI is not set
#
# UBI debugging options
#
# CONFIG_MTD_UBI_DEBUG is not set
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_SERIAL=m
# CONFIG_PARPORT_PC_FIFO is not set
# CONFIG_PARPORT_PC_SUPERIO is not set
CONFIG_PARPORT_PC_PCMCIA=m
# CONFIG_PARPORT_GSC is not set
# CONFIG_PARPORT_AX88796 is not set
CONFIG_PARPORT_1284=y
CONFIG_PARPORT_NOT_PC=y
CONFIG_PNP=y
# CONFIG_PNP_DEBUG_MESSAGES is not set
#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=m
# CONFIG_PARIDE is not set
CONFIG_BLK_CPQ_DA=m
CONFIG_BLK_CPQ_CISS_DA=m
CONFIG_CISS_SCSI_TAPE=y
CONFIG_BLK_DEV_DAC960=m
CONFIG_BLK_DEV_UMEM=m
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_DRBD=m
# CONFIG_DRBD_FAULT_INJECTION is not set
CONFIG_BLK_DEV_NBD=m
CONFIG_BLK_DEV_OSD=m
CONFIG_BLK_DEV_SX8=m
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=16384
# CONFIG_BLK_DEV_XIP is not set
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
CONFIG_ATA_OVER_ETH=m
CONFIG_XEN_BLKDEV_FRONTEND=m
CONFIG_VIRTIO_BLK=m
# CONFIG_BLK_DEV_HD is not set
CONFIG_BLK_DEV_RBD=m
CONFIG_MISC_DEVICES=y
# CONFIG_AD525X_DPOT is not set
# CONFIG_IBM_ASM is not set
# CONFIG_PHANTOM is not set
CONFIG_SGI_IOC4=m
CONFIG_TIFM_CORE=m
CONFIG_TIFM_7XX1=m
CONFIG_ICS932S401=m
CONFIG_ENCLOSURE_SERVICES=m
CONFIG_CS5535_MFGPT=m
CONFIG_CS5535_MFGPT_DEFAULT_IRQ=7
CONFIG_CS5535_CLOCK_EVENT_SRC=m
CONFIG_HP_ILO=m
CONFIG_APDS9802ALS=m
CONFIG_ISL29003=m
CONFIG_ISL29020=m
CONFIG_SENSORS_TSL2550=m
# CONFIG_SENSORS_BH1780 is not set
CONFIG_SENSORS_BH1770=m
CONFIG_SENSORS_APDS990X=m
# CONFIG_HMC6352 is not set
# CONFIG_DS1682 is not set
CONFIG_VMWARE_BALLOON=m
# CONFIG_BMP085 is not set
CONFIG_PCH_PHUB=m
# CONFIG_C2PORT is not set
#
# EEPROM support
#
CONFIG_EEPROM_AT24=m
CONFIG_EEPROM_LEGACY=m
CONFIG_EEPROM_MAX6875=m
CONFIG_EEPROM_93CX6=m
CONFIG_CB710_CORE=m
# CONFIG_CB710_DEBUG is not set
CONFIG_CB710_DEBUG_ASSUMPTIONS=y
CONFIG_IWMC3200TOP=m
# CONFIG_IWMC3200TOP_DEBUG is not set
CONFIG_IWMC3200TOP_DEBUGFS=y
#
# Texas Instruments shared transport line discipline
#
# CONFIG_TI_ST is not set
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set
#
# SCSI device support
#
CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_TGT=m
CONFIG_SCSI_NETLINK=y
CONFIG_SCSI_PROC_FS=y
#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_ST=m
CONFIG_CHR_DEV_OSST=m
CONFIG_BLK_DEV_SR=y
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=y
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_ENCLOSURE=m
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SCAN_ASYNC=y
CONFIG_SCSI_WAIT_SCAN=m
#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
CONFIG_SCSI_FC_TGT_ATTRS=y
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_SCSI_SAS_ATTRS=m
CONFIG_SCSI_SAS_LIBSAS=m
CONFIG_SCSI_SAS_ATA=y
CONFIG_SCSI_SAS_HOST_SMP=y
# CONFIG_SCSI_SAS_LIBSAS_DEBUG is not set
CONFIG_SCSI_SRP_ATTRS=m
CONFIG_SCSI_SRP_TGT_ATTRS=y
CONFIG_SCSI_LOWLEVEL=y
CONFIG_ISCSI_TCP=m
CONFIG_ISCSI_BOOT_SYSFS=m
CONFIG_SCSI_CXGB3_ISCSI=m
CONFIG_SCSI_CXGB4_ISCSI=m
CONFIG_SCSI_BNX2_ISCSI=m
CONFIG_BE2ISCSI=m
CONFIG_BLK_DEV_3W_XXXX_RAID=m
CONFIG_SCSI_HPSA=m
CONFIG_SCSI_3W_9XXX=m
CONFIG_SCSI_3W_SAS=m
CONFIG_SCSI_ACARD=m
CONFIG_SCSI_AACRAID=m
CONFIG_SCSI_AIC7XXX=m
CONFIG_AIC7XXX_CMDS_PER_DEVICE=4
CONFIG_AIC7XXX_RESET_DELAY_MS=15000
# CONFIG_AIC7XXX_DEBUG_ENABLE is not set
CONFIG_AIC7XXX_DEBUG_MASK=0
# CONFIG_AIC7XXX_REG_PRETTY_PRINT is not set
CONFIG_SCSI_AIC7XXX_OLD=m
CONFIG_SCSI_AIC79XX=m
CONFIG_AIC79XX_CMDS_PER_DEVICE=4
CONFIG_AIC79XX_RESET_DELAY_MS=15000
# CONFIG_AIC79XX_DEBUG_ENABLE is not set
CONFIG_AIC79XX_DEBUG_MASK=0
# CONFIG_AIC79XX_REG_PRETTY_PRINT is not set
CONFIG_SCSI_AIC94XX=m
# CONFIG_AIC94XX_DEBUG is not set
CONFIG_SCSI_MVSAS=m
# CONFIG_SCSI_MVSAS_DEBUG is not set
# CONFIG_SCSI_DPT_I2O is not set
CONFIG_SCSI_ADVANSYS=m
CONFIG_SCSI_ARCMSR=m
CONFIG_SCSI_ARCMSR_AER=y
CONFIG_MEGARAID_NEWGEN=y
CONFIG_MEGARAID_MM=m
CONFIG_MEGARAID_MAILBOX=m
CONFIG_MEGARAID_LEGACY=m
CONFIG_MEGARAID_SAS=m
CONFIG_SCSI_MPT2SAS=m
CONFIG_SCSI_MPT2SAS_MAX_SGE=128
CONFIG_SCSI_MPT2SAS_LOGGING=y
CONFIG_SCSI_HPTIOP=m
CONFIG_SCSI_BUSLOGIC=m
CONFIG_VMWARE_PVSCSI=m
CONFIG_LIBFC=m
CONFIG_LIBFCOE=m
CONFIG_FCOE=m
CONFIG_FCOE_FNIC=m
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
CONFIG_SCSI_GDTH=m
CONFIG_SCSI_IPS=m
CONFIG_SCSI_INITIO=m
CONFIG_SCSI_INIA100=m
CONFIG_SCSI_PPA=m
CONFIG_SCSI_IMM=m
# CONFIG_SCSI_IZIP_EPP16 is not set
# CONFIG_SCSI_IZIP_SLOW_CTR is not set
CONFIG_SCSI_STEX=m
CONFIG_SCSI_SYM53C8XX_2=m
CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1
CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16
CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
CONFIG_SCSI_SYM53C8XX_MMIO=y
# CONFIG_SCSI_IPR is not set
CONFIG_SCSI_QLOGIC_1280=m
CONFIG_SCSI_QLA_FC=m
CONFIG_SCSI_QLA_ISCSI=m
CONFIG_SCSI_LPFC=m
# CONFIG_SCSI_LPFC_DEBUG_FS is not set
CONFIG_SCSI_DC395x=m
CONFIG_SCSI_DC390T=m
CONFIG_SCSI_DEBUG=m
CONFIG_SCSI_PMCRAID=m
CONFIG_SCSI_PM8001=m
CONFIG_SCSI_SRP=m
CONFIG_SCSI_BFA_FC=m
CONFIG_SCSI_LOWLEVEL_PCMCIA=y
CONFIG_PCMCIA_AHA152X=m
# CONFIG_PCMCIA_FDOMAIN is not set
CONFIG_PCMCIA_QLOGIC=m
CONFIG_PCMCIA_SYM53C500=m
CONFIG_SCSI_DH=y
CONFIG_SCSI_DH_RDAC=m
CONFIG_SCSI_DH_HP_SW=m
CONFIG_SCSI_DH_EMC=m
CONFIG_SCSI_DH_ALUA=m
CONFIG_SCSI_OSD_INITIATOR=m
CONFIG_SCSI_OSD_ULD=m
CONFIG_SCSI_OSD_DPRINT_SENSE=1
# CONFIG_SCSI_OSD_DEBUG is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
CONFIG_SATA_PMP=y
#
# Controllers with non-SFF native interface
#
CONFIG_SATA_AHCI=y
CONFIG_SATA_AHCI_PLATFORM=m
CONFIG_SATA_INIC162X=m
CONFIG_SATA_ACARD_AHCI=m
CONFIG_SATA_SIL24=m
CONFIG_ATA_SFF=y
#
# SFF controllers with custom DMA interface
#
CONFIG_PDC_ADMA=m
CONFIG_SATA_QSTOR=m
CONFIG_SATA_SX4=m
CONFIG_ATA_BMDMA=y
#
# SATA SFF controllers with BMDMA
#
CONFIG_ATA_PIIX=y
CONFIG_SATA_MV=m
CONFIG_SATA_NV=m
CONFIG_SATA_PROMISE=m
CONFIG_SATA_SIL=m
CONFIG_SATA_SIS=m
CONFIG_SATA_SVW=m
CONFIG_SATA_ULI=m
CONFIG_SATA_VIA=m
CONFIG_SATA_VITESSE=m
#
# PATA SFF controllers with BMDMA
#
CONFIG_PATA_ALI=m
CONFIG_PATA_AMD=m
CONFIG_PATA_ARTOP=m
CONFIG_PATA_ATIIXP=m
CONFIG_PATA_ATP867X=m
CONFIG_PATA_CMD64X=m
CONFIG_PATA_CS5520=m
CONFIG_PATA_CS5530=m
CONFIG_PATA_CS5536=m
CONFIG_PATA_CYPRESS=m
CONFIG_PATA_EFAR=m
CONFIG_PATA_HPT366=m
CONFIG_PATA_HPT37X=m
CONFIG_PATA_HPT3X2N=m
CONFIG_PATA_HPT3X3=m
# CONFIG_PATA_HPT3X3_DMA is not set
CONFIG_PATA_IT8213=m
CONFIG_PATA_IT821X=m
CONFIG_PATA_JMICRON=m
CONFIG_PATA_MARVELL=m
CONFIG_PATA_NETCELL=m
CONFIG_PATA_NINJA32=m
CONFIG_PATA_NS87415=m
CONFIG_PATA_OLDPIIX=m
CONFIG_PATA_OPTIDMA=m
CONFIG_PATA_PDC2027X=m
CONFIG_PATA_PDC_OLD=m
# CONFIG_PATA_RADISYS is not set
CONFIG_PATA_RDC=m
# CONFIG_PATA_SC1200 is not set
CONFIG_PATA_SCH=m
CONFIG_PATA_SERVERWORKS=m
CONFIG_PATA_SIL680=m
CONFIG_PATA_SIS=m
CONFIG_PATA_TOSHIBA=m
CONFIG_PATA_TRIFLEX=m
CONFIG_PATA_VIA=m
CONFIG_PATA_WINBOND=m
#
# PIO-only SFF controllers
#
CONFIG_PATA_CMD640_PCI=m
CONFIG_PATA_MPIIX=m
CONFIG_PATA_NS87410=m
CONFIG_PATA_OPTI=m
CONFIG_PATA_PCMCIA=m
# CONFIG_PATA_RZ1000 is not set
#
# Generic fallback / legacy drivers
#
CONFIG_PATA_ACPI=m
CONFIG_ATA_GENERIC=m
# CONFIG_PATA_LEGACY is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_AUTODETECT=y
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
# CONFIG_MULTICORE_RAID456 is not set
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
CONFIG_BLK_DEV_DM=y
CONFIG_DM_DEBUG=y
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=y
CONFIG_DM_MIRROR=y
CONFIG_DM_RAID=m
CONFIG_DM_LOG_USERSPACE=m
CONFIG_DM_ZERO=y
CONFIG_DM_MULTIPATH=m
CONFIG_DM_MULTIPATH_QL=m
CONFIG_DM_MULTIPATH_ST=m
# CONFIG_DM_DELAY is not set
CONFIG_DM_UEVENT=y
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
CONFIG_TCM_PSCSI=m
CONFIG_FUSION=y
CONFIG_FUSION_SPI=m
CONFIG_FUSION_FC=m
CONFIG_FUSION_SAS=m
CONFIG_FUSION_MAX_SGE=40
CONFIG_FUSION_CTL=m
CONFIG_FUSION_LAN=m
CONFIG_FUSION_LOGGING=y
#
# IEEE 1394 (FireWire) support
#
CONFIG_FIREWIRE=m
CONFIG_FIREWIRE_OHCI=m
CONFIG_FIREWIRE_OHCI_DEBUG=y
CONFIG_FIREWIRE_SBP2=m
CONFIG_FIREWIRE_NET=m
CONFIG_FIREWIRE_NOSY=m
# CONFIG_I2O is not set
CONFIG_MACINTOSH_DRIVERS=y
CONFIG_MAC_EMUMOUSEBTN=y
CONFIG_NETDEVICES=y
CONFIG_IFB=m
CONFIG_DUMMY=m
CONFIG_BONDING=m
CONFIG_MACVLAN=m
CONFIG_MACVTAP=m
CONFIG_EQUALIZER=m
CONFIG_TUN=m
CONFIG_VETH=m
CONFIG_NET_SB1000=m
# CONFIG_ARCNET is not set
CONFIG_MII=m
CONFIG_PHYLIB=y
#
# MII PHY device drivers
#
CONFIG_MARVELL_PHY=m
CONFIG_DAVICOM_PHY=m
CONFIG_QSEMI_PHY=m
CONFIG_LXT_PHY=m
CONFIG_CICADA_PHY=m
CONFIG_VITESSE_PHY=m
CONFIG_SMSC_PHY=m
CONFIG_BROADCOM_PHY=m
CONFIG_BCM63XX_PHY=m
CONFIG_ICPLUS_PHY=m
CONFIG_REALTEK_PHY=m
CONFIG_NATIONAL_PHY=m
CONFIG_STE10XP=m
CONFIG_LSI_ET1011C_PHY=m
CONFIG_MICREL_PHY=m
CONFIG_FIXED_PHY=y
CONFIG_MDIO_BITBANG=m
# CONFIG_MDIO_GPIO is not set
CONFIG_NET_ETHERNET=y
CONFIG_HAPPYMEAL=m
CONFIG_SUNGEM=m
CONFIG_CASSINI=m
CONFIG_NET_VENDOR_3COM=y
CONFIG_VORTEX=m
CONFIG_TYPHOON=m
CONFIG_ETHOC=m
CONFIG_DNET=m
CONFIG_NET_TULIP=y
CONFIG_DE2104X=m
CONFIG_DE2104X_DSL=0
CONFIG_TULIP=m
# CONFIG_TULIP_MWI is not set
CONFIG_TULIP_MMIO=y
# CONFIG_TULIP_NAPI is not set
CONFIG_DE4X5=m
CONFIG_WINBOND_840=m
CONFIG_DM9102=m
CONFIG_ULI526X=m
CONFIG_PCMCIA_XIRCOM=m
# CONFIG_HP100 is not set
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
# CONFIG_IBM_NEW_EMAC_NO_FLOW_CTRL is not set
# CONFIG_IBM_NEW_EMAC_MAL_CLR_ICINTSTAT is not set
# CONFIG_IBM_NEW_EMAC_MAL_COMMON_ERR is not set
CONFIG_NET_PCI=y
CONFIG_PCNET32=m
CONFIG_AMD8111_ETH=m
CONFIG_ADAPTEC_STARFIRE=m
CONFIG_KSZ884X_PCI=m
CONFIG_B44=m
CONFIG_B44_PCI_AUTOSELECT=y
CONFIG_B44_PCICORE_AUTOSELECT=y
CONFIG_B44_PCI=y
CONFIG_FORCEDETH=m
CONFIG_E100=m
CONFIG_FEALNX=m
CONFIG_NATSEMI=m
CONFIG_NE2K_PCI=m
CONFIG_8139CP=m
CONFIG_8139TOO=m
# CONFIG_8139TOO_PIO is not set
# CONFIG_8139TOO_TUNE_TWISTER is not set
CONFIG_8139TOO_8129=y
# CONFIG_8139_OLD_RX_RESET is not set
CONFIG_R6040=m
CONFIG_SIS900=m
CONFIG_EPIC100=m
CONFIG_SMSC9420=m
CONFIG_SUNDANCE=m
# CONFIG_SUNDANCE_MMIO is not set
CONFIG_TLAN=m
# CONFIG_KS8842 is not set
# CONFIG_KS8851_MLL is not set
CONFIG_VIA_RHINE=m
CONFIG_VIA_RHINE_MMIO=y
CONFIG_SC92031=m
CONFIG_NET_POCKET=y
CONFIG_ATP=m
CONFIG_DE600=m
CONFIG_DE620=m
CONFIG_ATL2=m
CONFIG_NETDEV_1000=y
CONFIG_ACENIC=m
# CONFIG_ACENIC_OMIT_TIGON_I is not set
CONFIG_DL2K=m
CONFIG_E1000=m
CONFIG_E1000E=m
CONFIG_IP1000=m
CONFIG_IGB=m
CONFIG_IGB_DCA=y
CONFIG_IGBVF=m
CONFIG_NS83820=m
CONFIG_HAMACHI=m
CONFIG_YELLOWFIN=m
CONFIG_R8169=m
CONFIG_R8169_VLAN=y
CONFIG_SIS190=m
CONFIG_SKGE=m
# CONFIG_SKGE_DEBUG is not set
CONFIG_SKY2=m
# CONFIG_SKY2_DEBUG is not set
CONFIG_VIA_VELOCITY=m
CONFIG_TIGON3=m
CONFIG_BNX2=m
CONFIG_CNIC=m
CONFIG_QLA3XXX=m
CONFIG_ATL1=m
CONFIG_ATL1E=m
CONFIG_ATL1C=m
CONFIG_JME=m
CONFIG_STMMAC_ETH=m
# CONFIG_STMMAC_DA is not set
# CONFIG_STMMAC_DUAL_MAC is not set
CONFIG_PCH_GBE=m
CONFIG_NETDEV_10000=y
CONFIG_MDIO=m
CONFIG_CHELSIO_T1=m
CONFIG_CHELSIO_T1_1G=y
CONFIG_CHELSIO_T3_DEPENDS=y
CONFIG_CHELSIO_T3=m
CONFIG_CHELSIO_T4_DEPENDS=y
CONFIG_CHELSIO_T4=m
CONFIG_CHELSIO_T4VF_DEPENDS=y
CONFIG_CHELSIO_T4VF=m
CONFIG_ENIC=m
CONFIG_IXGBE=m
CONFIG_IXGBE_DCA=y
CONFIG_IXGBE_DCB=y
CONFIG_IXGBEVF=m
CONFIG_IXGB=m
CONFIG_S2IO=m
CONFIG_VXGE=m
# CONFIG_VXGE_DEBUG_TRACE_ALL is not set
CONFIG_MYRI10GE=m
CONFIG_MYRI10GE_DCA=y
CONFIG_NETXEN_NIC=m
CONFIG_NIU=m
CONFIG_MLX4_EN=m
CONFIG_MLX4_CORE=m
CONFIG_MLX4_DEBUG=y
CONFIG_TEHUTI=m
CONFIG_BNX2X=m
CONFIG_QLCNIC=m
CONFIG_QLGE=m
CONFIG_BNA=m
CONFIG_SFC=m
CONFIG_SFC_MTD=y
CONFIG_BE2NET=m
# CONFIG_TR is not set
CONFIG_WLAN=y
# CONFIG_PCMCIA_RAYCS is not set
CONFIG_LIBERTAS_THINFIRM=m
# CONFIG_LIBERTAS_THINFIRM_DEBUG is not set
CONFIG_LIBERTAS_THINFIRM_USB=m
CONFIG_AIRO=m
CONFIG_ATMEL=m
CONFIG_PCI_ATMEL=m
CONFIG_PCMCIA_ATMEL=m
CONFIG_AT76C50X_USB=m
CONFIG_AIRO_CS=m
CONFIG_PCMCIA_WL3501=m
# CONFIG_PRISM54 is not set
CONFIG_USB_ZD1201=m
CONFIG_USB_NET_RNDIS_WLAN=m
CONFIG_RTL8180=m
CONFIG_RTL8187=m
CONFIG_RTL8187_LEDS=y
CONFIG_ADM8211=m
CONFIG_MAC80211_HWSIM=m
CONFIG_MWL8K=m
CONFIG_ATH_COMMON=m
# CONFIG_ATH_DEBUG is not set
CONFIG_ATH5K=m
CONFIG_ATH5K_DEBUG=y
CONFIG_ATH5K_PCI=y
CONFIG_ATH9K_HW=m
CONFIG_ATH9K_COMMON=m
CONFIG_ATH9K=m
CONFIG_ATH9K_DEBUGFS=y
CONFIG_ATH9K_RATE_CONTROL=y
CONFIG_ATH9K_HTC=m
# CONFIG_ATH9K_HTC_DEBUGFS is not set
CONFIG_AR9170_USB=m
CONFIG_AR9170_LEDS=y
CONFIG_CARL9170=m
CONFIG_CARL9170_LEDS=y
# CONFIG_CARL9170_DEBUGFS is not set
CONFIG_CARL9170_WPC=y
CONFIG_B43=m
CONFIG_B43_PCI_AUTOSELECT=y
CONFIG_B43_PCICORE_AUTOSELECT=y
CONFIG_B43_PCMCIA=y
CONFIG_B43_SDIO=y
CONFIG_B43_PIO=y
CONFIG_B43_PHY_N=y
CONFIG_B43_PHY_LP=y
CONFIG_B43_LEDS=y
CONFIG_B43_HWRNG=y
# CONFIG_B43_DEBUG is not set
CONFIG_B43LEGACY=m
CONFIG_B43LEGACY_PCI_AUTOSELECT=y
CONFIG_B43LEGACY_PCICORE_AUTOSELECT=y
CONFIG_B43LEGACY_LEDS=y
CONFIG_B43LEGACY_HWRNG=y
# CONFIG_B43LEGACY_DEBUG is not set
CONFIG_B43LEGACY_DMA=y
CONFIG_B43LEGACY_PIO=y
CONFIG_B43LEGACY_DMA_AND_PIO_MODE=y
# CONFIG_B43LEGACY_DMA_MODE is not set
# CONFIG_B43LEGACY_PIO_MODE is not set
CONFIG_HOSTAP=m
CONFIG_HOSTAP_FIRMWARE=y
CONFIG_HOSTAP_FIRMWARE_NVRAM=y
CONFIG_HOSTAP_PLX=m
CONFIG_HOSTAP_PCI=m
CONFIG_HOSTAP_CS=m
CONFIG_IPW2100=m
CONFIG_IPW2100_MONITOR=y
# CONFIG_IPW2100_DEBUG is not set
CONFIG_IPW2200=m
CONFIG_IPW2200_MONITOR=y
CONFIG_IPW2200_RADIOTAP=y
CONFIG_IPW2200_PROMISCUOUS=y
CONFIG_IPW2200_QOS=y
# CONFIG_IPW2200_DEBUG is not set
CONFIG_LIBIPW=m
# CONFIG_LIBIPW_DEBUG is not set
CONFIG_IWLWIFI=m
#
# Debugging Options
#
CONFIG_IWLWIFI_DEBUG=y
CONFIG_IWLWIFI_DEBUGFS=y
# CONFIG_IWLWIFI_DEBUG_EXPERIMENTAL_UCODE is not set
# CONFIG_IWLWIFI_DEVICE_TRACING is not set
CONFIG_IWLAGN=m
CONFIG_IWL4965=y
CONFIG_IWL5000=y
CONFIG_IWL3945=m
CONFIG_IWM=m
# CONFIG_IWM_DEBUG is not set
# CONFIG_IWM_TRACING is not set
CONFIG_LIBERTAS=m
CONFIG_LIBERTAS_USB=m
CONFIG_LIBERTAS_CS=m
CONFIG_LIBERTAS_SDIO=m
# CONFIG_LIBERTAS_DEBUG is not set
CONFIG_LIBERTAS_MESH=y
CONFIG_HERMES=m
# CONFIG_HERMES_PRISM is not set
CONFIG_HERMES_CACHE_FW_ON_INIT=y
CONFIG_PLX_HERMES=m
CONFIG_TMD_HERMES=m
CONFIG_NORTEL_HERMES=m
CONFIG_PCMCIA_HERMES=m
CONFIG_PCMCIA_SPECTRUM=m
CONFIG_ORINOCO_USB=m
CONFIG_P54_COMMON=m
CONFIG_P54_USB=m
CONFIG_P54_PCI=m
CONFIG_P54_LEDS=y
CONFIG_RT2X00=m
CONFIG_RT2400PCI=m
CONFIG_RT2500PCI=m
CONFIG_RT61PCI=m
CONFIG_RT2800PCI=m
CONFIG_RT2800PCI_RT33XX=y
CONFIG_RT2800PCI_RT35XX=y
CONFIG_RT2500USB=m
CONFIG_RT73USB=m
CONFIG_RT2800USB=m
CONFIG_RT2800USB_RT33XX=y
CONFIG_RT2800USB_RT35XX=y
CONFIG_RT2800USB_UNKNOWN=y
CONFIG_RT2800_LIB=m
CONFIG_RT2X00_LIB_PCI=m
CONFIG_RT2X00_LIB_USB=m
CONFIG_RT2X00_LIB=m
CONFIG_RT2X00_LIB_HT=y
CONFIG_RT2X00_LIB_FIRMWARE=y
CONFIG_RT2X00_LIB_CRYPTO=y
CONFIG_RT2X00_LIB_LEDS=y
CONFIG_RT2X00_LIB_DEBUGFS=y
# CONFIG_RT2X00_DEBUG is not set
CONFIG_RTL8192CE=m
CONFIG_RTLWIFI=m
CONFIG_WL1251=m
CONFIG_WL1251_SDIO=m
CONFIG_WL12XX_MENU=m
CONFIG_WL12XX=m
# CONFIG_WL12XX_HT is not set
CONFIG_WL12XX_SDIO=m
# CONFIG_WL12XX_SDIO_TEST is not set
CONFIG_WL12XX_PLATFORM_DATA=y
CONFIG_ZD1211RW=m
# CONFIG_ZD1211RW_DEBUG is not set
#
# WiMAX Wireless Broadband devices
#
CONFIG_WIMAX_I2400M=m
CONFIG_WIMAX_I2400M_USB=m
CONFIG_WIMAX_I2400M_SDIO=m
CONFIG_WIMAX_IWMC3200_SDIO=y
CONFIG_WIMAX_I2400M_DEBUG_LEVEL=8
#
# USB Network Adapters
#
CONFIG_USB_CATC=m
CONFIG_USB_KAWETH=m
CONFIG_USB_PEGASUS=m
CONFIG_USB_RTL8150=m
CONFIG_USB_USBNET=m
CONFIG_USB_NET_AX8817X=m
CONFIG_USB_NET_CDCETHER=m
CONFIG_USB_NET_CDC_EEM=m
CONFIG_USB_NET_CDC_NCM=m
CONFIG_USB_NET_DM9601=m
CONFIG_USB_NET_SMSC75XX=m
CONFIG_USB_NET_SMSC95XX=m
CONFIG_USB_NET_GL620A=m
CONFIG_USB_NET_NET1080=m
CONFIG_USB_NET_PLUSB=m
CONFIG_USB_NET_MCS7830=m
CONFIG_USB_NET_RNDIS_HOST=m
CONFIG_USB_NET_CDC_SUBSET=m
CONFIG_USB_ALI_M5632=y
CONFIG_USB_AN2720=y
CONFIG_USB_BELKIN=y
CONFIG_USB_ARMLINUX=y
CONFIG_USB_EPSON2888=y
CONFIG_USB_KC2190=y
CONFIG_USB_NET_ZAURUS=m
CONFIG_USB_NET_CX82310_ETH=m
CONFIG_USB_HSO=m
CONFIG_USB_NET_INT51X1=m
CONFIG_USB_CDC_PHONET=m
CONFIG_USB_IPHETH=m
CONFIG_USB_SIERRA_NET=m
CONFIG_NET_PCMCIA=y
CONFIG_PCMCIA_3C589=m
CONFIG_PCMCIA_3C574=m
CONFIG_PCMCIA_FMVJ18X=m
CONFIG_PCMCIA_PCNET=m
CONFIG_PCMCIA_NMCLAN=m
CONFIG_PCMCIA_SMC91C92=m
CONFIG_PCMCIA_XIRC2PS=m
CONFIG_PCMCIA_AXNET=m
# CONFIG_WAN is not set
CONFIG_ATM_DRIVERS=y
# CONFIG_ATM_DUMMY is not set
CONFIG_ATM_TCP=m
# CONFIG_ATM_LANAI is not set
CONFIG_ATM_ENI=m
# CONFIG_ATM_ENI_DEBUG is not set
# CONFIG_ATM_ENI_TUNE_BURST is not set
CONFIG_ATM_FIRESTREAM=m
# CONFIG_ATM_ZATM is not set
CONFIG_ATM_NICSTAR=m
# CONFIG_ATM_NICSTAR_USE_SUNI is not set
# CONFIG_ATM_NICSTAR_USE_IDT77105 is not set
# CONFIG_ATM_IDT77252 is not set
# CONFIG_ATM_AMBASSADOR is not set
# CONFIG_ATM_HORIZON is not set
# CONFIG_ATM_IA is not set
# CONFIG_ATM_FORE200E is not set
CONFIG_ATM_HE=m
# CONFIG_ATM_HE_USE_SUNI is not set
CONFIG_ATM_SOLOS=m
CONFIG_IEEE802154_DRIVERS=m
CONFIG_IEEE802154_FAKEHARD=m
#
# CAIF transport drivers
#
CONFIG_XEN_NETDEV_FRONTEND=m
CONFIG_FDDI=y
# CONFIG_DEFXX is not set
CONFIG_SKFP=m
# CONFIG_HIPPI is not set
CONFIG_PLIP=m
CONFIG_PPP=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
# CONFIG_PPP_BSDCOMP is not set
CONFIG_PPP_MPPE=m
CONFIG_PPPOE=m
CONFIG_PPTP=m
CONFIG_PPPOATM=m
CONFIG_PPPOL2TP=m
CONFIG_SLIP=m
CONFIG_SLIP_COMPRESSED=y
CONFIG_SLHC=m
CONFIG_SLIP_SMART=y
# CONFIG_SLIP_MODE_SLIP6 is not set
CONFIG_NET_FC=y
CONFIG_NETCONSOLE=m
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_NETPOLL=y
CONFIG_NETPOLL_TRAP=y
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_VIRTIO_NET=m
CONFIG_VMXNET3=m
CONFIG_ISDN=y
CONFIG_ISDN_I4L=m
CONFIG_ISDN_PPP=y
CONFIG_ISDN_PPP_VJ=y
CONFIG_ISDN_MPP=y
CONFIG_IPPP_FILTER=y
# CONFIG_ISDN_PPP_BSDCOMP is not set
CONFIG_ISDN_AUDIO=y
CONFIG_ISDN_TTY_FAX=y
#
# ISDN feature submodules
#
CONFIG_ISDN_DIVERSION=m
#
# ISDN4Linux hardware drivers
#
#
# Passive cards
#
CONFIG_ISDN_DRV_HISAX=m
#
# D-channel protocol features
#
CONFIG_HISAX_EURO=y
CONFIG_DE_AOC=y
CONFIG_HISAX_NO_SENDCOMPLETE=y
CONFIG_HISAX_NO_LLC=y
CONFIG_HISAX_NO_KEYPAD=y
CONFIG_HISAX_1TR6=y
CONFIG_HISAX_NI1=y
CONFIG_HISAX_MAX_CARDS=8
#
# HiSax supported cards
#
CONFIG_HISAX_16_3=y
CONFIG_HISAX_TELESPCI=y
CONFIG_HISAX_S0BOX=y
CONFIG_HISAX_FRITZPCI=y
CONFIG_HISAX_AVM_A1_PCMCIA=y
CONFIG_HISAX_ELSA=y
CONFIG_HISAX_DIEHLDIVA=y
CONFIG_HISAX_SEDLBAUER=y
CONFIG_HISAX_NETJET=y
CONFIG_HISAX_NETJET_U=y
CONFIG_HISAX_NICCY=y
CONFIG_HISAX_BKM_A4T=y
CONFIG_HISAX_SCT_QUADRO=y
CONFIG_HISAX_GAZEL=y
CONFIG_HISAX_HFC_PCI=y
CONFIG_HISAX_W6692=y
CONFIG_HISAX_HFC_SX=y
CONFIG_HISAX_ENTERNOW_PCI=y
# CONFIG_HISAX_DEBUG is not set
#
# HiSax PCMCIA card service modules
#
CONFIG_HISAX_SEDLBAUER_CS=m
CONFIG_HISAX_ELSA_CS=m
CONFIG_HISAX_AVM_A1_CS=m
CONFIG_HISAX_TELES_CS=m
#
# HiSax sub driver modules
#
CONFIG_HISAX_ST5481=m
# CONFIG_HISAX_HFCUSB is not set
CONFIG_HISAX_HFC4S8S=m
CONFIG_HISAX_FRITZ_PCIPNP=m
#
# Active cards
#
CONFIG_ISDN_CAPI=m
CONFIG_ISDN_DRV_AVMB1_VERBOSE_REASON=y
# CONFIG_CAPI_TRACE is not set
CONFIG_ISDN_CAPI_MIDDLEWARE=y
CONFIG_ISDN_CAPI_CAPI20=m
CONFIG_ISDN_CAPI_CAPIFS_BOOL=y
CONFIG_ISDN_CAPI_CAPIFS=m
CONFIG_ISDN_CAPI_CAPIDRV=m
#
# CAPI hardware drivers
#
CONFIG_CAPI_AVM=y
CONFIG_ISDN_DRV_AVMB1_B1PCI=m
CONFIG_ISDN_DRV_AVMB1_B1PCIV4=y
CONFIG_ISDN_DRV_AVMB1_B1PCMCIA=m
CONFIG_ISDN_DRV_AVMB1_AVM_CS=m
CONFIG_ISDN_DRV_AVMB1_T1PCI=m
CONFIG_ISDN_DRV_AVMB1_C4=m
CONFIG_CAPI_EICON=y
CONFIG_ISDN_DIVAS=m
CONFIG_ISDN_DIVAS_BRIPCI=y
CONFIG_ISDN_DIVAS_PRIPCI=y
CONFIG_ISDN_DIVAS_DIVACAPI=m
CONFIG_ISDN_DIVAS_USERIDI=m
CONFIG_ISDN_DIVAS_MAINT=m
CONFIG_ISDN_DRV_GIGASET=m
CONFIG_GIGASET_CAPI=y
# CONFIG_GIGASET_I4L is not set
# CONFIG_GIGASET_DUMMYLL is not set
CONFIG_GIGASET_BASE=m
CONFIG_GIGASET_M105=m
CONFIG_GIGASET_M101=m
# CONFIG_GIGASET_DEBUG is not set
CONFIG_HYSDN=m
CONFIG_HYSDN_CAPI=y
CONFIG_MISDN=m
CONFIG_MISDN_DSP=m
CONFIG_MISDN_L1OIP=m
#
# mISDN hardware drivers
#
CONFIG_MISDN_HFCPCI=m
CONFIG_MISDN_HFCMULTI=m
CONFIG_MISDN_HFCUSB=m
CONFIG_MISDN_AVMFRITZ=m
CONFIG_MISDN_SPEEDFAX=m
CONFIG_MISDN_INFINEON=m
CONFIG_MISDN_W6692=m
CONFIG_MISDN_NETJET=m
CONFIG_MISDN_IPAC=m
CONFIG_MISDN_ISAR=m
CONFIG_ISDN_HDLC=m
# CONFIG_PHONE is not set
#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_FF_MEMLESS=y
CONFIG_INPUT_POLLDEV=m
CONFIG_INPUT_SPARSEKMAP=m
#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=m
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set
CONFIG_XEN_KBDDEV_FRONTEND=y
#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ADP5588=m
CONFIG_KEYBOARD_ATKBD=y
CONFIG_KEYBOARD_QT2160=m
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_GPIO is not set
# CONFIG_KEYBOARD_GPIO_POLLED is not set
# CONFIG_KEYBOARD_TCA6416 is not set
# CONFIG_KEYBOARD_MATRIX is not set
# CONFIG_KEYBOARD_LM8323 is not set
CONFIG_KEYBOARD_MAX7359=m
# CONFIG_KEYBOARD_MCS is not set
# CONFIG_KEYBOARD_NEWTON is not set
CONFIG_KEYBOARD_OPENCORES=m
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
CONFIG_MOUSE_PS2_ELANTECH=y
CONFIG_MOUSE_PS2_SENTELIC=y
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
CONFIG_MOUSE_SERIAL=m
CONFIG_MOUSE_APPLETOUCH=m
CONFIG_MOUSE_BCM5974=m
CONFIG_MOUSE_VSXXXAA=m
# CONFIG_MOUSE_GPIO is not set
CONFIG_MOUSE_SYNAPTICS_I2C=m
CONFIG_INPUT_JOYSTICK=y
CONFIG_JOYSTICK_ANALOG=m
CONFIG_JOYSTICK_A3D=m
CONFIG_JOYSTICK_ADI=m
CONFIG_JOYSTICK_COBRA=m
CONFIG_JOYSTICK_GF2K=m
CONFIG_JOYSTICK_GRIP=m
CONFIG_JOYSTICK_GRIP_MP=m
CONFIG_JOYSTICK_GUILLEMOT=m
CONFIG_JOYSTICK_INTERACT=m
CONFIG_JOYSTICK_SIDEWINDER=m
CONFIG_JOYSTICK_TMDC=m
CONFIG_JOYSTICK_IFORCE=m
CONFIG_JOYSTICK_IFORCE_USB=y
CONFIG_JOYSTICK_IFORCE_232=y
CONFIG_JOYSTICK_WARRIOR=m
CONFIG_JOYSTICK_MAGELLAN=m
CONFIG_JOYSTICK_SPACEORB=m
CONFIG_JOYSTICK_SPACEBALL=m
CONFIG_JOYSTICK_STINGER=m
CONFIG_JOYSTICK_TWIDJOY=m
CONFIG_JOYSTICK_ZHENHUA=m
CONFIG_JOYSTICK_DB9=m
CONFIG_JOYSTICK_GAMECON=m
CONFIG_JOYSTICK_TURBOGRAFX=m
# CONFIG_JOYSTICK_AS5011 is not set
CONFIG_JOYSTICK_JOYDUMP=m
CONFIG_JOYSTICK_XPAD=m
CONFIG_JOYSTICK_XPAD_FF=y
CONFIG_JOYSTICK_XPAD_LEDS=y
CONFIG_JOYSTICK_WALKERA0701=m
CONFIG_INPUT_TABLET=y
CONFIG_TABLET_USB_ACECAD=m
CONFIG_TABLET_USB_AIPTEK=m
CONFIG_TABLET_USB_GTCO=m
CONFIG_TABLET_USB_HANWANG=m
CONFIG_TABLET_USB_KBTAB=m
CONFIG_TABLET_USB_WACOM=m
CONFIG_INPUT_TOUCHSCREEN=y
# CONFIG_TOUCHSCREEN_AD7879 is not set
# CONFIG_TOUCHSCREEN_BU21013 is not set
# CONFIG_TOUCHSCREEN_CY8CTMG110 is not set
CONFIG_TOUCHSCREEN_DYNAPRO=m
# CONFIG_TOUCHSCREEN_HAMPSHIRE is not set
CONFIG_TOUCHSCREEN_EETI=m
CONFIG_TOUCHSCREEN_FUJITSU=m
CONFIG_TOUCHSCREEN_GUNZE=m
CONFIG_TOUCHSCREEN_ELO=m
CONFIG_TOUCHSCREEN_WACOM_W8001=m
CONFIG_TOUCHSCREEN_MCS5000=m
CONFIG_TOUCHSCREEN_MTOUCH=m
CONFIG_TOUCHSCREEN_INEXIO=m
CONFIG_TOUCHSCREEN_MK712=m
CONFIG_TOUCHSCREEN_PENMOUNT=m
# CONFIG_TOUCHSCREEN_QT602240 is not set
CONFIG_TOUCHSCREEN_TOUCHRIGHT=m
CONFIG_TOUCHSCREEN_TOUCHWIN=m
# CONFIG_TOUCHSCREEN_WM97XX is not set
CONFIG_TOUCHSCREEN_USB_COMPOSITE=m
CONFIG_TOUCHSCREEN_USB_EGALAX=y
CONFIG_TOUCHSCREEN_USB_PANJIT=y
CONFIG_TOUCHSCREEN_USB_3M=y
CONFIG_TOUCHSCREEN_USB_ITM=y
CONFIG_TOUCHSCREEN_USB_ETURBO=y
CONFIG_TOUCHSCREEN_USB_GUNZE=y
CONFIG_TOUCHSCREEN_USB_DMC_TSC10=y
CONFIG_TOUCHSCREEN_USB_IRTOUCH=y
CONFIG_TOUCHSCREEN_USB_IDEALTEK=y
CONFIG_TOUCHSCREEN_USB_GENERAL_TOUCH=y
CONFIG_TOUCHSCREEN_USB_GOTOP=y
CONFIG_TOUCHSCREEN_USB_JASTEC=y
CONFIG_TOUCHSCREEN_USB_E2I=y
CONFIG_TOUCHSCREEN_USB_ZYTRONIC=y
CONFIG_TOUCHSCREEN_USB_ETT_TC45USB=y
CONFIG_TOUCHSCREEN_USB_NEXIO=y
CONFIG_TOUCHSCREEN_TOUCHIT213=m
CONFIG_TOUCHSCREEN_TSC2007=m
CONFIG_TOUCHSCREEN_ST1232=m
# CONFIG_TOUCHSCREEN_TPS6507X is not set
CONFIG_INPUT_MISC=y
# CONFIG_INPUT_AD714X is not set
CONFIG_INPUT_PCSPKR=m
CONFIG_INPUT_APANEL=m
CONFIG_INPUT_ATLAS_BTNS=m
CONFIG_INPUT_ATI_REMOTE=m
CONFIG_INPUT_ATI_REMOTE2=m
CONFIG_INPUT_KEYSPAN_REMOTE=m
CONFIG_INPUT_APPLEIR=m
CONFIG_INPUT_POWERMATE=m
CONFIG_INPUT_YEALINK=m
CONFIG_INPUT_CM109=m
CONFIG_INPUT_UINPUT=m
# CONFIG_INPUT_PCF8574 is not set
CONFIG_INPUT_GPIO_ROTARY_ENCODER=m
# CONFIG_INPUT_ADXL34X is not set
CONFIG_INPUT_CMA3000=m
CONFIG_INPUT_CMA3000_I2C=m
#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m
CONFIG_SERIO_ALTERA_PS2=m
# CONFIG_SERIO_PS2MULT is not set
CONFIG_GAMEPORT=m
CONFIG_GAMEPORT_NS558=m
CONFIG_GAMEPORT_L4=m
CONFIG_GAMEPORT_EMU10K1=m
CONFIG_GAMEPORT_FM801=m
#
# Character devices
#
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
# CONFIG_DEVKMEM is not set
CONFIG_SERIAL_NONSTANDARD=y
# CONFIG_COMPUTONE is not set
CONFIG_ROCKETPORT=m
CONFIG_CYCLADES=m
# CONFIG_CYZ_INTR is not set
# CONFIG_DIGIEPCA is not set
# CONFIG_MOXA_INTELLIO is not set
# CONFIG_MOXA_SMARTIO is not set
# CONFIG_ISI is not set
CONFIG_SYNCLINK=m
CONFIG_SYNCLINKMP=m
CONFIG_SYNCLINK_GT=m
CONFIG_N_HDLC=m
CONFIG_N_GSM=m
# CONFIG_RISCOM8 is not set
# CONFIG_SPECIALIX is not set
# CONFIG_STALDRV is not set
CONFIG_NOZOMI=m
#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_CS=m
CONFIG_SERIAL_8250_NR_UARTS=32
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
# CONFIG_SERIAL_8250_DETECT_IRQ is not set
CONFIG_SERIAL_8250_RSA=y
#
# Non-8250 serial port support
#
# CONFIG_SERIAL_MFD_HSU is not set
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_CONSOLE_POLL=y
CONFIG_SERIAL_JSM=m
# CONFIG_SERIAL_TIMBERDALE is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
# CONFIG_SERIAL_PCH_UART is not set
CONFIG_UNIX98_PTYS=y
CONFIG_DEVPTS_MULTIPLE_INSTANCES=y
# CONFIG_LEGACY_PTYS is not set
CONFIG_CRASH=m
CONFIG_PRINTER=m
CONFIG_LP_CONSOLE=y
CONFIG_PPDEV=m
CONFIG_HVC_DRIVER=y
CONFIG_HVC_IRQ=y
CONFIG_HVC_XEN=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_IPMI_HANDLER=m
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_TIMERIOMEM=m
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_AMD=m
CONFIG_HW_RANDOM_VIA=m
CONFIG_HW_RANDOM_VIRTIO=m
CONFIG_NVRAM=y
CONFIG_R3964=m
# CONFIG_APPLICOM is not set
#
# PCMCIA character devices
#
# CONFIG_SYNCLINK_CS is not set
CONFIG_CARDMAN_4000=m
CONFIG_CARDMAN_4040=m
CONFIG_IPWIRELESS=m
CONFIG_MWAVE=m
CONFIG_RAW_DRIVER=y
CONFIG_MAX_RAW_DEVS=8192
CONFIG_HPET=y
# CONFIG_HPET_MMAP is not set
CONFIG_HANGCHECK_TIMER=m
CONFIG_TCG_TPM=y
CONFIG_TCG_TIS=y
CONFIG_TCG_NSC=m
CONFIG_TCG_ATMEL=m
CONFIG_TCG_INFINEON=m
CONFIG_TELCLOCK=m
CONFIG_DEVPORT=y
# CONFIG_RAMOOPS is not set
CONFIG_I2C=m
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=m
# CONFIG_I2C_MUX is not set
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_SMBUS=m
CONFIG_I2C_ALGOBIT=m
CONFIG_I2C_ALGOPCA=m
#
# I2C Hardware Bus support
#
#
# PC SMBus host controller drivers
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD756_S4882=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_I801=m
CONFIG_I2C_ISCH=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_NFORCE2_S4985=m
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
CONFIG_I2C_SIS96X=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
#
# ACPI drivers
#
CONFIG_I2C_SCMI=m
#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_GPIO is not set
# CONFIG_I2C_INTEL_MID is not set
# CONFIG_I2C_OCORES is not set
CONFIG_I2C_PCA_PLATFORM=m
CONFIG_I2C_SIMTEC=m
# CONFIG_I2C_XILINX is not set
# CONFIG_I2C_EG20T is not set
#
# External I2C/SMBus adapter drivers
#
CONFIG_I2C_PARPORT=m
CONFIG_I2C_PARPORT_LIGHT=m
# CONFIG_I2C_TAOS_EVM is not set
CONFIG_I2C_TINY_USB=m
#
# Other I2C/SMBus bus drivers
#
CONFIG_I2C_STUB=m
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_SPI is not set
#
# PPS support
#
CONFIG_PPS=m
# CONFIG_PPS_DEBUG is not set
#
# PPS clients support
#
# CONFIG_PPS_CLIENT_KTIMER is not set
CONFIG_PPS_CLIENT_LDISC=m
CONFIG_PPS_CLIENT_PARPORT=m
#
# PPS generators support
#
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
CONFIG_GPIOLIB=y
# CONFIG_DEBUG_GPIO is not set
CONFIG_GPIO_SYSFS=y
#
# Memory mapped GPIO expanders:
#
# CONFIG_GPIO_BASIC_MMIO is not set
# CONFIG_GPIO_IT8761E is not set
CONFIG_GPIO_SCH=m
# CONFIG_GPIO_VX855 is not set
#
# I2C GPIO expanders:
#
# CONFIG_GPIO_MAX7300 is not set
# CONFIG_GPIO_MAX732X is not set
# CONFIG_GPIO_PCA953X is not set
# CONFIG_GPIO_PCF857X is not set
# CONFIG_GPIO_ADP5588 is not set
#
# PCI GPIO expanders:
#
# CONFIG_GPIO_CS5535 is not set
CONFIG_GPIO_LANGWELL=y
# CONFIG_GPIO_PCH is not set
# CONFIG_GPIO_ML_IOH is not set
# CONFIG_GPIO_RDC321X is not set
#
# SPI GPIO expanders:
#
#
# AC97 GPIO expanders:
#
#
# MODULbus GPIO expanders:
#
CONFIG_W1=m
CONFIG_W1_CON=y
#
# 1-wire Bus Masters
#
# CONFIG_W1_MASTER_MATROX is not set
CONFIG_W1_MASTER_DS2490=m
CONFIG_W1_MASTER_DS2482=m
# CONFIG_W1_MASTER_GPIO is not set
#
# 1-wire Slaves
#
CONFIG_W1_SLAVE_THERM=m
CONFIG_W1_SLAVE_SMEM=m
CONFIG_W1_SLAVE_DS2423=m
CONFIG_W1_SLAVE_DS2431=m
CONFIG_W1_SLAVE_DS2433=m
CONFIG_W1_SLAVE_DS2433_CRC=y
CONFIG_W1_SLAVE_DS2760=m
CONFIG_W1_SLAVE_BQ27000=m
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_TEST_POWER is not set
# CONFIG_BATTERY_DS2760 is not set
# CONFIG_BATTERY_DS2782 is not set
# CONFIG_BATTERY_BQ20Z75 is not set
# CONFIG_BATTERY_BQ27x00 is not set
# CONFIG_BATTERY_MAX17040 is not set
# CONFIG_BATTERY_MAX17042 is not set
# CONFIG_CHARGER_ISP1704 is not set
# CONFIG_CHARGER_GPIO is not set
CONFIG_HWMON=y
CONFIG_HWMON_VID=m
# CONFIG_HWMON_DEBUG_CHIP is not set
#
# Native drivers
#
CONFIG_SENSORS_ABITUGURU=m
CONFIG_SENSORS_ABITUGURU3=m
CONFIG_SENSORS_AD7414=m
CONFIG_SENSORS_AD7418=m
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1026=m
CONFIG_SENSORS_ADM1029=m
CONFIG_SENSORS_ADM1031=m
CONFIG_SENSORS_ADM9240=m
CONFIG_SENSORS_ADT7411=m
CONFIG_SENSORS_ADT7462=m
CONFIG_SENSORS_ADT7470=m
CONFIG_SENSORS_ADT7475=m
CONFIG_SENSORS_ASC7621=m
CONFIG_SENSORS_K8TEMP=m
CONFIG_SENSORS_K10TEMP=m
CONFIG_SENSORS_ASB100=m
CONFIG_SENSORS_ATXP1=m
CONFIG_SENSORS_DS620=m
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_I5K_AMB=m
CONFIG_SENSORS_F71805F=m
CONFIG_SENSORS_F71882FG=m
CONFIG_SENSORS_F75375S=m
CONFIG_SENSORS_FSCHMD=m
CONFIG_SENSORS_G760A=m
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_GL520SM=m
# CONFIG_SENSORS_GPIO_FAN is not set
CONFIG_SENSORS_CORETEMP=m
CONFIG_SENSORS_PKGTEMP=m
CONFIG_SENSORS_IBMAEM=m
CONFIG_SENSORS_IBMPEX=m
CONFIG_SENSORS_IT87=m
# CONFIG_SENSORS_JC42 is not set
CONFIG_SENSORS_LM63=m
CONFIG_SENSORS_LM73=m
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM87=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_LM92=m
CONFIG_SENSORS_LM93=m
CONFIG_SENSORS_LTC4215=m
CONFIG_SENSORS_LTC4245=m
CONFIG_SENSORS_LTC4261=m
CONFIG_SENSORS_LM95241=m
CONFIG_SENSORS_MAX1619=m
CONFIG_SENSORS_MAX6650=m
CONFIG_SENSORS_PC87360=m
CONFIG_SENSORS_PC87427=m
CONFIG_SENSORS_PCF8591=m
CONFIG_SENSORS_SHT15=m
CONFIG_SENSORS_SHT21=m
CONFIG_SENSORS_SIS5595=m
# CONFIG_SENSORS_SMM665 is not set
CONFIG_SENSORS_DME1737=m
CONFIG_SENSORS_EMC1403=m
# CONFIG_SENSORS_EMC2103 is not set
CONFIG_SENSORS_SMSC47M1=m
CONFIG_SENSORS_SMSC47M192=m
CONFIG_SENSORS_SMSC47B397=m
CONFIG_SENSORS_ADS7828=m
CONFIG_SENSORS_AMC6821=m
CONFIG_SENSORS_THMC50=m
CONFIG_SENSORS_TMP102=m
CONFIG_SENSORS_TMP401=m
CONFIG_SENSORS_TMP421=m
CONFIG_SENSORS_VIA_CPUTEMP=m
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_VT1211=m
CONFIG_SENSORS_VT8231=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83791D=m
CONFIG_SENSORS_W83792D=m
CONFIG_SENSORS_W83793=m
CONFIG_SENSORS_W83795=m
# CONFIG_SENSORS_W83795_FANCTRL is not set
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83L786NG=m
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m
CONFIG_SENSORS_LIS3_I2C=m
CONFIG_SENSORS_APPLESMC=m
#
# ACPI drivers
#
CONFIG_SENSORS_ATK0110=m
CONFIG_SENSORS_LIS3LV02D=m
CONFIG_THERMAL=y
CONFIG_THERMAL_HWMON=y
CONFIG_WATCHDOG=y
# CONFIG_WATCHDOG_NOWAYOUT is not set
#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=m
# CONFIG_ACQUIRE_WDT is not set
# CONFIG_ADVANTECH_WDT is not set
CONFIG_ALIM1535_WDT=m
CONFIG_ALIM7101_WDT=m
CONFIG_F71808E_WDT=m
CONFIG_SP5100_TCO=m
CONFIG_GEODE_WDT=m
# CONFIG_SC520_WDT is not set
CONFIG_SBC_FITPC2_WATCHDOG=m
# CONFIG_EUROTECH_WDT is not set
CONFIG_IB700_WDT=m
CONFIG_IBMASR=m
# CONFIG_WAFER_WDT is not set
CONFIG_I6300ESB_WDT=m
CONFIG_ITCO_WDT=m
CONFIG_ITCO_VENDOR_SUPPORT=y
CONFIG_IT8712F_WDT=m
CONFIG_IT87_WDT=m
CONFIG_HP_WATCHDOG=m
CONFIG_HPWDT_NMI_DECODING=y
# CONFIG_SC1200_WDT is not set
# CONFIG_PC87413_WDT is not set
CONFIG_NV_TCO=m
# CONFIG_60XX_WDT is not set
# CONFIG_SBC8360_WDT is not set
# CONFIG_CPU5_WDT is not set
CONFIG_SMSC_SCH311X_WDT=m
# CONFIG_SMSC37B787_WDT is not set
CONFIG_W83627HF_WDT=m
CONFIG_W83697HF_WDT=m
CONFIG_W83697UG_WDT=m
CONFIG_W83877F_WDT=m
CONFIG_W83977F_WDT=m
CONFIG_MACHZ_WDT=m
# CONFIG_SBC_EPX_C3_WATCHDOG is not set
#
# PCI-based Watchdog Cards
#
CONFIG_PCIPCWATCHDOG=m
CONFIG_WDTPCI=m
#
# USB-based Watchdog Cards
#
CONFIG_USBPCWATCHDOG=m
CONFIG_SSB_POSSIBLE=y
#
# Sonics Silicon Backplane
#
CONFIG_SSB=m
CONFIG_SSB_SPROM=y
CONFIG_SSB_BLOCKIO=y
CONFIG_SSB_PCIHOST_POSSIBLE=y
CONFIG_SSB_PCIHOST=y
CONFIG_SSB_B43_PCI_BRIDGE=y
CONFIG_SSB_PCMCIAHOST_POSSIBLE=y
CONFIG_SSB_PCMCIAHOST=y
CONFIG_SSB_SDIOHOST_POSSIBLE=y
CONFIG_SSB_SDIOHOST=y
# CONFIG_SSB_DEBUG is not set
CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
CONFIG_SSB_DRIVER_PCICORE=y
CONFIG_MFD_SUPPORT=y
CONFIG_MFD_CORE=m
CONFIG_MFD_SM501=m
CONFIG_MFD_SM501_GPIO=y
# CONFIG_HTC_PASIC3 is not set
# CONFIG_UCB1400_CORE is not set
# CONFIG_TPS65010 is not set
# CONFIG_TPS6507X is not set
# CONFIG_MFD_TMIO is not set
CONFIG_MFD_WM8400=m
# CONFIG_MFD_PCF50633 is not set
# CONFIG_ABX500_CORE is not set
CONFIG_MFD_CS5535=m
# CONFIG_MFD_TIMBERDALE is not set
CONFIG_LPC_SCH=m
# CONFIG_MFD_RDC321X is not set
# CONFIG_MFD_JANZ_CMODIO is not set
CONFIG_MFD_VX855=m
CONFIG_MFD_WL1273_CORE=m
# CONFIG_REGULATOR is not set
CONFIG_MEDIA_SUPPORT=m
#
# Multimedia core support
#
CONFIG_VIDEO_DEV=m
CONFIG_VIDEO_V4L2_COMMON=m
CONFIG_DVB_CORE=m
CONFIG_VIDEO_MEDIA=m
#
# Multimedia drivers
#
CONFIG_VIDEO_SAA7146=m
CONFIG_VIDEO_SAA7146_VV=m
CONFIG_RC_CORE=m
CONFIG_LIRC=m
CONFIG_RC_MAP=m
CONFIG_IR_NEC_DECODER=m
CONFIG_IR_RC5_DECODER=m
CONFIG_IR_RC6_DECODER=m
CONFIG_IR_JVC_DECODER=m
CONFIG_IR_SONY_DECODER=m
CONFIG_IR_RC5_SZ_DECODER=m
CONFIG_IR_LIRC_CODEC=m
CONFIG_IR_ENE=m
CONFIG_IR_IMON=m
CONFIG_IR_MCEUSB=m
CONFIG_IR_ITE_CIR=m
CONFIG_IR_NUVOTON=m
CONFIG_IR_STREAMZAP=m
CONFIG_IR_WINBOND_CIR=m
CONFIG_RC_LOOPBACK=m
CONFIG_MEDIA_ATTACH=y
CONFIG_MEDIA_TUNER=m
CONFIG_MEDIA_TUNER_CUSTOMISE=y
#
# Customize TV tuners
#
CONFIG_MEDIA_TUNER_SIMPLE=m
CONFIG_MEDIA_TUNER_TDA8290=m
CONFIG_MEDIA_TUNER_TDA827X=m
CONFIG_MEDIA_TUNER_TDA18271=m
CONFIG_MEDIA_TUNER_TDA9887=m
CONFIG_MEDIA_TUNER_TEA5761=m
CONFIG_MEDIA_TUNER_TEA5767=m
CONFIG_MEDIA_TUNER_MT20XX=m
CONFIG_MEDIA_TUNER_MT2060=m
CONFIG_MEDIA_TUNER_MT2266=m
CONFIG_MEDIA_TUNER_MT2131=m
CONFIG_MEDIA_TUNER_QT1010=m
CONFIG_MEDIA_TUNER_XC2028=m
CONFIG_MEDIA_TUNER_XC5000=m
CONFIG_MEDIA_TUNER_MXL5005S=m
CONFIG_MEDIA_TUNER_MXL5007T=m
CONFIG_MEDIA_TUNER_MC44S803=m
CONFIG_MEDIA_TUNER_MAX2165=m
CONFIG_MEDIA_TUNER_TDA18218=m
CONFIG_VIDEO_V4L2=m
CONFIG_VIDEOBUF_GEN=m
CONFIG_VIDEOBUF_DMA_SG=m
CONFIG_VIDEOBUF_VMALLOC=m
CONFIG_VIDEOBUF_DVB=m
CONFIG_VIDEO_BTCX=m
CONFIG_VIDEO_TVEEPROM=m
CONFIG_VIDEO_TUNER=m
CONFIG_VIDEO_CAPTURE_DRIVERS=y
# CONFIG_VIDEO_ADV_DEBUG is not set
# CONFIG_VIDEO_FIXED_MINOR_RANGES is not set
CONFIG_VIDEO_HELPER_CHIPS_AUTO=y
CONFIG_VIDEO_IR_I2C=m
#
# Audio decoders
#
CONFIG_VIDEO_TVAUDIO=m
CONFIG_VIDEO_TDA7432=m
CONFIG_VIDEO_TDA9840=m
CONFIG_VIDEO_TEA6415C=m
CONFIG_VIDEO_TEA6420=m
CONFIG_VIDEO_MSP3400=m
CONFIG_VIDEO_CS5345=m
CONFIG_VIDEO_CS53L32A=m
CONFIG_VIDEO_M52790=m
CONFIG_VIDEO_WM8775=m
CONFIG_VIDEO_WM8739=m
CONFIG_VIDEO_VP27SMPX=m
#
# RDS decoders
#
CONFIG_VIDEO_SAA6588=m
#
# Video decoders
#
CONFIG_VIDEO_BT819=m
CONFIG_VIDEO_BT856=m
CONFIG_VIDEO_BT866=m
CONFIG_VIDEO_KS0127=m
CONFIG_VIDEO_OV7670=m
CONFIG_VIDEO_MT9V011=m
CONFIG_VIDEO_SAA7110=m
CONFIG_VIDEO_SAA711X=m
CONFIG_VIDEO_SAA717X=m
CONFIG_VIDEO_TVP5150=m
CONFIG_VIDEO_VPX3220=m
#
# Video and audio decoders
#
CONFIG_VIDEO_CX25840=m
#
# MPEG video encoders
#
CONFIG_VIDEO_CX2341X=m
#
# Video encoders
#
CONFIG_VIDEO_SAA7127=m
CONFIG_VIDEO_SAA7185=m
CONFIG_VIDEO_ADV7170=m
CONFIG_VIDEO_ADV7175=m
#
# Video improvement chips
#
CONFIG_VIDEO_UPD64031A=m
CONFIG_VIDEO_UPD64083=m
# CONFIG_VIDEO_VIVI is not set
CONFIG_VIDEO_BT848=m
CONFIG_VIDEO_BT848_DVB=y
CONFIG_VIDEO_BWQCAM=m
CONFIG_VIDEO_CQCAM=m
CONFIG_VIDEO_W9966=m
CONFIG_VIDEO_CPIA2=m
CONFIG_VIDEO_ZORAN=m
CONFIG_VIDEO_ZORAN_DC30=m
CONFIG_VIDEO_ZORAN_ZR36060=m
CONFIG_VIDEO_ZORAN_BUZ=m
CONFIG_VIDEO_ZORAN_DC10=m
CONFIG_VIDEO_ZORAN_LML33=m
CONFIG_VIDEO_ZORAN_LML33R10=m
CONFIG_VIDEO_ZORAN_AVS6EYES=m
CONFIG_VIDEO_MEYE=m
CONFIG_VIDEO_SAA7134=m
CONFIG_VIDEO_SAA7134_ALSA=m
CONFIG_VIDEO_SAA7134_RC=y
CONFIG_VIDEO_SAA7134_DVB=m
CONFIG_VIDEO_MXB=m
CONFIG_VIDEO_HEXIUM_ORION=m
CONFIG_VIDEO_HEXIUM_GEMINI=m
# CONFIG_VIDEO_TIMBERDALE is not set
CONFIG_VIDEO_CX88=m
CONFIG_VIDEO_CX88_ALSA=m
CONFIG_VIDEO_CX88_BLACKBIRD=m
CONFIG_VIDEO_CX88_DVB=m
CONFIG_VIDEO_CX88_MPEG=m
CONFIG_VIDEO_CX88_VP3054=m
CONFIG_VIDEO_CX23885=m
CONFIG_VIDEO_AU0828=m
CONFIG_VIDEO_IVTV=m
CONFIG_VIDEO_FB_IVTV=m
CONFIG_VIDEO_CX18=m
CONFIG_VIDEO_CX18_ALSA=m
CONFIG_VIDEO_SAA7164=m
# CONFIG_VIDEO_CAFE_CCIC is not set
CONFIG_VIDEO_SR030PC30=m
CONFIG_VIDEO_VIA_CAMERA=m
CONFIG_SOC_CAMERA=m
CONFIG_SOC_CAMERA_IMX074=m
CONFIG_SOC_CAMERA_MT9M001=m
CONFIG_SOC_CAMERA_MT9M111=m
CONFIG_SOC_CAMERA_MT9T031=m
CONFIG_SOC_CAMERA_MT9T112=m
CONFIG_SOC_CAMERA_MT9V022=m
CONFIG_SOC_CAMERA_RJ54N1=m
CONFIG_SOC_CAMERA_TW9910=m
CONFIG_SOC_CAMERA_PLATFORM=m
CONFIG_SOC_CAMERA_OV2640=m
CONFIG_SOC_CAMERA_OV6650=m
CONFIG_SOC_CAMERA_OV772X=m
CONFIG_SOC_CAMERA_OV9640=m
CONFIG_V4L_USB_DRIVERS=y
CONFIG_USB_VIDEO_CLASS=m
CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y
CONFIG_USB_GSPCA=m
CONFIG_USB_M5602=m
CONFIG_USB_STV06XX=m
CONFIG_USB_GL860=m
CONFIG_USB_GSPCA_BENQ=m
CONFIG_USB_GSPCA_CONEX=m
CONFIG_USB_GSPCA_CPIA1=m
CONFIG_USB_GSPCA_ETOMS=m
CONFIG_USB_GSPCA_FINEPIX=m
CONFIG_USB_GSPCA_JEILINJ=m
CONFIG_USB_GSPCA_KONICA=m
CONFIG_USB_GSPCA_MARS=m
CONFIG_USB_GSPCA_MR97310A=m
CONFIG_USB_GSPCA_OV519=m
CONFIG_USB_GSPCA_OV534=m
CONFIG_USB_GSPCA_OV534_9=m
CONFIG_USB_GSPCA_PAC207=m
CONFIG_USB_GSPCA_PAC7302=m
CONFIG_USB_GSPCA_PAC7311=m
CONFIG_USB_GSPCA_SN9C2028=m
CONFIG_USB_GSPCA_SN9C20X=m
CONFIG_USB_GSPCA_SONIXB=m
CONFIG_USB_GSPCA_SONIXJ=m
CONFIG_USB_GSPCA_SPCA500=m
CONFIG_USB_GSPCA_SPCA501=m
CONFIG_USB_GSPCA_SPCA505=m
CONFIG_USB_GSPCA_SPCA506=m
CONFIG_USB_GSPCA_SPCA508=m
CONFIG_USB_GSPCA_SPCA561=m
CONFIG_USB_GSPCA_SPCA1528=m
CONFIG_USB_GSPCA_SQ905=m
CONFIG_USB_GSPCA_SQ905C=m
CONFIG_USB_GSPCA_SQ930X=m
CONFIG_USB_GSPCA_STK014=m
CONFIG_USB_GSPCA_STV0680=m
CONFIG_USB_GSPCA_SUNPLUS=m
CONFIG_USB_GSPCA_T613=m
CONFIG_USB_GSPCA_TV8532=m
CONFIG_USB_GSPCA_VC032X=m
CONFIG_USB_GSPCA_XIRLINK_CIT=m
CONFIG_USB_GSPCA_ZC3XX=m
CONFIG_VIDEO_PVRUSB2=m
CONFIG_VIDEO_PVRUSB2_SYSFS=y
CONFIG_VIDEO_PVRUSB2_DVB=y
# CONFIG_VIDEO_PVRUSB2_DEBUGIFC is not set
CONFIG_VIDEO_HDPVR=m
CONFIG_VIDEO_EM28XX=m
CONFIG_VIDEO_EM28XX_ALSA=m
CONFIG_VIDEO_EM28XX_DVB=m
CONFIG_VIDEO_TLG2300=m
CONFIG_VIDEO_CX231XX=m
CONFIG_VIDEO_CX231XX_RC=y
CONFIG_VIDEO_CX231XX_ALSA=m
CONFIG_VIDEO_CX231XX_DVB=m
CONFIG_VIDEO_USBVISION=m
# CONFIG_USB_ET61X251 is not set
# CONFIG_USB_SN9C102 is not set
CONFIG_USB_PWC=m
# CONFIG_USB_PWC_DEBUG is not set
CONFIG_USB_PWC_INPUT_EVDEV=y
CONFIG_USB_ZR364XX=m
CONFIG_USB_STKWEBCAM=m
CONFIG_USB_S2255=m
CONFIG_V4L_MEM2MEM_DRIVERS=y
# CONFIG_VIDEO_MEM2MEM_TESTDEV is not set
CONFIG_RADIO_ADAPTERS=y
CONFIG_RADIO_MAXIRADIO=m
CONFIG_RADIO_MAESTRO=m
CONFIG_I2C_SI4713=m
CONFIG_RADIO_SI4713=m
CONFIG_USB_DSBR=m
CONFIG_RADIO_SI470X=y
CONFIG_USB_SI470X=m
CONFIG_I2C_SI470X=m
CONFIG_USB_MR800=m
# CONFIG_RADIO_TEA5764 is not set
# CONFIG_RADIO_SAA7706H is not set
# CONFIG_RADIO_TEF6862 is not set
CONFIG_RADIO_WL1273=m
CONFIG_DVB_MAX_ADAPTERS=8
CONFIG_DVB_DYNAMIC_MINORS=y
CONFIG_DVB_CAPTURE_DRIVERS=y
#
# Supported SAA7146 based PCI Adapters
#
CONFIG_TTPCI_EEPROM=m
CONFIG_DVB_AV7110=m
CONFIG_DVB_AV7110_OSD=y
CONFIG_DVB_BUDGET_CORE=m
CONFIG_DVB_BUDGET=m
CONFIG_DVB_BUDGET_CI=m
CONFIG_DVB_BUDGET_AV=m
CONFIG_DVB_BUDGET_PATCH=m
#
# Supported USB Adapters
#
CONFIG_DVB_USB=m
# CONFIG_DVB_USB_DEBUG is not set
CONFIG_DVB_USB_A800=m
CONFIG_DVB_USB_DIBUSB_MB=m
# CONFIG_DVB_USB_DIBUSB_MB_FAULTY is not set
CONFIG_DVB_USB_DIBUSB_MC=m
CONFIG_DVB_USB_DIB0700=m
CONFIG_DVB_USB_UMT_010=m
CONFIG_DVB_USB_CXUSB=m
CONFIG_DVB_USB_M920X=m
CONFIG_DVB_USB_GL861=m
CONFIG_DVB_USB_AU6610=m
CONFIG_DVB_USB_DIGITV=m
CONFIG_DVB_USB_VP7045=m
CONFIG_DVB_USB_VP702X=m
CONFIG_DVB_USB_GP8PSK=m
CONFIG_DVB_USB_NOVA_T_USB2=m
CONFIG_DVB_USB_TTUSB2=m
CONFIG_DVB_USB_DTT200U=m
CONFIG_DVB_USB_OPERA1=m
CONFIG_DVB_USB_AF9005=m
CONFIG_DVB_USB_AF9005_REMOTE=m
CONFIG_DVB_USB_DW2102=m
CONFIG_DVB_USB_CINERGY_T2=m
CONFIG_DVB_USB_ANYSEE=m
CONFIG_DVB_USB_DTV5100=m
CONFIG_DVB_USB_AF9015=m
CONFIG_DVB_USB_CE6230=m
CONFIG_DVB_USB_FRIIO=m
CONFIG_DVB_USB_EC168=m
CONFIG_DVB_USB_AZ6027=m
CONFIG_DVB_USB_LME2510=m
CONFIG_DVB_TTUSB_BUDGET=m
CONFIG_DVB_TTUSB_DEC=m
CONFIG_SMS_SIANO_MDTV=m
#
# Siano module components
#
CONFIG_SMS_USB_DRV=m
CONFIG_SMS_SDIO_DRV=m
#
# Supported FlexCopII (B2C2) Adapters
#
CONFIG_DVB_B2C2_FLEXCOP=m
CONFIG_DVB_B2C2_FLEXCOP_PCI=m
CONFIG_DVB_B2C2_FLEXCOP_USB=m
# CONFIG_DVB_B2C2_FLEXCOP_DEBUG is not set
#
# Supported BT878 Adapters
#
CONFIG_DVB_BT8XX=m
#
# Supported Pluto2 Adapters
#
CONFIG_DVB_PLUTO2=m
#
# Supported SDMC DM1105 Adapters
#
CONFIG_DVB_DM1105=m
CONFIG_DVB_FIREDTV=m
CONFIG_DVB_FIREDTV_FIREWIRE=y
# CONFIG_DVB_FIREDTV_IEEE1394 is not set
CONFIG_DVB_FIREDTV_INPUT=y
#
# Supported Earthsoft PT1 Adapters
#
CONFIG_DVB_PT1=m
#
# Supported Mantis Adapters
#
CONFIG_MANTIS_CORE=m
CONFIG_DVB_MANTIS=m
CONFIG_DVB_HOPPER=m
#
# Supported nGene Adapters
#
CONFIG_DVB_NGENE=m
#
# Supported DVB Frontends
#
CONFIG_DVB_FE_CUSTOMISE=y
#
# Customise DVB Frontends
#
#
# Multistandard (satellite) frontends
#
CONFIG_DVB_STB0899=m
CONFIG_DVB_STB6100=m
CONFIG_DVB_STV090x=m
CONFIG_DVB_STV6110x=m
#
# DVB-S (satellite) frontends
#
CONFIG_DVB_CX24110=m
CONFIG_DVB_CX24123=m
CONFIG_DVB_MT312=m
CONFIG_DVB_ZL10036=m
CONFIG_DVB_ZL10039=m
CONFIG_DVB_S5H1420=m
CONFIG_DVB_STV0288=m
CONFIG_DVB_STB6000=m
CONFIG_DVB_STV0299=m
CONFIG_DVB_STV6110=m
CONFIG_DVB_STV0900=m
CONFIG_DVB_TDA8083=m
CONFIG_DVB_TDA10086=m
CONFIG_DVB_TDA8261=m
CONFIG_DVB_VES1X93=m
CONFIG_DVB_TUNER_ITD1000=m
CONFIG_DVB_TUNER_CX24113=m
CONFIG_DVB_TDA826X=m
CONFIG_DVB_TUA6100=m
CONFIG_DVB_CX24116=m
CONFIG_DVB_SI21XX=m
CONFIG_DVB_DS3000=m
CONFIG_DVB_MB86A16=m
#
# DVB-T (terrestrial) frontends
#
CONFIG_DVB_SP8870=m
CONFIG_DVB_SP887X=m
CONFIG_DVB_CX22700=m
CONFIG_DVB_CX22702=m
CONFIG_DVB_S5H1432=m
CONFIG_DVB_DRX397XD=m
CONFIG_DVB_L64781=m
CONFIG_DVB_TDA1004X=m
CONFIG_DVB_NXT6000=m
CONFIG_DVB_MT352=m
CONFIG_DVB_ZL10353=m
CONFIG_DVB_DIB3000MB=m
CONFIG_DVB_DIB3000MC=m
CONFIG_DVB_DIB7000M=m
CONFIG_DVB_DIB7000P=m
CONFIG_DVB_TDA10048=m
CONFIG_DVB_AF9013=m
CONFIG_DVB_EC100=m
#
# DVB-C (cable) frontends
#
CONFIG_DVB_VES1820=m
CONFIG_DVB_TDA10021=m
CONFIG_DVB_TDA10023=m
CONFIG_DVB_STV0297=m
#
# ATSC (North American/Korean Terrestrial/Cable DTV) frontends
#
CONFIG_DVB_NXT200X=m
CONFIG_DVB_OR51211=m
CONFIG_DVB_OR51132=m
CONFIG_DVB_BCM3510=m
CONFIG_DVB_LGDT330X=m
CONFIG_DVB_LGDT3305=m
CONFIG_DVB_S5H1409=m
CONFIG_DVB_AU8522=m
CONFIG_DVB_S5H1411=m
#
# ISDB-T (terrestrial) frontends
#
CONFIG_DVB_S921=m
CONFIG_DVB_DIB8000=m
CONFIG_DVB_MB86A20S=m
#
# Digital terrestrial only tuners/PLL
#
CONFIG_DVB_PLL=m
CONFIG_DVB_TUNER_DIB0070=m
CONFIG_DVB_TUNER_DIB0090=m
#
# SEC control devices for DVB-S
#
CONFIG_DVB_LNBP21=m
CONFIG_DVB_ISL6405=m
CONFIG_DVB_ISL6421=m
CONFIG_DVB_ISL6423=m
CONFIG_DVB_LGS8GL5=m
CONFIG_DVB_LGS8GXX=m
CONFIG_DVB_ATBM8830=m
CONFIG_DVB_TDA665x=m
CONFIG_DVB_IX2505V=m
#
# Tools to develop new frontends
#
CONFIG_DVB_DUMMY_FE=m
#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_AGP_SIS=y
CONFIG_AGP_VIA=y
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=16
CONFIG_VGA_SWITCHEROO=y
CONFIG_DRM=m
CONFIG_DRM_KMS_HELPER=m
CONFIG_DRM_TTM=m
CONFIG_DRM_TDFX=m
CONFIG_DRM_R128=m
CONFIG_DRM_RADEON=m
CONFIG_DRM_RADEON_KMS=y
CONFIG_DRM_I810=m
# CONFIG_DRM_I830 is not set
CONFIG_DRM_I915=m
CONFIG_DRM_I915_KMS=y
CONFIG_DRM_MGA=m
CONFIG_DRM_SIS=m
CONFIG_DRM_VIA=m
CONFIG_DRM_SAVAGE=m
CONFIG_STUB_POULSBO=m
CONFIG_VGASTATE=m
CONFIG_VIDEO_OUTPUT_CONTROL=m
CONFIG_FB=y
# CONFIG_FIRMWARE_EDID is not set
CONFIG_FB_DDC=m
CONFIG_FB_BOOT_VESA_SUPPORT=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
CONFIG_FB_SYS_FILLRECT=y
CONFIG_FB_SYS_COPYAREA=y
CONFIG_FB_SYS_IMAGEBLIT=y
# CONFIG_FB_FOREIGN_ENDIAN is not set
CONFIG_FB_SYS_FOPS=y
# CONFIG_FB_WMT_GE_ROPS is not set
CONFIG_FB_DEFERRED_IO=y
CONFIG_FB_SVGALIB=m
# CONFIG_FB_MACMODES is not set
CONFIG_FB_BACKLIGHT=y
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y
#
# Frame buffer hardware drivers
#
CONFIG_FB_CIRRUS=m
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
CONFIG_FB_VGA16=m
# CONFIG_FB_UVESA is not set
CONFIG_FB_VESA=y
CONFIG_FB_EFI=y
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_S1D13XXX is not set
CONFIG_FB_NVIDIA=m
CONFIG_FB_NVIDIA_I2C=y
# CONFIG_FB_NVIDIA_DEBUG is not set
CONFIG_FB_NVIDIA_BACKLIGHT=y
CONFIG_FB_RIVA=m
# CONFIG_FB_RIVA_I2C is not set
# CONFIG_FB_RIVA_DEBUG is not set
CONFIG_FB_RIVA_BACKLIGHT=y
# CONFIG_FB_LE80578 is not set
CONFIG_FB_MATROX=m
CONFIG_FB_MATROX_MILLENIUM=y
CONFIG_FB_MATROX_MYSTIQUE=y
CONFIG_FB_MATROX_G=y
CONFIG_FB_MATROX_I2C=m
CONFIG_FB_MATROX_MAVEN=m
CONFIG_FB_RADEON=m
CONFIG_FB_RADEON_I2C=y
CONFIG_FB_RADEON_BACKLIGHT=y
# CONFIG_FB_RADEON_DEBUG is not set
CONFIG_FB_ATY128=m
CONFIG_FB_ATY128_BACKLIGHT=y
CONFIG_FB_ATY=m
CONFIG_FB_ATY_CT=y
CONFIG_FB_ATY_GENERIC_LCD=y
CONFIG_FB_ATY_GX=y
CONFIG_FB_ATY_BACKLIGHT=y
CONFIG_FB_S3=m
CONFIG_FB_SAVAGE=m
CONFIG_FB_SAVAGE_I2C=y
CONFIG_FB_SAVAGE_ACCEL=y
# CONFIG_FB_SIS is not set
CONFIG_FB_VIA=m
# CONFIG_FB_VIA_DIRECT_PROCFS is not set
CONFIG_FB_NEOMAGIC=m
CONFIG_FB_KYRO=m
CONFIG_FB_3DFX=m
CONFIG_FB_3DFX_ACCEL=y
CONFIG_FB_3DFX_I2C=y
CONFIG_FB_VOODOO1=m
# CONFIG_FB_VT8623 is not set
CONFIG_FB_TRIDENT=m
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_CARMINE is not set
# CONFIG_FB_GEODE is not set
# CONFIG_FB_TMIO is not set
CONFIG_FB_SM501=m
# CONFIG_FB_UDL is not set
CONFIG_FB_VIRTUAL=m
CONFIG_XEN_FBDEV_FRONTEND=y
CONFIG_FB_METRONOME=m
CONFIG_FB_MB862XX=m
CONFIG_FB_MB862XX_PCI_GDC=y
# CONFIG_FB_BROADSHEET is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_LCD_CLASS_DEVICE=m
CONFIG_LCD_PLATFORM=m
CONFIG_BACKLIGHT_CLASS_DEVICE=y
# CONFIG_BACKLIGHT_GENERIC is not set
CONFIG_BACKLIGHT_PROGEAR=m
CONFIG_BACKLIGHT_APPLE=m
# CONFIG_BACKLIGHT_SAHARA is not set
# CONFIG_BACKLIGHT_ADP8860 is not set
#
# Display device support
#
CONFIG_DISPLAY_SUPPORT=m
#
# Display hardware drivers
#
#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_SOUND=m
CONFIG_SOUND_OSS_CORE=y
CONFIG_SOUND_OSS_CORE_PRECLAIM=y
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_JACK=y
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_HRTIMER=m
CONFIG_SND_SEQ_HRTIMER_DEFAULT=y
CONFIG_SND_DYNAMIC_MINORS=y
# CONFIG_SND_SUPPORT_OLD_API is not set
CONFIG_SND_VERBOSE_PROCFS=y
CONFIG_SND_VERBOSE_PRINTK=y
CONFIG_SND_DEBUG=y
# CONFIG_SND_DEBUG_VERBOSE is not set
CONFIG_SND_PCM_XRUN_DEBUG=y
CONFIG_SND_VMASTER=y
CONFIG_SND_DMA_SGBUF=y
CONFIG_SND_RAWMIDI_SEQ=m
CONFIG_SND_OPL3_LIB_SEQ=m
# CONFIG_SND_OPL4_LIB_SEQ is not set
# CONFIG_SND_SBAWE_SEQ is not set
CONFIG_SND_EMU10K1_SEQ=m
CONFIG_SND_MPU401_UART=m
CONFIG_SND_OPL3_LIB=m
CONFIG_SND_VX_LIB=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_DRIVERS=y
CONFIG_SND_PCSP=m
CONFIG_SND_DUMMY=m
CONFIG_SND_ALOOP=m
CONFIG_SND_VIRMIDI=m
CONFIG_SND_MTPAV=m
CONFIG_SND_MTS64=m
CONFIG_SND_SERIAL_U16550=m
CONFIG_SND_MPU401=m
CONFIG_SND_PORTMAN2X4=m
CONFIG_SND_AC97_POWER_SAVE=y
CONFIG_SND_AC97_POWER_SAVE_DEFAULT=0
CONFIG_SND_SB_COMMON=m
CONFIG_SND_SB16_DSP=m
CONFIG_SND_PCI=y
CONFIG_SND_AD1889=m
CONFIG_SND_ALS300=m
CONFIG_SND_ALS4000=m
CONFIG_SND_ALI5451=m
CONFIG_SND_ASIHPI=m
CONFIG_SND_ATIIXP=m
CONFIG_SND_ATIIXP_MODEM=m
CONFIG_SND_AU8810=m
CONFIG_SND_AU8820=m
CONFIG_SND_AU8830=m
# CONFIG_SND_AW2 is not set
CONFIG_SND_AZT3328=m
CONFIG_SND_BT87X=m
# CONFIG_SND_BT87X_OVERCLOCK is not set
CONFIG_SND_CA0106=m
CONFIG_SND_CMIPCI=m
CONFIG_SND_OXYGEN_LIB=m
CONFIG_SND_OXYGEN=m
CONFIG_SND_CS4281=m
CONFIG_SND_CS46XX=m
CONFIG_SND_CS46XX_NEW_DSP=y
CONFIG_SND_CS5530=m
CONFIG_SND_CS5535AUDIO=m
CONFIG_SND_CTXFI=m
CONFIG_SND_DARLA20=m
CONFIG_SND_GINA20=m
CONFIG_SND_LAYLA20=m
CONFIG_SND_DARLA24=m
CONFIG_SND_GINA24=m
CONFIG_SND_LAYLA24=m
CONFIG_SND_MONA=m
CONFIG_SND_MIA=m
CONFIG_SND_ECHO3G=m
CONFIG_SND_INDIGO=m
CONFIG_SND_INDIGOIO=m
CONFIG_SND_INDIGODJ=m
CONFIG_SND_INDIGOIOX=m
CONFIG_SND_INDIGODJX=m
CONFIG_SND_EMU10K1=m
CONFIG_SND_EMU10K1X=m
CONFIG_SND_ENS1370=m
CONFIG_SND_ENS1371=m
CONFIG_SND_ES1938=m
CONFIG_SND_ES1968=m
CONFIG_SND_ES1968_INPUT=y
CONFIG_SND_FM801=m
CONFIG_SND_FM801_TEA575X_BOOL=y
CONFIG_SND_FM801_TEA575X=m
CONFIG_SND_HDA_INTEL=m
CONFIG_SND_HDA_HWDEP=y
CONFIG_SND_HDA_RECONFIG=y
CONFIG_SND_HDA_INPUT_BEEP=y
CONFIG_SND_HDA_INPUT_BEEP_MODE=0
CONFIG_SND_HDA_INPUT_JACK=y
CONFIG_SND_HDA_PATCH_LOADER=y
CONFIG_SND_HDA_CODEC_REALTEK=y
CONFIG_SND_HDA_CODEC_ANALOG=y
CONFIG_SND_HDA_CODEC_SIGMATEL=y
CONFIG_SND_HDA_CODEC_VIA=y
CONFIG_SND_HDA_CODEC_HDMI=y
CONFIG_SND_HDA_CODEC_CIRRUS=y
CONFIG_SND_HDA_CODEC_CONEXANT=y
CONFIG_SND_HDA_CODEC_CA0110=y
CONFIG_SND_HDA_CODEC_CMEDIA=y
CONFIG_SND_HDA_CODEC_SI3054=y
CONFIG_SND_HDA_GENERIC=y
CONFIG_SND_HDA_POWER_SAVE=y
CONFIG_SND_HDA_POWER_SAVE_DEFAULT=0
CONFIG_SND_HDSP=m
CONFIG_SND_HDSPM=m
CONFIG_SND_ICE1712=m
CONFIG_SND_ICE1724=m
CONFIG_SND_INTEL8X0=m
CONFIG_SND_INTEL8X0M=m
CONFIG_SND_KORG1212=m
CONFIG_SND_LX6464ES=m
CONFIG_SND_MAESTRO3=m
CONFIG_SND_MAESTRO3_INPUT=y
CONFIG_SND_MIXART=m
CONFIG_SND_NM256=m
CONFIG_SND_PCXHR=m
CONFIG_SND_RIPTIDE=m
CONFIG_SND_RME32=m
CONFIG_SND_RME96=m
CONFIG_SND_RME9652=m
CONFIG_SND_SONICVIBES=m
CONFIG_SND_TRIDENT=m
CONFIG_SND_VIA82XX=m
CONFIG_SND_VIA82XX_MODEM=m
CONFIG_SND_VIRTUOSO=m
CONFIG_SND_VX222=m
CONFIG_SND_YMFPCI=m
CONFIG_SND_USB=y
CONFIG_SND_USB_AUDIO=m
CONFIG_SND_USB_UA101=m
CONFIG_SND_USB_USX2Y=m
CONFIG_SND_USB_CAIAQ=m
CONFIG_SND_USB_CAIAQ_INPUT=y
CONFIG_SND_USB_US122L=m
CONFIG_SND_PCMCIA=y
CONFIG_SND_VXPOCKET=m
CONFIG_SND_PDAUDIOCF=m
# CONFIG_SND_SOC is not set
# CONFIG_SOUND_PRIME is not set
CONFIG_AC97_BUS=m
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
CONFIG_HIDRAW=y
#
# USB Input Devices
#
CONFIG_USB_HID=y
CONFIG_HID_PID=y
CONFIG_USB_HIDDEV=y
#
# Special HID drivers
#
CONFIG_HID_3M_PCT=y
CONFIG_HID_A4TECH=y
CONFIG_HID_ACRUX=m
CONFIG_HID_ACRUX_FF=m
CONFIG_HID_APPLE=y
CONFIG_HID_BELKIN=y
CONFIG_HID_CANDO=m
CONFIG_HID_CHERRY=y
CONFIG_HID_CHICONY=y
CONFIG_HID_PRODIKEYS=m
CONFIG_HID_CYPRESS=y
CONFIG_HID_DRAGONRISE=m
CONFIG_DRAGONRISE_FF=y
CONFIG_HID_EMS_FF=m
CONFIG_HID_EGALAX=m
CONFIG_HID_ELECOM=m
CONFIG_HID_EZKEY=y
CONFIG_HID_KYE=y
CONFIG_HID_UCLOGIC=m
CONFIG_HID_WALTOP=m
CONFIG_HID_GYRATION=m
CONFIG_HID_TWINHAN=m
CONFIG_HID_KENSINGTON=y
CONFIG_HID_LOGITECH=y
CONFIG_LOGITECH_FF=y
CONFIG_LOGIRUMBLEPAD2_FF=y
CONFIG_LOGIG940_FF=y
CONFIG_LOGIWII_FF=y
CONFIG_HID_MAGICMOUSE=m
CONFIG_HID_MICROSOFT=y
CONFIG_HID_MOSART=y
CONFIG_HID_MONTEREY=y
CONFIG_HID_MULTITOUCH=m
CONFIG_HID_NTRIG=y
CONFIG_HID_ORTEK=m
CONFIG_HID_PANTHERLORD=m
CONFIG_PANTHERLORD_FF=y
CONFIG_HID_PETALYNX=m
CONFIG_HID_PICOLCD=m
CONFIG_HID_PICOLCD_FB=y
CONFIG_HID_PICOLCD_BACKLIGHT=y
CONFIG_HID_PICOLCD_LCD=y
CONFIG_HID_PICOLCD_LEDS=y
CONFIG_HID_QUANTA=y
CONFIG_HID_ROCCAT=m
CONFIG_HID_ROCCAT_KONE=m
CONFIG_HID_ROCCAT_KONEPLUS=m
CONFIG_HID_ROCCAT_PYRA=m
CONFIG_HID_SAMSUNG=m
CONFIG_HID_SONY=m
CONFIG_HID_STANTUM=y
CONFIG_HID_SUNPLUS=m
CONFIG_HID_GREENASIA=m
CONFIG_GREENASIA_FF=y
CONFIG_HID_SMARTJOYPLUS=m
CONFIG_SMARTJOYPLUS_FF=y
CONFIG_HID_TIVO_SLIDE=m
CONFIG_HID_TOPSEED=m
CONFIG_HID_THRUSTMASTER=m
CONFIG_THRUSTMASTER_FF=y
CONFIG_HID_WACOM=m
CONFIG_HID_WACOM_POWER_SUPPLY=y
CONFIG_HID_ZEROPLUS=m
CONFIG_ZEROPLUS_FF=y
CONFIG_HID_ZYDACRON=m
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y
#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_DEVICE_CLASS is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
CONFIG_USB_SUSPEND=y
# CONFIG_USB_OTG is not set
CONFIG_USB_MON=y
CONFIG_USB_WUSB=m
CONFIG_USB_WUSB_CBAF=m
# CONFIG_USB_WUSB_CBAF_DEBUG is not set
#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
CONFIG_USB_XHCI_HCD=m
# CONFIG_USB_XHCI_HCD_DEBUGGING is not set
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
# CONFIG_USB_OXU210HP_HCD is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_ISP1760_HCD is not set
CONFIG_USB_ISP1362_HCD=m
CONFIG_USB_OHCI_HCD=y
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=y
CONFIG_USB_U132_HCD=m
CONFIG_USB_SL811_HCD=m
# CONFIG_USB_SL811_CS is not set
# CONFIG_USB_R8A66597_HCD is not set
CONFIG_USB_WHCI_HCD=m
CONFIG_USB_HWA_HCD=m
#
# USB Device Class drivers
#
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
CONFIG_USB_WDM=m
CONFIG_USB_TMC=m
#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
#
#
# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
CONFIG_USB_STORAGE_DATAFAB=m
CONFIG_USB_STORAGE_FREECOM=m
CONFIG_USB_STORAGE_ISD200=m
CONFIG_USB_STORAGE_USBAT=m
CONFIG_USB_STORAGE_SDDR09=m
CONFIG_USB_STORAGE_SDDR55=m
CONFIG_USB_STORAGE_JUMPSHOT=m
CONFIG_USB_STORAGE_ALAUDA=m
CONFIG_USB_STORAGE_ONETOUCH=m
CONFIG_USB_STORAGE_KARMA=m
CONFIG_USB_STORAGE_CYPRESS_ATACB=m
CONFIG_USB_UAS=m
# CONFIG_USB_LIBUSUAL is not set
#
# USB Imaging devices
#
CONFIG_USB_MDC800=m
CONFIG_USB_MICROTEK=m
#
# USB port drivers
#
CONFIG_USB_USS720=m
CONFIG_USB_SERIAL=m
CONFIG_USB_EZUSB=y
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_AIRCABLE=m
CONFIG_USB_SERIAL_ARK3116=m
CONFIG_USB_SERIAL_BELKIN=m
CONFIG_USB_SERIAL_CH341=m
CONFIG_USB_SERIAL_WHITEHEAT=m
CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
CONFIG_USB_SERIAL_CP210X=m
CONFIG_USB_SERIAL_CYPRESS_M8=m
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_FUNSOFT=m
CONFIG_USB_SERIAL_VISOR=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
CONFIG_USB_SERIAL_GARMIN=m
CONFIG_USB_SERIAL_IPW=m
CONFIG_USB_SERIAL_IUU=m
CONFIG_USB_SERIAL_KEYSPAN_PDA=m
CONFIG_USB_SERIAL_KEYSPAN=m
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
CONFIG_USB_SERIAL_MOS7720=m
CONFIG_USB_SERIAL_MOS7715_PARPORT=y
CONFIG_USB_SERIAL_MOS7840=m
CONFIG_USB_SERIAL_MOTOROLA=m
CONFIG_USB_SERIAL_NAVMAN=m
CONFIG_USB_SERIAL_PL2303=m
CONFIG_USB_SERIAL_OTI6858=m
CONFIG_USB_SERIAL_QCAUX=m
CONFIG_USB_SERIAL_QUALCOMM=m
CONFIG_USB_SERIAL_SPCP8X5=m
CONFIG_USB_SERIAL_HP4X=m
CONFIG_USB_SERIAL_SAFE=m
CONFIG_USB_SERIAL_SAFE_PADDED=y
CONFIG_USB_SERIAL_SAMBA=m
CONFIG_USB_SERIAL_SIEMENS_MPI=m
CONFIG_USB_SERIAL_SIERRAWIRELESS=m
CONFIG_USB_SERIAL_SYMBOL=m
CONFIG_USB_SERIAL_TI=m
CONFIG_USB_SERIAL_CYBERJACK=m
CONFIG_USB_SERIAL_XIRCOM=m
CONFIG_USB_SERIAL_WWAN=m
CONFIG_USB_SERIAL_OPTION=m
CONFIG_USB_SERIAL_OMNINET=m
CONFIG_USB_SERIAL_OPTICON=m
CONFIG_USB_SERIAL_VIVOPAY_SERIAL=m
# CONFIG_USB_SERIAL_ZIO is not set
CONFIG_USB_SERIAL_SSU100=m
CONFIG_USB_SERIAL_DEBUG=m
#
# USB Miscellaneous drivers
#
CONFIG_USB_EMI62=m
CONFIG_USB_EMI26=m
CONFIG_USB_ADUTUX=m
CONFIG_USB_SEVSEG=m
# CONFIG_USB_RIO500 is not set
CONFIG_USB_LEGOTOWER=m
CONFIG_USB_LCD=m
CONFIG_USB_LED=m
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
CONFIG_USB_IDMOUSE=m
CONFIG_USB_FTDI_ELAN=m
CONFIG_USB_APPLEDISPLAY=m
CONFIG_USB_SISUSBVGA=m
CONFIG_USB_SISUSBVGA_CON=y
CONFIG_USB_LD=m
CONFIG_USB_TRANCEVIBRATOR=m
CONFIG_USB_IOWARRIOR=m
# CONFIG_USB_TEST is not set
CONFIG_USB_ISIGHTFW=m
CONFIG_USB_YUREX=m
CONFIG_USB_ATM=m
CONFIG_USB_SPEEDTOUCH=m
CONFIG_USB_CXACRU=m
CONFIG_USB_UEAGLEATM=m
CONFIG_USB_XUSBATM=m
# CONFIG_USB_GADGET is not set
#
# OTG and related infrastructure
#
CONFIG_USB_OTG_UTILS=y
# CONFIG_USB_GPIO_VBUS is not set
CONFIG_NOP_USB_XCEIV=m
CONFIG_UWB=m
CONFIG_UWB_HWA=m
CONFIG_UWB_WHCI=m
CONFIG_UWB_I1480U=m
CONFIG_MMC=m
# CONFIG_MMC_DEBUG is not set
# CONFIG_MMC_UNSAFE_RESUME is not set
# CONFIG_MMC_CLKGATE is not set
#
# MMC/SD/SDIO Card Drivers
#
CONFIG_MMC_BLOCK=m
CONFIG_MMC_BLOCK_MINORS=8
CONFIG_MMC_BLOCK_BOUNCE=y
CONFIG_SDIO_UART=m
# CONFIG_MMC_TEST is not set
#
# MMC/SD/SDIO Host Controller Drivers
#
CONFIG_MMC_SDHCI=m
CONFIG_MMC_SDHCI_PCI=m
CONFIG_MMC_RICOH_MMC=y
CONFIG_MMC_SDHCI_PLTFM=m
CONFIG_MMC_WBSD=m
CONFIG_MMC_TIFM_SD=m
CONFIG_MMC_SDRICOH_CS=m
CONFIG_MMC_CB710=m
CONFIG_MMC_VIA_SDMMC=m
CONFIG_MMC_USHC=m
CONFIG_MEMSTICK=m
# CONFIG_MEMSTICK_DEBUG is not set
#
# MemoryStick drivers
#
# CONFIG_MEMSTICK_UNSAFE_RESUME is not set
CONFIG_MSPRO_BLOCK=m
#
# MemoryStick Host Controller Drivers
#
CONFIG_MEMSTICK_TIFM_MS=m
CONFIG_MEMSTICK_JMICRON_38X=m
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y
#
# LED drivers
#
CONFIG_LEDS_ALIX2=m
# CONFIG_LEDS_PCA9532 is not set
# CONFIG_LEDS_GPIO is not set
CONFIG_LEDS_LP3944=m
CONFIG_LEDS_LP5521=m
CONFIG_LEDS_LP5523=m
CONFIG_LEDS_CLEVO_MAIL=m
# CONFIG_LEDS_PCA955X is not set
# CONFIG_LEDS_BD2802 is not set
CONFIG_LEDS_INTEL_SS4200=m
CONFIG_LEDS_LT3593=m
CONFIG_LEDS_DELL_NETBOOKS=m
CONFIG_LEDS_TRIGGERS=y
#
# LED Triggers
#
CONFIG_LEDS_TRIGGER_TIMER=m
CONFIG_LEDS_TRIGGER_HEARTBEAT=m
CONFIG_LEDS_TRIGGER_BACKLIGHT=m
CONFIG_LEDS_TRIGGER_GPIO=m
CONFIG_LEDS_TRIGGER_DEFAULT_ON=m
#
# iptables trigger is under Netfilter config (LED target)
#
CONFIG_NFC_DEVICES=y
CONFIG_PN544_NFC=m
CONFIG_ACCESSIBILITY=y
CONFIG_A11Y_BRAILLE_CONSOLE=y
CONFIG_INFINIBAND=m
CONFIG_INFINIBAND_USER_MAD=m
CONFIG_INFINIBAND_USER_ACCESS=m
CONFIG_INFINIBAND_USER_MEM=y
CONFIG_INFINIBAND_ADDR_TRANS=y
CONFIG_INFINIBAND_MTHCA=m
CONFIG_INFINIBAND_MTHCA_DEBUG=y
CONFIG_INFINIBAND_IPATH=m
CONFIG_INFINIBAND_QIB=m
CONFIG_INFINIBAND_AMSO1100=m
# CONFIG_INFINIBAND_AMSO1100_DEBUG is not set
CONFIG_INFINIBAND_CXGB3=m
# CONFIG_INFINIBAND_CXGB3_DEBUG is not set
CONFIG_INFINIBAND_CXGB4=m
CONFIG_MLX4_INFINIBAND=m
CONFIG_INFINIBAND_NES=m
# CONFIG_INFINIBAND_NES_DEBUG is not set
CONFIG_INFINIBAND_IPOIB=m
CONFIG_INFINIBAND_IPOIB_CM=y
CONFIG_INFINIBAND_IPOIB_DEBUG=y
CONFIG_INFINIBAND_IPOIB_DEBUG_DATA=y
CONFIG_INFINIBAND_SRP=m
CONFIG_INFINIBAND_ISER=m
CONFIG_EDAC=y
#
# Reporting subsystems
#
# CONFIG_EDAC_DEBUG is not set
CONFIG_EDAC_DECODE_MCE=m
CONFIG_EDAC_MCE_INJ=m
CONFIG_EDAC_MM_EDAC=m
CONFIG_EDAC_MCE=y
CONFIG_EDAC_AMD64=m
# CONFIG_EDAC_AMD64_ERROR_INJECTION is not set
CONFIG_EDAC_E752X=m
CONFIG_EDAC_I82975X=m
CONFIG_EDAC_I3000=m
CONFIG_EDAC_I3200=m
CONFIG_EDAC_X38=m
CONFIG_EDAC_I5400=m
CONFIG_EDAC_I7CORE=m
CONFIG_EDAC_I5000=m
CONFIG_EDAC_I5100=m
CONFIG_EDAC_I7300=m
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"
# CONFIG_RTC_DEBUG is not set
#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set
#
# I2C RTC drivers
#
CONFIG_RTC_DRV_DS1307=m
CONFIG_RTC_DRV_DS1374=m
CONFIG_RTC_DRV_DS1672=m
CONFIG_RTC_DRV_DS3232=m
CONFIG_RTC_DRV_MAX6900=m
CONFIG_RTC_DRV_RS5C372=m
CONFIG_RTC_DRV_ISL1208=m
CONFIG_RTC_DRV_ISL12022=m
CONFIG_RTC_DRV_X1205=m
CONFIG_RTC_DRV_PCF8563=m
CONFIG_RTC_DRV_PCF8583=m
CONFIG_RTC_DRV_M41T80=m
CONFIG_RTC_DRV_M41T80_WDT=y
CONFIG_RTC_DRV_BQ32K=m
# CONFIG_RTC_DRV_S35390A is not set
CONFIG_RTC_DRV_FM3130=m
CONFIG_RTC_DRV_RX8581=m
CONFIG_RTC_DRV_RX8025=m
#
# SPI RTC drivers
#
#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=y
CONFIG_RTC_DRV_DS1286=m
CONFIG_RTC_DRV_DS1511=m
CONFIG_RTC_DRV_DS1553=m
CONFIG_RTC_DRV_DS1742=m
CONFIG_RTC_DRV_STK17TA8=m
# CONFIG_RTC_DRV_M48T86 is not set
CONFIG_RTC_DRV_M48T35=m
CONFIG_RTC_DRV_M48T59=m
CONFIG_RTC_DRV_MSM6242=m
CONFIG_RTC_DRV_BQ4802=m
CONFIG_RTC_DRV_RP5C01=m
CONFIG_RTC_DRV_V3020=m
#
# on-CPU RTC drivers
#
CONFIG_DMADEVICES=y
# CONFIG_DMADEVICES_DEBUG is not set
#
# DMA Devices
#
# CONFIG_INTEL_MID_DMAC is not set
CONFIG_INTEL_IOATDMA=m
# CONFIG_TIMB_DMA is not set
CONFIG_PCH_DMA=m
CONFIG_DMA_ENGINE=y
#
# DMA Clients
#
CONFIG_NET_DMA=y
CONFIG_ASYNC_TX_DMA=y
# CONFIG_DMATEST is not set
CONFIG_DCA=m
CONFIG_AUXDISPLAY=y
CONFIG_KS0108=m
CONFIG_KS0108_PORT=0x378
CONFIG_KS0108_DELAY=2
CONFIG_CFAG12864B=m
CONFIG_CFAG12864B_RATE=20
CONFIG_UIO=m
CONFIG_UIO_CIF=m
# CONFIG_UIO_PDRV is not set
# CONFIG_UIO_PDRV_GENIRQ is not set
CONFIG_UIO_AEC=m
CONFIG_UIO_SERCOS3=m
CONFIG_UIO_PCI_GENERIC=m
# CONFIG_UIO_NETX is not set
#
# Xen driver support
#
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=m
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=m
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=m
CONFIG_XEN_PLATFORM_PCI=m
CONFIG_SWIOTLB_XEN=y
CONFIG_STAGING=y
# CONFIG_STAGING_EXCLUDE_BUILD is not set
# CONFIG_ET131X is not set
# CONFIG_SLICOSS is not set
# CONFIG_VIDEO_GO7007 is not set
# CONFIG_VIDEO_CX25821 is not set
# CONFIG_VIDEO_TM6000 is not set
CONFIG_USB_DABUSB=m
CONFIG_USB_SE401=m
CONFIG_VIDEO_USBVIDEO=m
CONFIG_USB_VICAM=m
# CONFIG_USB_IP_COMMON is not set
# CONFIG_W35UND is not set
# CONFIG_PRISM2_USB is not set
# CONFIG_ECHO is not set
# CONFIG_BRCM80211 is not set
# CONFIG_RT2860 is not set
# CONFIG_RT2870 is not set
# CONFIG_COMEDI is not set
# CONFIG_ASUS_OLED is not set
# CONFIG_PANEL is not set
# CONFIG_R8187SE is not set
# CONFIG_RTL8192U is not set
# CONFIG_RTL8192E is not set
# CONFIG_R8712U is not set
# CONFIG_TRANZPORT is not set
# CONFIG_POHMELFS is not set
# CONFIG_AUTOFS_FS is not set
# CONFIG_IDE_PHISON is not set
# CONFIG_LINE6_USB is not set
CONFIG_DRM_VMWGFX=m
CONFIG_DRM_NOUVEAU=m
CONFIG_DRM_NOUVEAU_BACKLIGHT=y
CONFIG_DRM_NOUVEAU_DEBUG=y
#
# I2C encoder or helper chips
#
CONFIG_DRM_I2C_CH7006=m
CONFIG_DRM_I2C_SIL164=m
# CONFIG_USB_SERIAL_QUATECH2 is not set
# CONFIG_USB_SERIAL_QUATECH_USB2 is not set
# CONFIG_VT6655 is not set
# CONFIG_VT6656 is not set
# CONFIG_HYPERV is not set
# CONFIG_VME_BUS is not set
# CONFIG_DX_SEP is not set
# CONFIG_IIO is not set
# CONFIG_ZRAM is not set
# CONFIG_WLAGS49_H2 is not set
# CONFIG_WLAGS49_H25 is not set
# CONFIG_SAMSUNG_LAPTOP is not set
# CONFIG_FB_SM7XX is not set
# CONFIG_VIDEO_DT3155 is not set
CONFIG_CRYSTALHD=m
#
# Texas Instruments shared transport line discipline
#
# CONFIG_ST_BT is not set
# CONFIG_FB_XGI is not set
CONFIG_LIRC_STAGING=y
CONFIG_LIRC_BT829=m
CONFIG_LIRC_IGORPLUGUSB=m
CONFIG_LIRC_IMON=m
CONFIG_LIRC_PARALLEL=m
CONFIG_LIRC_SASEM=m
CONFIG_LIRC_SERIAL=m
CONFIG_LIRC_SERIAL_TRANSMITTER=y
CONFIG_LIRC_SIR=m
CONFIG_LIRC_TTUSBIR=m
CONFIG_LIRC_ZILOG=m
# CONFIG_SMB_FS is not set
# CONFIG_EASYCAP is not set
# CONFIG_SOLO6X10 is not set
# CONFIG_ACPI_QUICKSTART is not set
CONFIG_MACH_NO_WESTBRIDGE=y
# CONFIG_ATH6K_LEGACY is not set
# CONFIG_USB_ENESTORAGE is not set
# CONFIG_BCM_WIMAX is not set
# CONFIG_FT1000 is not set
#
# Speakup console speech
#
# CONFIG_SPEAKUP is not set
# CONFIG_TOUCHSCREEN_CLEARPAD_TM1217 is not set
# CONFIG_TOUCHSCREEN_SYNAPTICS_I2C_RMI4 is not set
CONFIG_X86_PLATFORM_DEVICES=y
CONFIG_ACER_WMI=m
CONFIG_ACERHDF=m
CONFIG_ASUS_LAPTOP=m
CONFIG_DELL_LAPTOP=m
CONFIG_DELL_WMI=m
CONFIG_FUJITSU_LAPTOP=m
# CONFIG_FUJITSU_LAPTOP_DEBUG is not set
CONFIG_HP_WMI=m
CONFIG_MSI_LAPTOP=m
CONFIG_PANASONIC_LAPTOP=m
CONFIG_COMPAL_LAPTOP=m
CONFIG_SONY_LAPTOP=m
CONFIG_SONYPI_COMPAT=y
CONFIG_IDEAPAD_LAPTOP=m
CONFIG_THINKPAD_ACPI=m
CONFIG_THINKPAD_ACPI_ALSA_SUPPORT=y
# CONFIG_THINKPAD_ACPI_DEBUGFACILITIES is not set
# CONFIG_THINKPAD_ACPI_DEBUG is not set
# CONFIG_THINKPAD_ACPI_UNSAFE_LEDS is not set
CONFIG_THINKPAD_ACPI_VIDEO=y
CONFIG_THINKPAD_ACPI_HOTKEY_POLL=y
CONFIG_SENSORS_HDAPS=m
# CONFIG_INTEL_MENLOW is not set
CONFIG_EEEPC_LAPTOP=m
CONFIG_EEEPC_WMI=m
CONFIG_ACPI_WMI=m
CONFIG_MSI_WMI=m
# CONFIG_ACPI_ASUS is not set
CONFIG_TOPSTAR_LAPTOP=m
CONFIG_ACPI_TOSHIBA=m
CONFIG_TOSHIBA_BT_RFKILL=m
CONFIG_ACPI_CMPC=m
CONFIG_INTEL_IPS=m
# CONFIG_IBM_RTL is not set
#
# Firmware Drivers
#
CONFIG_EDD=m
# CONFIG_EDD_OFF is not set
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_EFI_VARS=y
CONFIG_DELL_RBU=m
CONFIG_DCDBAS=m
CONFIG_DMIID=y
CONFIG_ISCSI_IBFT_FIND=y
CONFIG_ISCSI_IBFT=m
#
# File systems
#
CONFIG_EXT2_FS=m
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT2_FS_XIP=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_DEFAULTS_TO_ORDERED=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_XATTR=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
# CONFIG_EXT4_DEBUG is not set
CONFIG_FS_XIP=y
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_JBD2=y
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=m
# CONFIG_REISERFS_CHECK is not set
CONFIG_REISERFS_PROC_INFO=y
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
CONFIG_REISERFS_FS_SECURITY=y
CONFIG_JFS_FS=m
CONFIG_JFS_POSIX_ACL=y
CONFIG_JFS_SECURITY=y
# CONFIG_JFS_DEBUG is not set
# CONFIG_JFS_STATISTICS is not set
CONFIG_XFS_FS=m
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
# CONFIG_XFS_RT is not set
# CONFIG_XFS_DEBUG is not set
CONFIG_GFS2_FS=m
CONFIG_GFS2_FS_LOCKING_DLM=y
CONFIG_OCFS2_FS=m
CONFIG_OCFS2_FS_O2CB=m
CONFIG_OCFS2_FS_USERSPACE_CLUSTER=m
# CONFIG_OCFS2_FS_STATS is not set
# CONFIG_OCFS2_DEBUG_MASKLOG is not set
# CONFIG_OCFS2_DEBUG_FS is not set
CONFIG_BTRFS_FS=m
CONFIG_BTRFS_FS_POSIX_ACL=y
CONFIG_NILFS2_FS=m
CONFIG_FS_POSIX_ACL=y
CONFIG_EXPORTFS=m
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y
CONFIG_QUOTA=y
CONFIG_QUOTA_NETLINK_INTERFACE=y
# CONFIG_PRINT_QUOTA_WARNING is not set
# CONFIG_QUOTA_DEBUG is not set
CONFIG_QUOTA_TREE=y
# CONFIG_QFMT_V1 is not set
CONFIG_QFMT_V2=y
CONFIG_QUOTACTL=y
CONFIG_QUOTACTL_COMPAT=y
CONFIG_AUTOFS4_FS=y
CONFIG_FUSE_FS=m
CONFIG_CUSE=m
CONFIG_GENERIC_ACL=y
#
# Caches
#
CONFIG_FSCACHE=m
CONFIG_FSCACHE_STATS=y
# CONFIG_FSCACHE_HISTOGRAM is not set
# CONFIG_FSCACHE_DEBUG is not set
CONFIG_FSCACHE_OBJECT_LIST=y
CONFIG_CACHEFILES=m
# CONFIG_CACHEFILES_DEBUG is not set
# CONFIG_CACHEFILES_HISTOGRAM is not set
#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y
#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="ascii"
# CONFIG_NTFS_FS is not set
#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_VMCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=m
CONFIG_MISC_FILESYSTEMS=y
# CONFIG_ADFS_FS is not set
CONFIG_AFFS_FS=m
CONFIG_ECRYPT_FS=m
CONFIG_HFS_FS=m
CONFIG_HFSPLUS_FS=m
CONFIG_BEFS_FS=m
# CONFIG_BEFS_DEBUG is not set
CONFIG_BFS_FS=m
CONFIG_EFS_FS=m
CONFIG_JFFS2_FS=m
CONFIG_JFFS2_FS_DEBUG=0
CONFIG_JFFS2_FS_WRITEBUFFER=y
# CONFIG_JFFS2_FS_WBUF_VERIFY is not set
CONFIG_JFFS2_SUMMARY=y
CONFIG_JFFS2_FS_XATTR=y
CONFIG_JFFS2_FS_POSIX_ACL=y
CONFIG_JFFS2_FS_SECURITY=y
# CONFIG_JFFS2_COMPRESSION_OPTIONS is not set
CONFIG_JFFS2_ZLIB=y
# CONFIG_JFFS2_LZO is not set
CONFIG_JFFS2_RTIME=y
# CONFIG_JFFS2_RUBIN is not set
CONFIG_UBIFS_FS=m
CONFIG_UBIFS_FS_XATTR=y
# CONFIG_UBIFS_FS_ADVANCED_COMPR is not set
CONFIG_UBIFS_FS_LZO=y
CONFIG_UBIFS_FS_ZLIB=y
# CONFIG_UBIFS_FS_DEBUG is not set
CONFIG_LOGFS=m
CONFIG_CRAMFS=m
CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_XATTR=y
CONFIG_SQUASHFS_LZO=y
CONFIG_SQUASHFS_XZ=y
# CONFIG_SQUASHFS_EMBEDDED is not set
CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3
CONFIG_VXFS_FS=m
CONFIG_MINIX_FS=m
CONFIG_OMFS_FS=m
# CONFIG_HPFS_FS is not set
CONFIG_QNX4FS_FS=m
CONFIG_ROMFS_FS=m
CONFIG_ROMFS_BACKED_BY_BLOCK=y
# CONFIG_ROMFS_BACKED_BY_MTD is not set
# CONFIG_ROMFS_BACKED_BY_BOTH is not set
CONFIG_ROMFS_ON_BLOCK=y
CONFIG_SYSV_FS=m
CONFIG_UFS_FS=m
# CONFIG_UFS_FS_WRITE is not set
# CONFIG_UFS_DEBUG is not set
CONFIG_EXOFS_FS=m
# CONFIG_EXOFS_DEBUG is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=m
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
CONFIG_NFS_V4_1=y
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_NFS_FSCACHE=y
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
# CONFIG_NFS_USE_NEW_IDMAPPER is not set
CONFIG_NFSD=m
# CONFIG_NFSD_DEPRECATED is not set
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_SUNRPC_XPRT_RDMA=m
CONFIG_RPCSEC_GSS_KRB5=m
CONFIG_CEPH_FS=m
CONFIG_CIFS=m
CONFIG_CIFS_STATS=y
# CONFIG_CIFS_STATS2 is not set
CONFIG_CIFS_WEAK_PW_HASH=y
CONFIG_CIFS_UPCALL=y
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
# CONFIG_CIFS_DEBUG2 is not set
CONFIG_CIFS_DFS_UPCALL=y
CONFIG_CIFS_FSCACHE=y
CONFIG_CIFS_ACL=y
CONFIG_CIFS_EXPERIMENTAL=y
CONFIG_NCP_FS=m
CONFIG_NCPFS_PACKET_SIGNING=y
CONFIG_NCPFS_IOCTL_LOCKING=y
CONFIG_NCPFS_STRONG=y
CONFIG_NCPFS_NFS_NS=y
CONFIG_NCPFS_OS2_NS=y
CONFIG_NCPFS_SMALLDOS=y
CONFIG_NCPFS_NLS=y
CONFIG_NCPFS_EXTRAS=y
CONFIG_CODA_FS=m
# CONFIG_AFS_FS is not set
CONFIG_9P_FS=m
CONFIG_9P_FSCACHE=y
CONFIG_9P_FS_POSIX_ACL=y
#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
# CONFIG_ATARI_PARTITION is not set
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
# CONFIG_LDM_PARTITION is not set
CONFIG_SGI_PARTITION=y
# CONFIG_ULTRIX_PARTITION is not set
CONFIG_SUN_PARTITION=y
CONFIG_KARMA_PARTITION=y
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_UTF8=m
CONFIG_DLM=m
CONFIG_DLM_DEBUG=y
#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
# CONFIG_ENABLE_WARN_DEPRECATED is not set
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
CONFIG_STRIP_ASM_SYMS=y
CONFIG_UNUSED_SYMBOLS=y
CONFIG_DEBUG_FS=y
CONFIG_HEADERS_CHECK=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_SHIRQ=y
CONFIG_LOCKUP_DETECTOR=y
CONFIG_HARDLOCKUP_DETECTOR=y
# CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
# CONFIG_DETECT_HUNG_TASK is not set
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
CONFIG_TIMER_STATS=y
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_SLUB_STATS is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
CONFIG_BKL=y
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
CONFIG_SPARSE_RCU_POINTER=y
# CONFIG_LOCK_STAT is not set
CONFIG_DEBUG_SPINLOCK_SLEEP=y
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_VIRTUAL is not set
# CONFIG_DEBUG_WRITECOUNT is not set
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_LIST=y
# CONFIG_TEST_LIST_SORT is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_DEBUG_CREDENTIALS is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
CONFIG_BOOT_PRINTK_DELAY=y
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_RCU_CPU_STALL_DETECTOR is not set
# CONFIG_KPROBES_SANITY_TEST is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# CONFIG_LKDTM is not set
# CONFIG_CPU_NOTIFIER_ERROR_INJECT is not set
# CONFIG_FAULT_INJECTION is not set
CONFIG_LATENCYTOP=y
# CONFIG_SYSCTL_SYSCALL_CHECK is not set
# CONFIG_DEBUG_PAGEALLOC is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FTRACE_NMI_ENTER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_RING_BUFFER=y
CONFIG_FTRACE_NMI_ENTER=y
CONFIG_EVENT_TRACING=y
CONFIG_EVENT_POWER_TRACING_DEPRECATED=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_RING_BUFFER_ALLOW_SWAP=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
# CONFIG_IRQSOFF_TRACER is not set
CONFIG_SCHED_TRACER=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
CONFIG_STACK_TRACER=y
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_KPROBE_EVENT=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_FTRACE_MCOUNT_RECORD=y
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_MMIOTRACE is not set
CONFIG_RING_BUFFER_BENCHMARK=m
CONFIG_PROVIDE_OHCI1394_DMA_INIT=y
# CONFIG_FIREWIRE_OHCI_REMOTE_DMA is not set
CONFIG_BUILD_DOCSRC=y
CONFIG_DYNAMIC_DEBUG=y
# CONFIG_DMA_API_DEBUG is not set
CONFIG_ATOMIC64_SELFTEST=y
CONFIG_ASYNC_RAID6_TEST=m
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
CONFIG_KGDB_TESTS=y
# CONFIG_KGDB_TESTS_ON_BOOT is not set
CONFIG_KGDB_LOW_LEVEL_TRAP=y
CONFIG_KGDB_KDB=y
CONFIG_KDB_KEYBOARD=y
CONFIG_HAVE_ARCH_KMEMCHECK=y
CONFIG_TEST_KSTRTOX=m
CONFIG_STRICT_DEVMEM=y
# CONFIG_X86_VERBOSE_BOOTUP is not set
CONFIG_EARLY_PRINTK=y
CONFIG_EARLY_PRINTK_DBGP=y
CONFIG_DEBUG_STACKOVERFLOW=y
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PER_CPU_MAPS is not set
# CONFIG_X86_PTDUMP is not set
CONFIG_DEBUG_RODATA=y
CONFIG_DEBUG_RODATA_TEST=y
# CONFIG_DEBUG_SET_MODULE_RONX is not set
CONFIG_DEBUG_NX_TEST=m
# CONFIG_IOMMU_DEBUG is not set
# CONFIG_IOMMU_STRESS is not set
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_X86_DECODER_SELFTEST=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
CONFIG_DEBUG_BOOT_PARAMS=y
# CONFIG_CPA_DEBUG is not set
CONFIG_OPTIMIZE_INLINING=y
# CONFIG_DEBUG_STRICT_USER_COPY_CHECKS is not set
#
# Security options
#
CONFIG_KEYS=y
CONFIG_TRUSTED_KEYS=m
CONFIG_ENCRYPTED_KEYS=m
CONFIG_KEYS_DEBUG_PROC_KEYS=y
# CONFIG_SECURITY_DMESG_RESTRICT is not set
CONFIG_SECURITY=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_NETWORK_XFRM=y
# CONFIG_SECURITY_PATH is not set
CONFIG_INTEL_TXT=y
CONFIG_LSM_MMAP_MIN_ADDR=65536
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
# CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set
# CONFIG_SECURITY_SMACK is not set
# CONFIG_SECURITY_TOMOYO is not set
# CONFIG_SECURITY_APPARMOR is not set
CONFIG_IMA=y
CONFIG_IMA_MEASURE_PCR_IDX=10
CONFIG_IMA_AUDIT=y
CONFIG_IMA_LSM_RULES=y
CONFIG_DEFAULT_SECURITY_SELINUX=y
# CONFIG_DEFAULT_SECURITY_DAC is not set
CONFIG_DEFAULT_SECURITY="selinux"
CONFIG_XOR_BLOCKS=m
CONFIG_ASYNC_CORE=m
CONFIG_ASYNC_MEMCPY=m
CONFIG_ASYNC_XOR=m
CONFIG_ASYNC_PQ=m
CONFIG_ASYNC_RAID6_RECOV=m
CONFIG_ASYNC_TX_DISABLE_PQ_VAL_DMA=y
CONFIG_ASYNC_TX_DISABLE_XOR_VAL_DMA=y
CONFIG_CRYPTO=y
#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=y
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP=m
CONFIG_CRYPTO_PCOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
CONFIG_CRYPTO_GF128MUL=m
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_PCRYPT=m
CONFIG_CRYPTO_WORKQUEUE=y
CONFIG_CRYPTO_CRYPTD=y
CONFIG_CRYPTO_AUTHENC=m
CONFIG_CRYPTO_TEST=m
#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=m
CONFIG_CRYPTO_GCM=m
CONFIG_CRYPTO_SEQIV=y
#
# Block modes
#
CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_CTR=y
CONFIG_CRYPTO_CTS=m
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_PCBC=m
CONFIG_CRYPTO_XTS=m
CONFIG_CRYPTO_FPU=y
#
# Hash modes
#
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_VMAC=m
#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_CRC32C_INTEL=m
CONFIG_CRYPTO_GHASH=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD128=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_RMD256=m
CONFIG_CRYPTO_RMD320=m
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL=m
#
# Ciphers
#
CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_X86_64=y
CONFIG_CRYPTO_AES_NI_INTEL=y
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_CAMELLIA=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_SALSA20=m
CONFIG_CRYPTO_SALSA20_X86_64=m
CONFIG_CRYPTO_SEED=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_X86_64=m
#
# Compression
#
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_ZLIB=m
CONFIG_CRYPTO_LZO=m
#
# Random Number Generation
#
CONFIG_CRYPTO_ANSI_CPRNG=m
CONFIG_CRYPTO_USER_API=y
CONFIG_CRYPTO_USER_API_HASH=y
CONFIG_CRYPTO_USER_API_SKCIPHER=y
CONFIG_CRYPTO_HW=y
CONFIG_CRYPTO_DEV_PADLOCK=m
CONFIG_CRYPTO_DEV_PADLOCK_AES=m
CONFIG_CRYPTO_DEV_PADLOCK_SHA=m
CONFIG_CRYPTO_DEV_HIFN_795X=m
CONFIG_CRYPTO_DEV_HIFN_795X_RNG=y
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
CONFIG_KVM_MMU_AUDIT=y
CONFIG_VHOST_NET=m
CONFIG_VIRTIO=y
CONFIG_VIRTIO_RING=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_BALLOON=m
CONFIG_BINARY_PRINTF=y
#
# Library routines
#
CONFIG_RAID6_PQ=m
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_GENERIC_FIND_LAST_BIT=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
CONFIG_CRC7=m
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_GENERIC_ALLOCATOR=y
CONFIG_REED_SOLOMON=m
CONFIG_REED_SOLOMON_DEC16=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_BTREE=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CHECK_SIGNATURE=y
CONFIG_NLATTR=y
CONFIG_LRU_CACHE=m
CONFIG_AVERAGE=y
CONFIG_PCI_DEFAULT_USE_CRS=y
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-06 15:44 ` Mel Gorman
@ 2011-05-06 19:14 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-06 19:14 UTC (permalink / raw)
To: Mel Gorman
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Fri, 2011-05-06 at 16:44 +0100, Mel Gorman wrote:
> Colin and James: Did you happen to switch from SLAB to SLUB between
> 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I
> didn't see the problem. Am restarting the tests with SLUB.
Aargh ... I'm an idiot. I should have thought of SLUB immediately ...
it's been causing oopses since debian switched to it.
So I recompiled the 2.6.38.4 stable kernel with SLAB instead of SLUB and
the problem goes away ... at least from three untar runs on a loaded
box ... of course it could manifest a few ms after I send this email ...
There are material differences, as well: SLAB isn't taking my system
down to very low memory on the untar ... it's keeping about 0.5Gb listed
as free. SLUB took that to under 100kb, so it could just be that SLAB
isn't wandering as close to the cliff edge?
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-06 19:14 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-06 19:14 UTC (permalink / raw)
To: Mel Gorman
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Fri, 2011-05-06 at 16:44 +0100, Mel Gorman wrote:
> Colin and James: Did you happen to switch from SLAB to SLUB between
> 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I
> didn't see the problem. Am restarting the tests with SLUB.
Aargh ... I'm an idiot. I should have thought of SLUB immediately ...
it's been causing oopses since debian switched to it.
So I recompiled the 2.6.38.4 stable kernel with SLAB instead of SLUB and
the problem goes away ... at least from three untar runs on a loaded
box ... of course it could manifest a few ms after I send this email ...
There are material differences, as well: SLAB isn't taking my system
down to very low memory on the untar ... it's keeping about 0.5Gb listed
as free. SLUB took that to under 100kb, so it could just be that SLAB
isn't wandering as close to the cliff edge?
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-06 19:14 ` James Bottomley
@ 2011-05-06 19:37 ` Mel Gorman
-1 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-05-06 19:37 UTC (permalink / raw)
To: James Bottomley
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Fri, May 06, 2011 at 02:14:37PM -0500, James Bottomley wrote:
> On Fri, 2011-05-06 at 16:44 +0100, Mel Gorman wrote:
> > Colin and James: Did you happen to switch from SLAB to SLUB between
> > 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I
> > didn't see the problem. Am restarting the tests with SLUB.
>
> Aargh ... I'm an idiot. I should have thought of SLUB immediately ...
> it's been causing oopses since debian switched to it.
>
> So I recompiled the 2.6.38.4 stable kernel with SLAB instead of SLUB and
> the problem goes away ... at least from three untar runs on a loaded
> box ... of course it could manifest a few ms after I send this email ...
>
> There are material differences, as well: SLAB isn't taking my system
> down to very low memory on the untar ... it's keeping about 0.5Gb listed
> as free. SLUB took that to under 100kb, so it could just be that SLAB
> isn't wandering as close to the cliff edge?
>
A comparison of watch-highorder.pl with SLAB and SLUB may be
enlightening as well as testing SLUB altering allocate_slab() to read
alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY | __GFP_NO_KSWAPD) & ~__GFP_NOFAIL;
i.e. try adding the __GFP_NO_KSWAPD. My own tests are still in progress
but I'm still not seeing the problem. I'm installing Fedora on another
test machine at the moment to see if X and other applications have to be
running to pressure high-order allocations properly.
--
Mel Gorman
SUSE Labs
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-06 19:37 ` Mel Gorman
0 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-05-06 19:37 UTC (permalink / raw)
To: James Bottomley
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Fri, May 06, 2011 at 02:14:37PM -0500, James Bottomley wrote:
> On Fri, 2011-05-06 at 16:44 +0100, Mel Gorman wrote:
> > Colin and James: Did you happen to switch from SLAB to SLUB between
> > 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I
> > didn't see the problem. Am restarting the tests with SLUB.
>
> Aargh ... I'm an idiot. I should have thought of SLUB immediately ...
> it's been causing oopses since debian switched to it.
>
> So I recompiled the 2.6.38.4 stable kernel with SLAB instead of SLUB and
> the problem goes away ... at least from three untar runs on a loaded
> box ... of course it could manifest a few ms after I send this email ...
>
> There are material differences, as well: SLAB isn't taking my system
> down to very low memory on the untar ... it's keeping about 0.5Gb listed
> as free. SLUB took that to under 100kb, so it could just be that SLAB
> isn't wandering as close to the cliff edge?
>
A comparison of watch-highorder.pl with SLAB and SLUB may be
enlightening as well as testing SLUB altering allocate_slab() to read
alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY | __GFP_NO_KSWAPD) & ~__GFP_NOFAIL;
i.e. try adding the __GFP_NO_KSWAPD. My own tests are still in progress
but I'm still not seeing the problem. I'm installing Fedora on another
test machine at the moment to see if X and other applications have to be
running to pressure high-order allocations properly.
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-06 8:07 ` Mel Gorman
@ 2011-05-09 18:16 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-09 18:16 UTC (permalink / raw)
To: Mel Gorman
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Fri, 2011-05-06 at 09:07 +0100, Mel Gorman wrote:
> On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote:
> > 1. High-order allocations? You machine is using i915 and RPC, something
> > neither of my test machine uses. i915 is potentially a source for
> > high-order allocations. I'm attaching a perl script. Please run it as
> > ./watch-highorder.pl --output /tmp/highorders.txt
> > while you are running tar. When kswapd is running for about 30
> > seconds, interrupt it with ctrl+c twice in quick succession and
> > post /tmp/highorders.txt
> >
>
> Attached this time :/
Here's the output (loaded with tar, evolution and firefox). The top
trace is different this time because your perl script perturbs the
system quite a bit. This was with your slub allocation fix applied.
James
---
1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> sg_kmalloc+0x24/0x26 <ffffffff81230eb9>
=> __sg_alloc_table+0x63/0x11c <ffffffff81230dbf>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd74b>
=> ra_submit+0x21/0x25 <ffffffff810dda8d>
=> ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc5a>
=> page_cache_sync_readahead+0x3d/0x40 <ffffffff810ddd49>
=> generic_file_aio_read+0x27d/0x5e0 <ffffffff810d64d0>
177 instances order=2 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> radix_tree_preload+0x31/0x81 <ffffffff81229399>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5>
1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107001>
=> do_wp_page+0x348/0x560 <ffffffff810ef711>
=> handle_pte_fault+0x73b/0x7a5 <ffffffff810f13b7>
=> handle_mm_fault+0x1bb/0x1ce <ffffffff810f1798>
=> do_page_fault+0x358/0x37a <ffffffff8147416e>
=> page_fault+0x25/0x30 <ffffffff81471415>
46 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> prepare_creds+0x26/0xae <ffffffff81074d4b>
=> sys_faccessat+0x37/0x162 <ffffffff8111d255>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e199>
=> __alloc_skb+0x40/0x133 <ffffffff813b5df7>
=> __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f4a>
1 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> alloc_inode+0x30/0x78 <ffffffff811317f8>
=> new_inode+0x1b/0x4b <ffffffff8113185b>
16 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107001>
=> do_huge_pmd_anonymous_page+0xbf/0x261 <ffffffff81115b6a>
=> handle_mm_fault+0x113/0x1ce <ffffffff810f16f0>
=> do_page_fault+0x358/0x37a <ffffffff8147416e>
=> page_fault+0x25/0x30 <ffffffff81471415>
5 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e2c>
=> __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f4a>
252 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> radix_tree_preload+0x31/0x81 <ffffffff81229399>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> skb_clone+0x50/0x70 <ffffffff813b5d97>
=> packet_rcv+0x101/0x2b2 <ffffffff81441e80>
1 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> alloc_inode+0x30/0x78 <ffffffff811317f8>
=> iget_locked+0x61/0xdd <ffffffff8113239a>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> sg_kmalloc+0x24/0x26 <ffffffff81230eb9>
=> __sg_alloc_table+0x63/0x11c <ffffffff81230dbf>
1 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> copy_process+0x66e/0x10c5 <ffffffff81053961>
=> do_fork+0x104/0x286 <ffffffff810544f7>
1 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> kernel_thread+0x75/0x77 <ffffffff81010bb6>
=> wait_for_helper+0x6b/0xa3 <ffffffff81068fca>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
133 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107001>
=> khugepaged+0x543/0xf2d <ffffffff81113cc1>
=> kthread+0x84/0x8c <ffffffff8106f2df>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
593 instances order=3 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> ext4_alloc_inode+0x1a/0x111 <ffffffff8119f498>
=> alloc_inode+0x1d/0x78 <ffffffff811317e5>
2 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2>
=> i915_gem_execbuffer2+0x4f/0x12e [i915] <ffffffffa0080f81>
14 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> d_alloc+0x26/0x18d <ffffffff8112e4c5>
=> shmem_file_setup+0xb6/0x160 <ffffffff810e70ad>
38 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> mmap_region+0x1b7/0x446 <ffffffff810f6555>
=> do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6a7c>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> idr_pre_get+0x2d/0x6f <ffffffff812266c4>
=> drm_gem_handle_create+0x2f/0x82 [drm] <ffffffffa0024f75>
400 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> sys_clone+0x28/0x2a <ffffffff8101150e>
=> stub_clone+0x13/0x20 <ffffffff81009ea3>
9 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e199>
=> __alloc_skb+0x40/0x133 <ffffffff813b5df7>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107001>
=> handle_pte_fault+0x16f/0x7a5 <ffffffff810f0deb>
=> handle_mm_fault+0x1bb/0x1ce <ffffffff810f1798>
=> do_page_fault+0x358/0x37a <ffffffff8147416e>
=> page_fault+0x25/0x30 <ffffffff81471415>
=> unix_stream_recvmsg+0x40f/0x536 <ffffffff8143dbe0>
=> sock_aio_read.part.7+0x10d/0x121 <ffffffff813afa54>
2 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> kernel_thread+0x75/0x77 <ffffffff81010bb6>
=> __call_usermodehelper+0x43/0x76 <ffffffff81069132>
=> process_one_work+0x186/0x298 <ffffffff8106b3ca>
14 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e2c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a>
781 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> kmalloc_large_node+0x56/0x95 <ffffffff8146a55d>
=> __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff08>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e2c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25c2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d217>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af778>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> __pollwait+0x5e/0xcc <ffffffff8112c9e7>
=> sock_poll_wait+0x18/0x1d <ffffffff8143cc50>
=> unix_poll+0x1c/0x9a <ffffffff8143cc71>
=> sock_poll+0x1a/0x1c <ffffffff813ad41a>
=> do_sys_poll+0x1fa/0x386 <ffffffff8112d75d>
48 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> __split_vma+0x6c/0x21b <ffffffff810f5099>
=> split_vma+0x20/0x22 <ffffffff810f59a6>
3 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> sk_prot_alloc+0x37/0x13a <ffffffff813b27d4>
=> sk_alloc+0x2c/0x8e <ffffffff813b2935>
24 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> proc_alloc_inode+0x20/0x91 <ffffffff8116829c>
=> alloc_inode+0x1d/0x78 <ffffffff811317e5>
1 instances order=1 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> ext4_mb_new_blocks+0x118/0x3c2 <ffffffff811b73a7>
=> ext4_ext_map_blocks+0x192b/0x1b5b <ffffffff811b0a5d>
5 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> sock_alloc_inode+0x1d/0xaa <ffffffff813ad605>
=> alloc_inode+0x1d/0x78 <ffffffff811317e5>
3 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> dst_alloc+0x48/0xaa <ffffffff813c6652>
=> __ip_route_output_key+0x561/0x764 <ffffffff813ee4f3>
3 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> __pollwait+0x5e/0xcc <ffffffff8112c9e7>
=> sock_poll_wait+0x18/0x1d <ffffffff8143cc50>
=> unix_poll+0x1c/0x9a <ffffffff8143cc71>
=> sock_poll+0x1a/0x1c <ffffffff813ad41a>
=> do_select+0x2fb/0x4f5 <ffffffff8112cf13>
4 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> __split_vma+0x6c/0x21b <ffffffff810f5099>
=> do_munmap+0x194/0x30b <ffffffff810f57ad>
229 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> dup_mm+0x1f8/0x486 <ffffffff81053039>
=> copy_process+0x917/0x10c5 <ffffffff81053c0a>
1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> bprm_mm_init+0x70/0x1a0 <ffffffff811239aa>
=> do_execve+0xd6/0x277 <ffffffff81123dd4>
2 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e2c>
=> find_skb+0x3a/0x82 <ffffffff813d1996>
2 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> __split_vma+0x6c/0x21b <ffffffff810f5099>
=> do_munmap+0x15f/0x30b <ffffffff810f5778>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107001>
=> handle_pte_fault+0x16f/0x7a5 <ffffffff810f0deb>
=> handle_mm_fault+0x1bb/0x1ce <ffffffff810f1798>
=> do_page_fault+0x358/0x37a <ffffffff8147416e>
=> page_fault+0x25/0x30 <ffffffff81471415>
501 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2d1>
=> do_filp_open+0xe7/0x60a <ffffffff81129bcf>
18 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d61>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144df4>
1370 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> d_alloc+0x26/0x18d <ffffffff8112e4c5>
=> d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d0e>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> kzalloc.constprop.1+0x13/0x15 [i915] <ffffffffa007fbfd>
=> i915_gem_do_execbuffer+0x306/0x1116 [i915] <ffffffffa007ff05>
1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> taskstats_exit+0x5b/0x2e3 <ffffffff810b0810>
=> do_exit+0x22a/0x6ef <ffffffff8105841f>
17 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> d_alloc+0x26/0x18d <ffffffff8112e4c5>
=> proc_fill_cache+0x82/0x135 <ffffffff8116903b>
99 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> inet_twsk_alloc+0x31/0x11e <ffffffff813f8a16>
=> tcp_time_wait+0xb9/0x29a <ffffffff8140e24f>
26 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5943>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff81196376>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d5408>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d617d>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6205>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2>
=> i915_gem_do_execbuffer+0x64a/0x1116 [i915] <ffffffffa0080249>
6 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> kernel_thread+0x75/0x77 <ffffffff81010bb6>
=> kthreadd+0xe7/0x124 <ffffffff8106f61f>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
2 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e199>
=> __alloc_skb+0x40/0x133 <ffffffff813b5df7>
=> find_skb+0x3a/0x82 <ffffffff813d1996>
131 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3047>
=> scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30cd>
6 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> d_alloc+0x26/0x18d <ffffffff8112e4c5>
=> d_alloc_pseudo+0x18/0x46 <ffffffff8112e6a7>
6 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d61>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144df4>
11 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> shmem_alloc_inode+0x1a/0x2f <ffffffff810e58a0>
=> alloc_inode+0x1d/0x78 <ffffffff811317e5>
4 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2>
=> i915_gem_do_execbuffer+0x64a/0x1116 [i915] <ffffffffa0080249>
680 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
8 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> copy_process+0xc6/0x10c5 <ffffffff810533b9>
=> do_fork+0x104/0x286 <ffffffff810544f7>
4 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107001>
=> do_huge_pmd_wp_page+0x15a/0x637 <ffffffff81114a01>
=> handle_mm_fault+0x169/0x1ce <ffffffff810f1746>
=> do_page_fault+0x358/0x37a <ffffffff8147416e>
=> page_fault+0x25/0x30 <ffffffff81471415>
193 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
140358 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
3 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e2c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a>
17 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd74b>
=> ra_submit+0x21/0x25 <ffffffff810dda8d>
=> ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc5a>
=> page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddce4>
=> generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6510>
9 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> kzalloc.constprop.15+0x13/0x15 [i915] <ffffffffa007a956>
=> i915_gem_alloc_object+0x27/0x111 [i915] <ffffffffa007e4b3>
14 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2d1>
=> alloc_file+0x1e/0xbf <ffffffff8111f3b6>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_COMP|GFP_ZERO
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> kmalloc_order_trace+0x2c/0x5e <ffffffff8110c718>
=> __kmalloc+0x37/0x10d <ffffffff8110df41>
=> kzalloc.constprop.19+0xe/0x10 <ffffffff810b8d8c>
=> tracing_open_pipe+0x40/0x147 <ffffffff810b907a>
=> __dentry_open+0x161/0x283 <ffffffff8111c95c>
High-order normal allocations: 145450
High-order atomic allocations: 927
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-09 18:16 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-09 18:16 UTC (permalink / raw)
To: Mel Gorman
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Fri, 2011-05-06 at 09:07 +0100, Mel Gorman wrote:
> On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote:
> > 1. High-order allocations? You machine is using i915 and RPC, something
> > neither of my test machine uses. i915 is potentially a source for
> > high-order allocations. I'm attaching a perl script. Please run it as
> > ./watch-highorder.pl --output /tmp/highorders.txt
> > while you are running tar. When kswapd is running for about 30
> > seconds, interrupt it with ctrl+c twice in quick succession and
> > post /tmp/highorders.txt
> >
>
> Attached this time :/
Here's the output (loaded with tar, evolution and firefox). The top
trace is different this time because your perl script perturbs the
system quite a bit. This was with your slub allocation fix applied.
James
---
1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> sg_kmalloc+0x24/0x26 <ffffffff81230eb9>
=> __sg_alloc_table+0x63/0x11c <ffffffff81230dbf>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd74b>
=> ra_submit+0x21/0x25 <ffffffff810dda8d>
=> ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc5a>
=> page_cache_sync_readahead+0x3d/0x40 <ffffffff810ddd49>
=> generic_file_aio_read+0x27d/0x5e0 <ffffffff810d64d0>
177 instances order=2 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> radix_tree_preload+0x31/0x81 <ffffffff81229399>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5>
1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107001>
=> do_wp_page+0x348/0x560 <ffffffff810ef711>
=> handle_pte_fault+0x73b/0x7a5 <ffffffff810f13b7>
=> handle_mm_fault+0x1bb/0x1ce <ffffffff810f1798>
=> do_page_fault+0x358/0x37a <ffffffff8147416e>
=> page_fault+0x25/0x30 <ffffffff81471415>
46 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> prepare_creds+0x26/0xae <ffffffff81074d4b>
=> sys_faccessat+0x37/0x162 <ffffffff8111d255>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e199>
=> __alloc_skb+0x40/0x133 <ffffffff813b5df7>
=> __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f4a>
1 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> alloc_inode+0x30/0x78 <ffffffff811317f8>
=> new_inode+0x1b/0x4b <ffffffff8113185b>
16 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107001>
=> do_huge_pmd_anonymous_page+0xbf/0x261 <ffffffff81115b6a>
=> handle_mm_fault+0x113/0x1ce <ffffffff810f16f0>
=> do_page_fault+0x358/0x37a <ffffffff8147416e>
=> page_fault+0x25/0x30 <ffffffff81471415>
5 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e2c>
=> __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f4a>
252 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> radix_tree_preload+0x31/0x81 <ffffffff81229399>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> skb_clone+0x50/0x70 <ffffffff813b5d97>
=> packet_rcv+0x101/0x2b2 <ffffffff81441e80>
1 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> alloc_inode+0x30/0x78 <ffffffff811317f8>
=> iget_locked+0x61/0xdd <ffffffff8113239a>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> sg_kmalloc+0x24/0x26 <ffffffff81230eb9>
=> __sg_alloc_table+0x63/0x11c <ffffffff81230dbf>
1 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> copy_process+0x66e/0x10c5 <ffffffff81053961>
=> do_fork+0x104/0x286 <ffffffff810544f7>
1 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> kernel_thread+0x75/0x77 <ffffffff81010bb6>
=> wait_for_helper+0x6b/0xa3 <ffffffff81068fca>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
133 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107001>
=> khugepaged+0x543/0xf2d <ffffffff81113cc1>
=> kthread+0x84/0x8c <ffffffff8106f2df>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
593 instances order=3 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> ext4_alloc_inode+0x1a/0x111 <ffffffff8119f498>
=> alloc_inode+0x1d/0x78 <ffffffff811317e5>
2 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2>
=> i915_gem_execbuffer2+0x4f/0x12e [i915] <ffffffffa0080f81>
14 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> d_alloc+0x26/0x18d <ffffffff8112e4c5>
=> shmem_file_setup+0xb6/0x160 <ffffffff810e70ad>
38 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> mmap_region+0x1b7/0x446 <ffffffff810f6555>
=> do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6a7c>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> idr_pre_get+0x2d/0x6f <ffffffff812266c4>
=> drm_gem_handle_create+0x2f/0x82 [drm] <ffffffffa0024f75>
400 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> sys_clone+0x28/0x2a <ffffffff8101150e>
=> stub_clone+0x13/0x20 <ffffffff81009ea3>
9 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e199>
=> __alloc_skb+0x40/0x133 <ffffffff813b5df7>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107001>
=> handle_pte_fault+0x16f/0x7a5 <ffffffff810f0deb>
=> handle_mm_fault+0x1bb/0x1ce <ffffffff810f1798>
=> do_page_fault+0x358/0x37a <ffffffff8147416e>
=> page_fault+0x25/0x30 <ffffffff81471415>
=> unix_stream_recvmsg+0x40f/0x536 <ffffffff8143dbe0>
=> sock_aio_read.part.7+0x10d/0x121 <ffffffff813afa54>
2 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> kernel_thread+0x75/0x77 <ffffffff81010bb6>
=> __call_usermodehelper+0x43/0x76 <ffffffff81069132>
=> process_one_work+0x186/0x298 <ffffffff8106b3ca>
14 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e2c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a>
781 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> kmalloc_large_node+0x56/0x95 <ffffffff8146a55d>
=> __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff08>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e2c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25c2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d217>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af778>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> __pollwait+0x5e/0xcc <ffffffff8112c9e7>
=> sock_poll_wait+0x18/0x1d <ffffffff8143cc50>
=> unix_poll+0x1c/0x9a <ffffffff8143cc71>
=> sock_poll+0x1a/0x1c <ffffffff813ad41a>
=> do_sys_poll+0x1fa/0x386 <ffffffff8112d75d>
48 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> __split_vma+0x6c/0x21b <ffffffff810f5099>
=> split_vma+0x20/0x22 <ffffffff810f59a6>
3 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> sk_prot_alloc+0x37/0x13a <ffffffff813b27d4>
=> sk_alloc+0x2c/0x8e <ffffffff813b2935>
24 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> proc_alloc_inode+0x20/0x91 <ffffffff8116829c>
=> alloc_inode+0x1d/0x78 <ffffffff811317e5>
1 instances order=1 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> ext4_mb_new_blocks+0x118/0x3c2 <ffffffff811b73a7>
=> ext4_ext_map_blocks+0x192b/0x1b5b <ffffffff811b0a5d>
5 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> sock_alloc_inode+0x1d/0xaa <ffffffff813ad605>
=> alloc_inode+0x1d/0x78 <ffffffff811317e5>
3 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> dst_alloc+0x48/0xaa <ffffffff813c6652>
=> __ip_route_output_key+0x561/0x764 <ffffffff813ee4f3>
3 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> __pollwait+0x5e/0xcc <ffffffff8112c9e7>
=> sock_poll_wait+0x18/0x1d <ffffffff8143cc50>
=> unix_poll+0x1c/0x9a <ffffffff8143cc71>
=> sock_poll+0x1a/0x1c <ffffffff813ad41a>
=> do_select+0x2fb/0x4f5 <ffffffff8112cf13>
4 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> __split_vma+0x6c/0x21b <ffffffff810f5099>
=> do_munmap+0x194/0x30b <ffffffff810f57ad>
229 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> dup_mm+0x1f8/0x486 <ffffffff81053039>
=> copy_process+0x917/0x10c5 <ffffffff81053c0a>
1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> bprm_mm_init+0x70/0x1a0 <ffffffff811239aa>
=> do_execve+0xd6/0x277 <ffffffff81123dd4>
2 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e2c>
=> find_skb+0x3a/0x82 <ffffffff813d1996>
2 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> __split_vma+0x6c/0x21b <ffffffff810f5099>
=> do_munmap+0x15f/0x30b <ffffffff810f5778>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107001>
=> handle_pte_fault+0x16f/0x7a5 <ffffffff810f0deb>
=> handle_mm_fault+0x1bb/0x1ce <ffffffff810f1798>
=> do_page_fault+0x358/0x37a <ffffffff8147416e>
=> page_fault+0x25/0x30 <ffffffff81471415>
501 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2d1>
=> do_filp_open+0xe7/0x60a <ffffffff81129bcf>
18 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d61>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144df4>
1370 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> d_alloc+0x26/0x18d <ffffffff8112e4c5>
=> d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d0e>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> kzalloc.constprop.1+0x13/0x15 [i915] <ffffffffa007fbfd>
=> i915_gem_do_execbuffer+0x306/0x1116 [i915] <ffffffffa007ff05>
1 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> taskstats_exit+0x5b/0x2e3 <ffffffff810b0810>
=> do_exit+0x22a/0x6ef <ffffffff8105841f>
17 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> d_alloc+0x26/0x18d <ffffffff8112e4c5>
=> proc_fill_cache+0x82/0x135 <ffffffff8116903b>
99 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> inet_twsk_alloc+0x31/0x11e <ffffffff813f8a16>
=> tcp_time_wait+0xb9/0x29a <ffffffff8140e24f>
26 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5943>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff81196376>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d5408>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d617d>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6205>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2>
=> i915_gem_do_execbuffer+0x64a/0x1116 [i915] <ffffffffa0080249>
6 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> kernel_thread+0x75/0x77 <ffffffff81010bb6>
=> kthreadd+0xe7/0x124 <ffffffff8106f61f>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
2 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e199>
=> __alloc_skb+0x40/0x133 <ffffffff813b5df7>
=> find_skb+0x3a/0x82 <ffffffff813d1996>
131 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3047>
=> scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30cd>
6 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> d_alloc+0x26/0x18d <ffffffff8112e4c5>
=> d_alloc_pseudo+0x18/0x46 <ffffffff8112e6a7>
6 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d61>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144df4>
11 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> shmem_alloc_inode+0x1a/0x2f <ffffffff810e58a0>
=> alloc_inode+0x1d/0x78 <ffffffff811317e5>
4 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2>
=> i915_gem_do_execbuffer+0x64a/0x1116 [i915] <ffffffffa0080249>
680 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
8 instances order=3 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> copy_process+0xc6/0x10c5 <ffffffff810533b9>
=> do_fork+0x104/0x286 <ffffffff810544f7>
4 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107001>
=> do_huge_pmd_wp_page+0x15a/0x637 <ffffffff81114a01>
=> handle_mm_fault+0x169/0x1ce <ffffffff810f1746>
=> do_page_fault+0x358/0x37a <ffffffff8147416e>
=> page_fault+0x25/0x30 <ffffffff81471415>
193 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
140358 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
3 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffa6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e2c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a>
17 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd74b>
=> ra_submit+0x21/0x25 <ffffffff810dda8d>
=> ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc5a>
=> page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddce4>
=> generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6510>
9 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfbb>
=> kzalloc.constprop.15+0x13/0x15 [i915] <ffffffffa007a956>
=> i915_gem_alloc_object+0x27/0x111 [i915] <ffffffffa007e4b3>
14 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
=> new_slab+0x50/0x199 <ffffffff8110dc24>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2d1>
=> alloc_file+0x1e/0xbf <ffffffff8111f3b6>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_COMP|GFP_ZERO
=> __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> kmalloc_order_trace+0x2c/0x5e <ffffffff8110c718>
=> __kmalloc+0x37/0x10d <ffffffff8110df41>
=> kzalloc.constprop.19+0xe/0x10 <ffffffff810b8d8c>
=> tracing_open_pipe+0x40/0x147 <ffffffff810b907a>
=> __dentry_open+0x161/0x283 <ffffffff8111c95c>
High-order normal allocations: 145450
High-order atomic allocations: 927
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-06 15:44 ` Mel Gorman
(?)
@ 2011-05-10 5:37 ` Colin Ian King
-1 siblings, 0 replies; 138+ messages in thread
From: Colin Ian King @ 2011-05-10 5:37 UTC (permalink / raw)
To: Mel Gorman
Cc: James Bottomley, Mel Gorman, Jan Kara, Chris Mason,
linux-fsdevel, linux-mm, linux-kernel, linux-ext4
On Fri, 2011-05-06 at 16:44 +0100, Mel Gorman wrote:
> On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote:
> >
> > 1. High-order allocations? You machine is using i915 and RPC, something
> > neither of my test machine uses. i915 is potentially a source for
> > high-order allocations. I'm attaching a perl script. Please run it as
> > ./watch-highorder.pl --output /tmp/highorders.txt
> > while you are running tar. When kswapd is running for about 30
> > seconds, interrupt it with ctrl+c twice in quick succession and
> > post /tmp/highorders.txt
> >
>
> Colin send me this information for his test case at least and I see
>
> 11932 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC
> => alloc_pages_current+0xa5/0x110 <ffffffff81149ef5>
> => new_slab+0x1f5/0x290 <ffffffff81153645>
> => __slab_alloc+0x262/0x390 <ffffffff81155192>
> => kmem_cache_alloc+0x115/0x120 <ffffffff81155ab5>
> => mempool_alloc_slab+0x15/0x20 <ffffffff8110e705>
> => mempool_alloc+0x59/0x140 <ffffffff8110ea49>
> => bio_alloc_bioset+0x3e/0xf0 <ffffffff811976ae>
> => bio_alloc+0x15/0x30 <ffffffff81197805>
>
> Colin and James: Did you happen to switch from SLAB to SLUB between
> 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I
> didn't see the problem. Am restarting the tests with SLUB.
So I tested with SLAB instead of SLUB and I reliably ran my copy test
for 4+ hours with several hundred iterations of the test. (Apologies
for taking time to respond, but I was travelling).
>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-10 5:37 ` Colin Ian King
0 siblings, 0 replies; 138+ messages in thread
From: Colin Ian King @ 2011-05-10 5:37 UTC (permalink / raw)
To: Mel Gorman
Cc: James Bottomley, Mel Gorman, Jan Kara, Chris Mason,
linux-fsdevel, linux-mm, linux-kernel, linux-ext4
On Fri, 2011-05-06 at 16:44 +0100, Mel Gorman wrote:
> On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote:
> >
> > 1. High-order allocations? You machine is using i915 and RPC, something
> > neither of my test machine uses. i915 is potentially a source for
> > high-order allocations. I'm attaching a perl script. Please run it as
> > ./watch-highorder.pl --output /tmp/highorders.txt
> > while you are running tar. When kswapd is running for about 30
> > seconds, interrupt it with ctrl+c twice in quick succession and
> > post /tmp/highorders.txt
> >
>
> Colin send me this information for his test case at least and I see
>
> 11932 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC
> => alloc_pages_current+0xa5/0x110 <ffffffff81149ef5>
> => new_slab+0x1f5/0x290 <ffffffff81153645>
> => __slab_alloc+0x262/0x390 <ffffffff81155192>
> => kmem_cache_alloc+0x115/0x120 <ffffffff81155ab5>
> => mempool_alloc_slab+0x15/0x20 <ffffffff8110e705>
> => mempool_alloc+0x59/0x140 <ffffffff8110ea49>
> => bio_alloc_bioset+0x3e/0xf0 <ffffffff811976ae>
> => bio_alloc+0x15/0x30 <ffffffff81197805>
>
> Colin and James: Did you happen to switch from SLAB to SLUB between
> 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I
> didn't see the problem. Am restarting the tests with SLUB.
So I tested with SLAB instead of SLUB and I reliably ran my copy test
for 4+ hours with several hundred iterations of the test. (Apologies
for taking time to respond, but I was travelling).
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-10 5:37 ` Colin Ian King
0 siblings, 0 replies; 138+ messages in thread
From: Colin Ian King @ 2011-05-10 5:37 UTC (permalink / raw)
To: Mel Gorman
Cc: James Bottomley, Mel Gorman, Jan Kara, Chris Mason,
linux-fsdevel, linux-mm, linux-kernel, linux-ext4
On Fri, 2011-05-06 at 16:44 +0100, Mel Gorman wrote:
> On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote:
> >
> > 1. High-order allocations? You machine is using i915 and RPC, something
> > neither of my test machine uses. i915 is potentially a source for
> > high-order allocations. I'm attaching a perl script. Please run it as
> > ./watch-highorder.pl --output /tmp/highorders.txt
> > while you are running tar. When kswapd is running for about 30
> > seconds, interrupt it with ctrl+c twice in quick succession and
> > post /tmp/highorders.txt
> >
>
> Colin send me this information for his test case at least and I see
>
> 11932 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC
> => alloc_pages_current+0xa5/0x110 <ffffffff81149ef5>
> => new_slab+0x1f5/0x290 <ffffffff81153645>
> => __slab_alloc+0x262/0x390 <ffffffff81155192>
> => kmem_cache_alloc+0x115/0x120 <ffffffff81155ab5>
> => mempool_alloc_slab+0x15/0x20 <ffffffff8110e705>
> => mempool_alloc+0x59/0x140 <ffffffff8110ea49>
> => bio_alloc_bioset+0x3e/0xf0 <ffffffff811976ae>
> => bio_alloc+0x15/0x30 <ffffffff81197805>
>
> Colin and James: Did you happen to switch from SLAB to SLUB between
> 2.6.37 and 2.6.38? My own tests were against SLAB which might be why I
> didn't see the problem. Am restarting the tests with SLUB.
So I tested with SLAB instead of SLUB and I reliably ran my copy test
for 4+ hours with several hundred iterations of the test. (Apologies
for taking time to respond, but I was travelling).
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-09 18:16 ` James Bottomley
@ 2011-05-10 10:21 ` Mel Gorman
-1 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-05-10 10:21 UTC (permalink / raw)
To: James Bottomley
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Mon, May 09, 2011 at 01:16:20PM -0500, James Bottomley wrote:
> On Fri, 2011-05-06 at 09:07 +0100, Mel Gorman wrote:
> > On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote:
> > > 1. High-order allocations? You machine is using i915 and RPC, something
> > > neither of my test machine uses. i915 is potentially a source for
> > > high-order allocations. I'm attaching a perl script. Please run it as
> > > ./watch-highorder.pl --output /tmp/highorders.txt
> > > while you are running tar. When kswapd is running for about 30
> > > seconds, interrupt it with ctrl+c twice in quick succession and
> > > post /tmp/highorders.txt
> > >
> >
> > Attached this time :/
>
> Here's the output (loaded with tar, evolution and firefox). The top
> trace is different this time because your perl script perturbs the
> system quite a bit. This was with your slub allocation fix applied.
>
I note that certain flags like __GFP_NO_KSWAPD are not recognised by
tracing which might explain why they are missing from the script output.
I regret the script perturbs the system quite a bit. It's possible it
can be made better by filtering events but it's not high on the list of
things to do.
How does the output compare without the fix? I can't find a similar
report in my inbox.
Does the fix help the system when the perl script is not running?
> 177 instances order=2 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => radix_tree_preload+0x31/0x81 <ffffffff81229399>
> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5>
>
Ouch.
> 46 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => prepare_creds+0x26/0xae <ffffffff81074d4b>
> => sys_faccessat+0x37/0x162 <ffffffff8111d255>
>
Less ouch, but still.
> 252 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => radix_tree_preload+0x31/0x81 <ffffffff81229399>
> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5>
>
Ouch again.
> 593 instances order=3 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => ext4_alloc_inode+0x1a/0x111 <ffffffff8119f498>
> => alloc_inode+0x1d/0x78 <ffffffff811317e5>
>
Again, filesystem-related calls are hitting high-order paths quite a
bit.
> 781 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => kmalloc_large_node+0x56/0x95 <ffffffff8146a55d>
> => __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff08>
> => __alloc_skb+0x75/0x133 <ffffffff813b5e2c>
> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a>
> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25c2>
> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d217>
> => __sock_sendmsg+0x69/0x76 <ffffffff813af778>
>
A number of network paths are also being hit although this is the worst.
> 501 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => get_empty_filp+0x7a/0x141 <ffffffff8111f2d1>
> => do_filp_open+0xe7/0x60a <ffffffff81129bcf>
>
More filesystem impairment.
> 1370 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => d_alloc+0x26/0x18d <ffffffff8112e4c5>
> => d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d0e>
>
*cries*
> 140358 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
> => mempool_alloc+0x68/0x116 <ffffffff810d70f6>
Wonder which pool this is!
It goes on. A number of filesystem and network paths are being hit
with high-order allocs. i915 was a red herring, it's present but not
in massive numbers. The filesystem, network and mempool allocations
are likely to be kicking kswapd awake frequently and hurting overall
system performance as a result.
I really would like to hear if the fix makes a big difference or
if we need to consider forcing SLUB high-order allocations bailing
at the first sign of trouble (e.g. by masking out __GFP_WAIT in
allocate_slab). Even with the fix applied, kswapd might be waking up
less but processes will still be getting stalled in direct compaction
and direct reclaim so it would still be jittery.
> High-order normal allocations: 145450
> High-order atomic allocations: 927
>
I bet a shiny penny that the high-order allocations for SLAB are lower
than this
--
Mel Gorman
SUSE Labs
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-10 10:21 ` Mel Gorman
0 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-05-10 10:21 UTC (permalink / raw)
To: James Bottomley
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Mon, May 09, 2011 at 01:16:20PM -0500, James Bottomley wrote:
> On Fri, 2011-05-06 at 09:07 +0100, Mel Gorman wrote:
> > On Fri, May 06, 2011 at 08:42:24AM +0100, Mel Gorman wrote:
> > > 1. High-order allocations? You machine is using i915 and RPC, something
> > > neither of my test machine uses. i915 is potentially a source for
> > > high-order allocations. I'm attaching a perl script. Please run it as
> > > ./watch-highorder.pl --output /tmp/highorders.txt
> > > while you are running tar. When kswapd is running for about 30
> > > seconds, interrupt it with ctrl+c twice in quick succession and
> > > post /tmp/highorders.txt
> > >
> >
> > Attached this time :/
>
> Here's the output (loaded with tar, evolution and firefox). The top
> trace is different this time because your perl script perturbs the
> system quite a bit. This was with your slub allocation fix applied.
>
I note that certain flags like __GFP_NO_KSWAPD are not recognised by
tracing which might explain why they are missing from the script output.
I regret the script perturbs the system quite a bit. It's possible it
can be made better by filtering events but it's not high on the list of
things to do.
How does the output compare without the fix? I can't find a similar
report in my inbox.
Does the fix help the system when the perl script is not running?
> 177 instances order=2 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => radix_tree_preload+0x31/0x81 <ffffffff81229399>
> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5>
>
Ouch.
> 46 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => prepare_creds+0x26/0xae <ffffffff81074d4b>
> => sys_faccessat+0x37/0x162 <ffffffff8111d255>
>
Less ouch, but still.
> 252 instances order=2 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => radix_tree_preload+0x31/0x81 <ffffffff81229399>
> => add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5>
>
Ouch again.
> 593 instances order=3 normal gfp_flags=GFP_NOFS|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => ext4_alloc_inode+0x1a/0x111 <ffffffff8119f498>
> => alloc_inode+0x1d/0x78 <ffffffff811317e5>
>
Again, filesystem-related calls are hitting high-order paths quite a
bit.
> 781 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => kmalloc_large_node+0x56/0x95 <ffffffff8146a55d>
> => __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff08>
> => __alloc_skb+0x75/0x133 <ffffffff813b5e2c>
> => sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b238a>
> => sock_alloc_send_skb+0x15/0x17 <ffffffff813b25c2>
> => unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d217>
> => __sock_sendmsg+0x69/0x76 <ffffffff813af778>
>
A number of network paths are also being hit although this is the worst.
> 501 instances order=1 normal gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => get_empty_filp+0x7a/0x141 <ffffffff8111f2d1>
> => do_filp_open+0xe7/0x60a <ffffffff81129bcf>
>
More filesystem impairment.
> 1370 instances order=1 normal gfp_flags=GFP_TEMPORARY|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => d_alloc+0x26/0x18d <ffffffff8112e4c5>
> => d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d0e>
>
*cries*
> 140358 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
> => __alloc_pages_nodemask+0x737/0x772 <ffffffff810dc0bd>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105435>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5da>
> => new_slab+0x50/0x199 <ffffffff8110dc24>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab66>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e42c>
> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
> => mempool_alloc+0x68/0x116 <ffffffff810d70f6>
Wonder which pool this is!
It goes on. A number of filesystem and network paths are being hit
with high-order allocs. i915 was a red herring, it's present but not
in massive numbers. The filesystem, network and mempool allocations
are likely to be kicking kswapd awake frequently and hurting overall
system performance as a result.
I really would like to hear if the fix makes a big difference or
if we need to consider forcing SLUB high-order allocations bailing
at the first sign of trouble (e.g. by masking out __GFP_WAIT in
allocate_slab). Even with the fix applied, kswapd might be waking up
less but processes will still be getting stalled in direct compaction
and direct reclaim so it would still be jittery.
> High-order normal allocations: 145450
> High-order atomic allocations: 927
>
I bet a shiny penny that the high-order allocations for SLAB are lower
than this
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-10 10:21 ` Mel Gorman
@ 2011-05-10 10:33 ` Pekka Enberg
-1 siblings, 0 replies; 138+ messages in thread
From: Pekka Enberg @ 2011-05-10 10:33 UTC (permalink / raw)
To: Mel Gorman
Cc: James Bottomley, Mel Gorman, Jan Kara, colin.king, Chris Mason,
linux-fsdevel, linux-mm, linux-kernel, linux-ext4,
Christoph Lameter
On Tue, May 10, 2011 at 1:21 PM, Mel Gorman <mgorman@novell.com> wrote:
> It goes on. A number of filesystem and network paths are being hit
> with high-order allocs. i915 was a red herring, it's present but not
> in massive numbers. The filesystem, network and mempool allocations
> are likely to be kicking kswapd awake frequently and hurting overall
> system performance as a result.
>
> I really would like to hear if the fix makes a big difference or
> if we need to consider forcing SLUB high-order allocations bailing
> at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> allocate_slab). Even with the fix applied, kswapd might be waking up
> less but processes will still be getting stalled in direct compaction
> and direct reclaim so it would still be jittery.
Yes, sounds reasonable to me.
Pekka
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-10 10:33 ` Pekka Enberg
0 siblings, 0 replies; 138+ messages in thread
From: Pekka Enberg @ 2011-05-10 10:33 UTC (permalink / raw)
To: Mel Gorman
Cc: James Bottomley, Mel Gorman, Jan Kara, colin.king, Chris Mason,
linux-fsdevel, linux-mm, linux-kernel, linux-ext4,
Christoph Lameter
On Tue, May 10, 2011 at 1:21 PM, Mel Gorman <mgorman@novell.com> wrote:
> It goes on. A number of filesystem and network paths are being hit
> with high-order allocs. i915 was a red herring, it's present but not
> in massive numbers. The filesystem, network and mempool allocations
> are likely to be kicking kswapd awake frequently and hurting overall
> system performance as a result.
>
> I really would like to hear if the fix makes a big difference or
> if we need to consider forcing SLUB high-order allocations bailing
> at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> allocate_slab). Even with the fix applied, kswapd might be waking up
> less but processes will still be getting stalled in direct compaction
> and direct reclaim so it would still be jittery.
Yes, sounds reasonable to me.
Pekka
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-10 10:21 ` Mel Gorman
@ 2011-05-10 14:01 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-10 14:01 UTC (permalink / raw)
To: Mel Gorman
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> I really would like to hear if the fix makes a big difference or
> if we need to consider forcing SLUB high-order allocations bailing
> at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> allocate_slab). Even with the fix applied, kswapd might be waking up
> less but processes will still be getting stalled in direct compaction
> and direct reclaim so it would still be jittery.
"the fix" being this
https://lkml.org/lkml/2011/3/5/121
In addition to your GFP_KSWAPD one?
OK, will retry with that.
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-10 14:01 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-10 14:01 UTC (permalink / raw)
To: Mel Gorman
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> I really would like to hear if the fix makes a big difference or
> if we need to consider forcing SLUB high-order allocations bailing
> at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> allocate_slab). Even with the fix applied, kswapd might be waking up
> less but processes will still be getting stalled in direct compaction
> and direct reclaim so it would still be jittery.
"the fix" being this
https://lkml.org/lkml/2011/3/5/121
In addition to your GFP_KSWAPD one?
OK, will retry with that.
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-10 14:01 ` James Bottomley
@ 2011-05-10 14:35 ` Mel Gorman
-1 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-05-10 14:35 UTC (permalink / raw)
To: James Bottomley
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
[-- Attachment #1: Type: text/plain, Size: 1044 bytes --]
On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> > I really would like to hear if the fix makes a big difference or
> > if we need to consider forcing SLUB high-order allocations bailing
> > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> > allocate_slab). Even with the fix applied, kswapd might be waking up
> > less but processes will still be getting stalled in direct compaction
> > and direct reclaim so it would still be jittery.
>
> "the fix" being this
>
> https://lkml.org/lkml/2011/3/5/121
>
Drop this for the moment. It was a long shot at best and there is little
evidence the problem is in this area.
I'm attaching two patches. The first is the NO_KSWAPD one to stop
kswapd being woken up by SLUB using speculative high-orders. The second
one is more drastic and prevents slub entering direct reclaim or
compaction. It applies on top of patch 1. These are both untested and
afraid are a bit rushed as well :(
--
Mel Gorman
SUSE Labs
[-- Attachment #2: mm-slub-do-not-wake-kswapd-for-slub-high-orders.patch --]
[-- Type: text/x-patch, Size: 1395 bytes --]
>From b48dee7d13980d4d901e3035dc6096c28c42c2ed Mon Sep 17 00:00:00 2001
From: Mel Gorman <mgorman@suse.de>
Date: Tue, 10 May 2011 15:13:30 +0100
Subject: [PATCH] mm: slub: Do not wake kswapd for SLUBs speculative high-order allocations
To avoid locking and per-cpu overhead, SLUB optimisically uses
high-order allocations and falls back to lower allocations if they fail.
However, by simply trying to allocate, kswapd is woken up to start
reclaiming at that order. On a desktop system, two users report that the
system is getting locked up with kswapd using large amounts of CPU.
Using SLAB instead of SLUB makes this problem go away.
This patch prevents kswapd being woken up for high-order allocations.
Not-signed-off-yet: Mel Gorman <mgorman@suse.de>
---
mm/slub.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c
index 9d2e5e4..98c358d 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -1170,7 +1170,7 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
* Let the initial higher-order allocation fail under memory pressure
* so we fall-back to the minimum order allocation.
*/
- alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY) & ~__GFP_NOFAIL;
+ alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY | __GFP_NO_KSWAPD) & ~__GFP_NOFAIL;
page = alloc_slab_page(alloc_gfp, node, oo);
if (unlikely(!page)) {
[-- Attachment #3: mm-slub-do-not-take-expensive-steps-for-slub-high-orders.patch --]
[-- Type: text/x-patch, Size: 2400 bytes --]
>From 59220aa310c0ba60afee29eeea1e602f4a374c60 Mon Sep 17 00:00:00 2001
From: Mel Gorman <mgorman@suse.de>
Date: Tue, 10 May 2011 15:30:20 +0100
Subject: [PATCH] mm: slub: Do not take expensive steps for SLUBs speculative high-order allocations
To avoid locking and per-cpu overhead, SLUB optimisically uses
high-order allocations and falls back to lower allocations if they
fail. However, by simply trying to allocate, the caller can enter
compaction or reclaim - both of which are likely to cost more than
the benefit of using high-order pages in SLUB. On a desktop system,
two users report that the system is getting locked up with kswapd
using large amounts of CPU. Using SLAB instead of SLUB makes this
problem go away.
This patch prevents SLUB taking any expensive steps when trying to use
high-order allocations. Instead, it is expected to fall back to smaller
orders more aggressively.
Not-signed-off-yet: Mel Gorman <mgorman@suse.de>
---
mm/page_alloc.c | 3 ++-
mm/slub.c | 3 ++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 9f8a97b..f160d93 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1972,6 +1972,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask)
{
int alloc_flags = ALLOC_WMARK_MIN | ALLOC_CPUSET;
const gfp_t wait = gfp_mask & __GFP_WAIT;
+ const gfp_t wakes_kswapd = !(gfp_mask & __GFP_NO_KSWAPD);
/* __GFP_HIGH is assumed to be the same as ALLOC_HIGH to save a branch. */
BUILD_BUG_ON(__GFP_HIGH != (__force gfp_t) ALLOC_HIGH);
@@ -1984,7 +1985,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask)
*/
alloc_flags |= (__force int) (gfp_mask & __GFP_HIGH);
- if (!wait) {
+ if (!wait && wakes_kswapd) {
/*
* Not worth trying to allocate harder for
* __GFP_NOMEMALLOC even if it can't schedule.
diff --git a/mm/slub.c b/mm/slub.c
index 98c358d..1071723 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -1170,7 +1170,8 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
* Let the initial higher-order allocation fail under memory pressure
* so we fall-back to the minimum order allocation.
*/
- alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY | __GFP_NO_KSWAPD) & ~__GFP_NOFAIL;
+ alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY | __GFP_NO_KSWAPD) &
+ ~(__GFP_NOFAIL | __GFP_WAIT);
page = alloc_slab_page(alloc_gfp, node, oo);
if (unlikely(!page)) {
^ permalink raw reply related [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-10 14:35 ` Mel Gorman
0 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-05-10 14:35 UTC (permalink / raw)
To: James Bottomley
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
[-- Attachment #1: Type: text/plain, Size: 1044 bytes --]
On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> > I really would like to hear if the fix makes a big difference or
> > if we need to consider forcing SLUB high-order allocations bailing
> > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> > allocate_slab). Even with the fix applied, kswapd might be waking up
> > less but processes will still be getting stalled in direct compaction
> > and direct reclaim so it would still be jittery.
>
> "the fix" being this
>
> https://lkml.org/lkml/2011/3/5/121
>
Drop this for the moment. It was a long shot at best and there is little
evidence the problem is in this area.
I'm attaching two patches. The first is the NO_KSWAPD one to stop
kswapd being woken up by SLUB using speculative high-orders. The second
one is more drastic and prevents slub entering direct reclaim or
compaction. It applies on top of patch 1. These are both untested and
afraid are a bit rushed as well :(
--
Mel Gorman
SUSE Labs
[-- Attachment #2: mm-slub-do-not-wake-kswapd-for-slub-high-orders.patch --]
[-- Type: text/x-patch, Size: 0 bytes --]
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-10 14:35 ` Mel Gorman
@ 2011-05-10 15:29 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-10 15:29 UTC (permalink / raw)
To: Mel Gorman
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
> On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> > > I really would like to hear if the fix makes a big difference or
> > > if we need to consider forcing SLUB high-order allocations bailing
> > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> > > allocate_slab). Even with the fix applied, kswapd might be waking up
> > > less but processes will still be getting stalled in direct compaction
> > > and direct reclaim so it would still be jittery.
> >
> > "the fix" being this
> >
> > https://lkml.org/lkml/2011/3/5/121
> >
>
> Drop this for the moment. It was a long shot at best and there is little
> evidence the problem is in this area.
>
> I'm attaching two patches. The first is the NO_KSWAPD one to stop
> kswapd being woken up by SLUB using speculative high-orders. The second
> one is more drastic and prevents slub entering direct reclaim or
> compaction. It applies on top of patch 1. These are both untested and
> afraid are a bit rushed as well :(
Preliminary results with both patches applied still show kswapd
periodically going up to 99% but it doesn't stay there, it comes back
down again (and, obviously, the system doesn't hang).
This is sysrq-M from a couple of times when it went up there:
[ 426.736958] SysRq : Show Memory
[ 426.736974] Mem-Info:
[ 426.736977] Node 0 DMA per-cpu:
[ 426.736983] CPU 0: hi: 0, btch: 1 usd: 0
[ 426.736986] CPU 1: hi: 0, btch: 1 usd: 0
[ 426.736989] CPU 2: hi: 0, btch: 1 usd: 0
[ 426.736993] CPU 3: hi: 0, btch: 1 usd: 0
[ 426.736996] Node 0 DMA32 per-cpu:
[ 426.737002] CPU 0: hi: 186, btch: 31 usd: 169
[ 426.737005] CPU 1: hi: 186, btch: 31 usd: 40
[ 426.737009] CPU 2: hi: 186, btch: 31 usd: 166
[ 426.737012] CPU 3: hi: 186, btch: 31 usd: 168
[ 426.737015] Node 0 Normal per-cpu:
[ 426.737020] CPU 0: hi: 0, btch: 1 usd: 0
[ 426.737024] CPU 1: hi: 0, btch: 1 usd: 0
[ 426.737027] CPU 2: hi: 0, btch: 1 usd: 0
[ 426.737030] CPU 3: hi: 0, btch: 1 usd: 0
[ 426.737036] active_anon:108658 inactive_anon:37031 isolated_anon:0
[ 426.737037] active_file:32006 inactive_file:41051 isolated_file:32
[ 426.737038] unevictable:8 dirty:41202 writeback:204 unstable:0
[ 426.737039] free:191520 slab_reclaimable:8490 slab_unreclaimable:27477
[ 426.737039] mapped:9176 shmem:26412 pagetables:5427 bounce:0
[ 426.737051] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 426.737064] lowmem_reserve[]: 0 1856 1862 1862
[ 426.737078] Node 0 DMA32 free:757892kB min:66820kB low:83524kB high:100228kB active_anon:434632kB inactive_anon:148112kB active_file:127784kB inactive_file:156924kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:1901408kB mlocked:32kB dirty:164808kB writeback:816kB mapped:36704kB shmem:105648kB slab_reclaimable:33676kB slab_unreclaimable:108372kB kernel_stack:2304kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
[ 426.737092] lowmem_reserve[]: 0 0 5 5
[ 426.737106] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[ 426.737119] lowmem_reserve[]: 0 0 0 0
[ 426.737132] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB
[ 426.737174] Node 0 DMA32: 945*4kB 585*8kB 6469*16kB 4871*32kB 3189*64kB 1338*128kB 228*256kB 58*512kB 24*1024kB 1*2048kB 0*4096kB = 757884kB
[ 426.737227] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB
[ 426.737257] 99595 total pagecache pages
[ 426.737260] 0 pages in swap cache
[ 426.737263] Swap cache stats: add 0, delete 0, find 0/0
[ 426.737266] Free swap = 3768316kB
[ 426.737268] Total swap = 3768316kB
[ 426.744603] 525808 pages RAM
[ 426.744612] 57618 pages reserved
[ 426.744614] 141551 pages shared
[ 426.744617] 186065 pages non-shared
[ 472.301810] SysRq : Show Memory
[ 472.301826] Mem-Info:
[ 472.301829] Node 0 DMA per-cpu:
[ 472.301835] CPU 0: hi: 0, btch: 1 usd: 0
[ 472.301839] CPU 1: hi: 0, btch: 1 usd: 0
[ 472.301842] CPU 2: hi: 0, btch: 1 usd: 0
[ 472.301845] CPU 3: hi: 0, btch: 1 usd: 0
[ 472.301848] Node 0 DMA32 per-cpu:
[ 472.301854] CPU 0: hi: 186, btch: 31 usd: 184
[ 472.301857] CPU 1: hi: 186, btch: 31 usd: 46
[ 472.301860] CPU 2: hi: 186, btch: 31 usd: 158
[ 472.301863] CPU 3: hi: 186, btch: 31 usd: 163
[ 472.301866] Node 0 Normal per-cpu:
[ 472.301871] CPU 0: hi: 0, btch: 1 usd: 0
[ 472.301874] CPU 1: hi: 0, btch: 1 usd: 0
[ 472.301878] CPU 2: hi: 0, btch: 1 usd: 0
[ 472.301881] CPU 3: hi: 0, btch: 1 usd: 0
[ 472.301886] active_anon:107673 inactive_anon:37031 isolated_anon:0
[ 472.301887] active_file:31533 inactive_file:33323 isolated_file:32
[ 472.301888] unevictable:8 dirty:26256 writeback:6475 unstable:0
[ 472.301889] free:198742 slab_reclaimable:9347 slab_unreclaimable:28647
[ 472.301889] mapped:8307 shmem:26412 pagetables:5427 bounce:0
[ 472.301901] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 472.301915] lowmem_reserve[]: 0 1856 1862 1862
[ 472.301928] Node 0 DMA32 free:786780kB min:66820kB low:83524kB high:100228kB active_anon:430692kB inactive_anon:148112kB active_file:125892kB inactive_file:126012kB unevictable:32kB isolated(anon):0kB isolated(file):128kB present:1901408kB mlocked:32kB dirty:105024kB writeback:25900kB mapped:33228kB shmem:105648kB slab_reclaimable:37104kB slab_unreclaimable:113052kB kernel_stack:2288kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3196 all_unreclaimable? no
[ 472.301943] lowmem_reserve[]: 0 0 5 5
[ 472.301956] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[ 472.301968] lowmem_reserve[]: 0 0 0 0
[ 472.301982] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB
[ 472.302015] Node 0 DMA32: 6121*4kB 1912*8kB 5094*16kB 4920*32kB 3168*64kB 1381*128kB 262*256kB 68*512kB 24*1024kB 1*2048kB 0*4096kB = 786756kB
[ 472.302048] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB
[ 472.302090] 91312 total pagecache pages
[ 472.302093] 0 pages in swap cache
[ 472.302096] Swap cache stats: add 0, delete 0, find 0/0
[ 472.302099] Free swap = 3768316kB
[ 472.302102] Total swap = 3768316kB
[ 472.309521] 525808 pages RAM
[ 472.309529] 57618 pages reserved
[ 472.309548] 142496 pages shared
[ 472.309551] 177182 pages non-shared
I'll finish off this verification, and then re-run with the watch-highorder script running.
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-10 15:29 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-10 15:29 UTC (permalink / raw)
To: Mel Gorman
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
> On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> > > I really would like to hear if the fix makes a big difference or
> > > if we need to consider forcing SLUB high-order allocations bailing
> > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> > > allocate_slab). Even with the fix applied, kswapd might be waking up
> > > less but processes will still be getting stalled in direct compaction
> > > and direct reclaim so it would still be jittery.
> >
> > "the fix" being this
> >
> > https://lkml.org/lkml/2011/3/5/121
> >
>
> Drop this for the moment. It was a long shot at best and there is little
> evidence the problem is in this area.
>
> I'm attaching two patches. The first is the NO_KSWAPD one to stop
> kswapd being woken up by SLUB using speculative high-orders. The second
> one is more drastic and prevents slub entering direct reclaim or
> compaction. It applies on top of patch 1. These are both untested and
> afraid are a bit rushed as well :(
Preliminary results with both patches applied still show kswapd
periodically going up to 99% but it doesn't stay there, it comes back
down again (and, obviously, the system doesn't hang).
This is sysrq-M from a couple of times when it went up there:
[ 426.736958] SysRq : Show Memory
[ 426.736974] Mem-Info:
[ 426.736977] Node 0 DMA per-cpu:
[ 426.736983] CPU 0: hi: 0, btch: 1 usd: 0
[ 426.736986] CPU 1: hi: 0, btch: 1 usd: 0
[ 426.736989] CPU 2: hi: 0, btch: 1 usd: 0
[ 426.736993] CPU 3: hi: 0, btch: 1 usd: 0
[ 426.736996] Node 0 DMA32 per-cpu:
[ 426.737002] CPU 0: hi: 186, btch: 31 usd: 169
[ 426.737005] CPU 1: hi: 186, btch: 31 usd: 40
[ 426.737009] CPU 2: hi: 186, btch: 31 usd: 166
[ 426.737012] CPU 3: hi: 186, btch: 31 usd: 168
[ 426.737015] Node 0 Normal per-cpu:
[ 426.737020] CPU 0: hi: 0, btch: 1 usd: 0
[ 426.737024] CPU 1: hi: 0, btch: 1 usd: 0
[ 426.737027] CPU 2: hi: 0, btch: 1 usd: 0
[ 426.737030] CPU 3: hi: 0, btch: 1 usd: 0
[ 426.737036] active_anon:108658 inactive_anon:37031 isolated_anon:0
[ 426.737037] active_file:32006 inactive_file:41051 isolated_file:32
[ 426.737038] unevictable:8 dirty:41202 writeback:204 unstable:0
[ 426.737039] free:191520 slab_reclaimable:8490 slab_unreclaimable:27477
[ 426.737039] mapped:9176 shmem:26412 pagetables:5427 bounce:0
[ 426.737051] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 426.737064] lowmem_reserve[]: 0 1856 1862 1862
[ 426.737078] Node 0 DMA32 free:757892kB min:66820kB low:83524kB high:100228kB active_anon:434632kB inactive_anon:148112kB active_file:127784kB inactive_file:156924kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:1901408kB mlocked:32kB dirty:164808kB writeback:816kB mapped:36704kB shmem:105648kB slab_reclaimable:33676kB slab_unreclaimable:108372kB kernel_stack:2304kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
[ 426.737092] lowmem_reserve[]: 0 0 5 5
[ 426.737106] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[ 426.737119] lowmem_reserve[]: 0 0 0 0
[ 426.737132] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB
[ 426.737174] Node 0 DMA32: 945*4kB 585*8kB 6469*16kB 4871*32kB 3189*64kB 1338*128kB 228*256kB 58*512kB 24*1024kB 1*2048kB 0*4096kB = 757884kB
[ 426.737227] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB
[ 426.737257] 99595 total pagecache pages
[ 426.737260] 0 pages in swap cache
[ 426.737263] Swap cache stats: add 0, delete 0, find 0/0
[ 426.737266] Free swap = 3768316kB
[ 426.737268] Total swap = 3768316kB
[ 426.744603] 525808 pages RAM
[ 426.744612] 57618 pages reserved
[ 426.744614] 141551 pages shared
[ 426.744617] 186065 pages non-shared
[ 472.301810] SysRq : Show Memory
[ 472.301826] Mem-Info:
[ 472.301829] Node 0 DMA per-cpu:
[ 472.301835] CPU 0: hi: 0, btch: 1 usd: 0
[ 472.301839] CPU 1: hi: 0, btch: 1 usd: 0
[ 472.301842] CPU 2: hi: 0, btch: 1 usd: 0
[ 472.301845] CPU 3: hi: 0, btch: 1 usd: 0
[ 472.301848] Node 0 DMA32 per-cpu:
[ 472.301854] CPU 0: hi: 186, btch: 31 usd: 184
[ 472.301857] CPU 1: hi: 186, btch: 31 usd: 46
[ 472.301860] CPU 2: hi: 186, btch: 31 usd: 158
[ 472.301863] CPU 3: hi: 186, btch: 31 usd: 163
[ 472.301866] Node 0 Normal per-cpu:
[ 472.301871] CPU 0: hi: 0, btch: 1 usd: 0
[ 472.301874] CPU 1: hi: 0, btch: 1 usd: 0
[ 472.301878] CPU 2: hi: 0, btch: 1 usd: 0
[ 472.301881] CPU 3: hi: 0, btch: 1 usd: 0
[ 472.301886] active_anon:107673 inactive_anon:37031 isolated_anon:0
[ 472.301887] active_file:31533 inactive_file:33323 isolated_file:32
[ 472.301888] unevictable:8 dirty:26256 writeback:6475 unstable:0
[ 472.301889] free:198742 slab_reclaimable:9347 slab_unreclaimable:28647
[ 472.301889] mapped:8307 shmem:26412 pagetables:5427 bounce:0
[ 472.301901] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 472.301915] lowmem_reserve[]: 0 1856 1862 1862
[ 472.301928] Node 0 DMA32 free:786780kB min:66820kB low:83524kB high:100228kB active_anon:430692kB inactive_anon:148112kB active_file:125892kB inactive_file:126012kB unevictable:32kB isolated(anon):0kB isolated(file):128kB present:1901408kB mlocked:32kB dirty:105024kB writeback:25900kB mapped:33228kB shmem:105648kB slab_reclaimable:37104kB slab_unreclaimable:113052kB kernel_stack:2288kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3196 all_unreclaimable? no
[ 472.301943] lowmem_reserve[]: 0 0 5 5
[ 472.301956] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[ 472.301968] lowmem_reserve[]: 0 0 0 0
[ 472.301982] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB
[ 472.302015] Node 0 DMA32: 6121*4kB 1912*8kB 5094*16kB 4920*32kB 3168*64kB 1381*128kB 262*256kB 68*512kB 24*1024kB 1*2048kB 0*4096kB = 786756kB
[ 472.302048] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB
[ 472.302090] 91312 total pagecache pages
[ 472.302093] 0 pages in swap cache
[ 472.302096] Swap cache stats: add 0, delete 0, find 0/0
[ 472.302099] Free swap = 3768316kB
[ 472.302102] Total swap = 3768316kB
[ 472.309521] 525808 pages RAM
[ 472.309529] 57618 pages reserved
[ 472.309548] 142496 pages shared
[ 472.309551] 177182 pages non-shared
I'll finish off this verification, and then re-run with the watch-highorder script running.
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-10 15:29 ` James Bottomley
@ 2011-05-10 15:57 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-10 15:57 UTC (permalink / raw)
To: Mel Gorman
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote:
> On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
> > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> > > > I really would like to hear if the fix makes a big difference or
> > > > if we need to consider forcing SLUB high-order allocations bailing
> > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> > > > allocate_slab). Even with the fix applied, kswapd might be waking up
> > > > less but processes will still be getting stalled in direct compaction
> > > > and direct reclaim so it would still be jittery.
> > >
> > > "the fix" being this
> > >
> > > https://lkml.org/lkml/2011/3/5/121
> > >
> >
> > Drop this for the moment. It was a long shot at best and there is little
> > evidence the problem is in this area.
> >
> > I'm attaching two patches. The first is the NO_KSWAPD one to stop
> > kswapd being woken up by SLUB using speculative high-orders. The second
> > one is more drastic and prevents slub entering direct reclaim or
> > compaction. It applies on top of patch 1. These are both untested and
> > afraid are a bit rushed as well :(
>
> Preliminary results with both patches applied still show kswapd
> periodically going up to 99% but it doesn't stay there, it comes back
> down again (and, obviously, the system doesn't hang).
This is a second run with the watch highorders.
At the end of the run, the system hung temporarily and now comes back
with CPU3 spinning in all system time at kswapd shrink_slab
James
---
97 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063>
=> scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9>
127 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> sys_clone+0x28/0x2a <ffffffff8101150e>
=> stub_clone+0x13/0x20 <ffffffff81009ea3>
2 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d81>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144e14>
47 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> radix_tree_preload+0x31/0x81 <ffffffff812293b9>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc_trace+0x7b/0xd8 <ffffffff8110e2ce>
=> inode_doinit_with_dentry+0x124/0x45c <ffffffff811e96db>
=> selinux_d_instantiate+0x1c/0x1e <ffffffff811e9a2f>
3 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfdb>
=> kzalloc.constprop.15+0x13/0x15 [i915] <ffffffffa007a956>
=> i915_gem_alloc_object+0x27/0x111 [i915] <ffffffffa007e4b3>
4 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> mmap_region+0x1b7/0x446 <ffffffff810f6575>
=> do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6a9c>
27 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76b>
=> ra_submit+0x21/0x25 <ffffffff810ddaad>
=> ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7a>
=> page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddd04>
=> generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6510>
159 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
446 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f1>
=> do_filp_open+0xe7/0x60a <ffffffff81129bef>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> arch_dup_task_struct+0x42/0x8b <ffffffff81010f47>
=> copy_process+0xf9/0x10c5 <ffffffff810533ec>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sk_stream_alloc_skb+0x38/0xd3 <ffffffff813fc5a9>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5943>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff81196396>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d5408>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d617d>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6205>
25 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f1>
=> alloc_file+0x1e/0xbf <ffffffff8111f3d6>
115 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d81>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144e14>
56 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> d_alloc+0x26/0x18d <ffffffff8112e4e5>
=> d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d2e>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> alloc_buffer_head+0x21/0x49 <ffffffff81141405>
=> alloc_page_buffers+0x32/0xce <ffffffff81141533>
9 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107021>
=> khugepaged+0x543/0xf2d <ffffffff81113ce1>
=> kthread+0x84/0x8c <ffffffff8106f2df>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
347 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4b8>
=> alloc_inode+0x1d/0x78 <ffffffff81131805>
56 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5943>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff81196396>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d5408>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d617d>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6205>
60 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> dup_mm+0x1f8/0x486 <ffffffff81053039>
=> copy_process+0x917/0x10c5 <ffffffff81053c0a>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> bprm_mm_init+0x70/0x1a0 <ffffffff811239ca>
=> do_execve+0xd6/0x277 <ffffffff81123df4>
6 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107021>
=> do_huge_pmd_anonymous_page+0xbf/0x261 <ffffffff81115b8a>
=> handle_mm_fault+0x113/0x1ce <ffffffff810f1710>
=> do_page_fault+0x358/0x37a <ffffffff814741ae>
=> page_fault+0x25/0x30 <ffffffff81471455>
6 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> do_brk+0x1cd/0x293 <ffffffff810f5fd8>
=> load_elf_binary+0xd2d/0x175f <ffffffff8115ccf2>
4 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> kernel_thread+0x75/0x77 <ffffffff81010bb6>
=> kthreadd+0xe7/0x124 <ffffffff8106f61f>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
4 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e1b9>
=> __alloc_skb+0x40/0x133 <ffffffff813b5e17>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
534 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> kmalloc_large_node+0x56/0x95 <ffffffff8146a57d>
=> __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff28>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_COMP|GFP_ZERO
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> kmalloc_order_trace+0x2c/0x5e <ffffffff8110c738>
=> __kmalloc+0x37/0x10d <ffffffff8110df61>
=> kzalloc.constprop.19+0xe/0x10 <ffffffff810b8d8c>
=> tracing_open_pipe+0x40/0x147 <ffffffff810b907a>
=> __dentry_open+0x161/0x283 <ffffffff8111c97c>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76b>
=> ra_submit+0x21/0x25 <ffffffff810ddaad>
=> filemap_fault+0x17b/0x36c <ffffffff810d6ada>
=> __do_fault+0x55/0x377 <ffffffff810ee887>
=> handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f43>
55 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
8 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> shmem_alloc_inode+0x1a/0x2f <ffffffff810e58c0>
=> alloc_inode+0x1d/0x78 <ffffffff81131805>
563 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
12 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> __split_vma+0x6c/0x21b <ffffffff810f50b9>
=> split_vma+0x20/0x22 <ffffffff810f59c6>
5 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> __pollwait+0x5e/0xcc <ffffffff8112ca07>
=> sock_poll_wait+0x18/0x1d <ffffffff8143cc70>
=> unix_poll+0x1c/0x9a <ffffffff8143cc91>
=> sock_poll+0x1a/0x1c <ffffffff813ad43a>
=> do_select+0x2fb/0x4f5 <ffffffff8112cf33>
145216 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
48 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> proc_alloc_inode+0x20/0x91 <ffffffff811682bc>
=> alloc_inode+0x1d/0x78 <ffffffff81131805>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> copy_process+0xc6/0x10c5 <ffffffff810533b9>
=> do_fork+0x104/0x286 <ffffffff810544f7>
31 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> prepare_creds+0x26/0xae <ffffffff81074d4b>
=> sys_faccessat+0x37/0x162 <ffffffff8111d275>
1 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f6a>
High-order normal allocations: 147471
High-order atomic allocations: 720
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-10 15:57 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-10 15:57 UTC (permalink / raw)
To: Mel Gorman
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote:
> On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
> > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> > > > I really would like to hear if the fix makes a big difference or
> > > > if we need to consider forcing SLUB high-order allocations bailing
> > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> > > > allocate_slab). Even with the fix applied, kswapd might be waking up
> > > > less but processes will still be getting stalled in direct compaction
> > > > and direct reclaim so it would still be jittery.
> > >
> > > "the fix" being this
> > >
> > > https://lkml.org/lkml/2011/3/5/121
> > >
> >
> > Drop this for the moment. It was a long shot at best and there is little
> > evidence the problem is in this area.
> >
> > I'm attaching two patches. The first is the NO_KSWAPD one to stop
> > kswapd being woken up by SLUB using speculative high-orders. The second
> > one is more drastic and prevents slub entering direct reclaim or
> > compaction. It applies on top of patch 1. These are both untested and
> > afraid are a bit rushed as well :(
>
> Preliminary results with both patches applied still show kswapd
> periodically going up to 99% but it doesn't stay there, it comes back
> down again (and, obviously, the system doesn't hang).
This is a second run with the watch highorders.
At the end of the run, the system hung temporarily and now comes back
with CPU3 spinning in all system time at kswapd shrink_slab
James
---
97 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063>
=> scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9>
127 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> sys_clone+0x28/0x2a <ffffffff8101150e>
=> stub_clone+0x13/0x20 <ffffffff81009ea3>
2 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d81>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144e14>
47 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> radix_tree_preload+0x31/0x81 <ffffffff812293b9>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d5>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc_trace+0x7b/0xd8 <ffffffff8110e2ce>
=> inode_doinit_with_dentry+0x124/0x45c <ffffffff811e96db>
=> selinux_d_instantiate+0x1c/0x1e <ffffffff811e9a2f>
3 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfdb>
=> kzalloc.constprop.15+0x13/0x15 [i915] <ffffffffa007a956>
=> i915_gem_alloc_object+0x27/0x111 [i915] <ffffffffa007e4b3>
4 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> mmap_region+0x1b7/0x446 <ffffffff810f6575>
=> do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6a9c>
27 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76b>
=> ra_submit+0x21/0x25 <ffffffff810ddaad>
=> ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7a>
=> page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddd04>
=> generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6510>
159 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
446 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f1>
=> do_filp_open+0xe7/0x60a <ffffffff81129bef>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> arch_dup_task_struct+0x42/0x8b <ffffffff81010f47>
=> copy_process+0xf9/0x10c5 <ffffffff810533ec>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sk_stream_alloc_skb+0x38/0xd3 <ffffffff813fc5a9>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5943>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff81196396>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d5408>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d617d>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6205>
25 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f1>
=> alloc_file+0x1e/0xbf <ffffffff8111f3d6>
115 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d81>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144e14>
56 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> d_alloc+0x26/0x18d <ffffffff8112e4e5>
=> d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d2e>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> alloc_buffer_head+0x21/0x49 <ffffffff81141405>
=> alloc_page_buffers+0x32/0xce <ffffffff81141533>
9 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107021>
=> khugepaged+0x543/0xf2d <ffffffff81113ce1>
=> kthread+0x84/0x8c <ffffffff8106f2df>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
347 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4b8>
=> alloc_inode+0x1d/0x78 <ffffffff81131805>
56 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5943>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff81196396>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d5408>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d617d>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6205>
60 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> dup_mm+0x1f8/0x486 <ffffffff81053039>
=> copy_process+0x917/0x10c5 <ffffffff81053c0a>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> bprm_mm_init+0x70/0x1a0 <ffffffff811239ca>
=> do_execve+0xd6/0x277 <ffffffff81123df4>
6 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107021>
=> do_huge_pmd_anonymous_page+0xbf/0x261 <ffffffff81115b8a>
=> handle_mm_fault+0x113/0x1ce <ffffffff810f1710>
=> do_page_fault+0x358/0x37a <ffffffff814741ae>
=> page_fault+0x25/0x30 <ffffffff81471455>
6 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> do_brk+0x1cd/0x293 <ffffffff810f5fd8>
=> load_elf_binary+0xd2d/0x175f <ffffffff8115ccf2>
4 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> kernel_thread+0x75/0x77 <ffffffff81010bb6>
=> kthreadd+0xe7/0x124 <ffffffff8106f61f>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
4 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e1b9>
=> __alloc_skb+0x40/0x133 <ffffffff813b5e17>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
534 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> kmalloc_large_node+0x56/0x95 <ffffffff8146a57d>
=> __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff28>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_COMP|GFP_ZERO
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> kmalloc_order_trace+0x2c/0x5e <ffffffff8110c738>
=> __kmalloc+0x37/0x10d <ffffffff8110df61>
=> kzalloc.constprop.19+0xe/0x10 <ffffffff810b8d8c>
=> tracing_open_pipe+0x40/0x147 <ffffffff810b907a>
=> __dentry_open+0x161/0x283 <ffffffff8111c97c>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d5778>
=> __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76b>
=> ra_submit+0x21/0x25 <ffffffff810ddaad>
=> filemap_fault+0x17b/0x36c <ffffffff810d6ada>
=> __do_fault+0x55/0x377 <ffffffff810ee887>
=> handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f43>
55 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
8 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> shmem_alloc_inode+0x1a/0x2f <ffffffff810e58c0>
=> alloc_inode+0x1d/0x78 <ffffffff81131805>
563 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
12 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> __split_vma+0x6c/0x21b <ffffffff810f50b9>
=> split_vma+0x20/0x22 <ffffffff810f59c6>
5 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d2>
=> __pollwait+0x5e/0xcc <ffffffff8112ca07>
=> sock_poll_wait+0x18/0x1d <ffffffff8143cc70>
=> unix_poll+0x1c/0x9a <ffffffff8143cc91>
=> sock_poll+0x1a/0x1c <ffffffff813ad43a>
=> do_select+0x2fb/0x4f5 <ffffffff8112cf33>
145216 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e81>
=> mempool_alloc+0x68/0x116 <ffffffff810d70f6>
48 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> proc_alloc_inode+0x20/0x91 <ffffffff811682bc>
=> alloc_inode+0x1d/0x78 <ffffffff81131805>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> copy_process+0xc6/0x10c5 <ffffffff810533b9>
=> do_fork+0x104/0x286 <ffffffff810544f7>
31 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e44c>
=> prepare_creds+0x26/0xae <ffffffff81074d4b>
=> sys_faccessat+0x37/0x162 <ffffffff8111d275>
1 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0da>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105455>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fa>
=> new_slab+0x50/0x199 <ffffffff8110dc44>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffc6>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f6a>
High-order normal allocations: 147471
High-order atomic allocations: 720
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-10 15:57 ` James Bottomley
@ 2011-05-10 17:05 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-10 17:05 UTC (permalink / raw)
To: Mel Gorman
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Tue, 2011-05-10 at 15:57 +0000, James Bottomley wrote:
> On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote:
> > On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
> > > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> > > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> > > > > I really would like to hear if the fix makes a big difference or
> > > > > if we need to consider forcing SLUB high-order allocations bailing
> > > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> > > > > allocate_slab). Even with the fix applied, kswapd might be waking up
> > > > > less but processes will still be getting stalled in direct compaction
> > > > > and direct reclaim so it would still be jittery.
> > > >
> > > > "the fix" being this
> > > >
> > > > https://lkml.org/lkml/2011/3/5/121
> > > >
> > >
> > > Drop this for the moment. It was a long shot at best and there is little
> > > evidence the problem is in this area.
> > >
> > > I'm attaching two patches. The first is the NO_KSWAPD one to stop
> > > kswapd being woken up by SLUB using speculative high-orders. The second
> > > one is more drastic and prevents slub entering direct reclaim or
> > > compaction. It applies on top of patch 1. These are both untested and
> > > afraid are a bit rushed as well :(
> >
> > Preliminary results with both patches applied still show kswapd
> > periodically going up to 99% but it doesn't stay there, it comes back
> > down again (and, obviously, the system doesn't hang).
>
> This is a second run with the watch highorders.
>
> At the end of the run, the system hung temporarily and now comes back
> with CPU3 spinning in all system time at kswapd shrink_slab
Here's a trace in the same situation with the ftrace stack entries
bumped to 16 as requested on IRC. There was no hang for this one.
James
---
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> shmem_alloc_inode+0x1a/0x2f <ffffffff810e58c4>
=> alloc_inode+0x1d/0x78 <ffffffff81131809>
=> new_inode+0x1b/0x4b <ffffffff8113187f>
=> shmem_get_inode+0x45/0x1f6 <ffffffff810e49b3>
=> shmem_file_setup+0xe4/0x160 <ffffffff810e70ff>
=> drm_gem_object_init+0x34/0x61 [drm] <ffffffffa0024ffc>
=> i915_gem_alloc_object+0x41/0x111 [i915] <ffffffffa007e4cd>
=> i915_gem_create_ioctl+0x3f/0xcf [i915] <ffffffffa007e5dc>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
701 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> scsi_sg_alloc+0x2d/0x2f <ffffffff812f9e90>
=> __sg_alloc_table+0x63/0x11c <ffffffff81230ddf>
=> scsi_alloc_sgtable+0x2a/0x4f <ffffffff812f9ebc>
=> scsi_init_sgtable+0x21/0x61 <ffffffff812f9f02>
=> scsi_init_io+0x32/0x13b <ffffffff812f9f74>
=> scsi_setup_fs_cmnd+0xb9/0xbe <ffffffff812fa1b0>
=> sd_prep_fn+0x27b/0xb6d <ffffffff813026e9>
=> blk_peek_request+0xe1/0x1b6 <ffffffff81213411>
189 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> get_request+0x180/0x262 <ffffffff812124f6>
=> get_request_wait+0x35/0x193 <ffffffff81212f71>
=> __make_request+0x2b4/0x400 <ffffffff81213898>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> submit_bh+0xe6/0x105 <ffffffff81140a6f>
=> __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
=> block_write_full_page_endio+0x8a/0x97 <ffffffff81143671>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> inet_twsk_alloc+0x31/0x11e <ffffffff813f8a36>
=> tcp_time_wait+0xb9/0x29a <ffffffff8140e26f>
=> tcp_fin+0x7b/0x10f <ffffffff8140191a>
=> tcp_data_queue+0x1e4/0x876 <ffffffff8140207f>
=> tcp_rcv_state_process+0x880/0x8cf <ffffffff81404f67>
=> tcp_v4_do_rcv+0x312/0x363 <ffffffff8140b698>
=> tcp_v4_rcv+0x434/0x69b <ffffffff8140d209>
=> ip_local_deliver_finish+0x13d/0x1ae <ffffffff813f104d>
=> NF_HOOK.constprop.3+0x51/0x58 <ffffffff813f0eb2>
=> ip_local_deliver+0x4f/0x54 <ffffffff813f1205>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333>
=> ext4_iget+0x58/0x64c <ffffffff81192a48>
=> ext4_lookup+0x93/0xf3 <ffffffff8119a79b>
=> d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b>
=> __lookup_hash+0x7f/0x88 <ffffffff81127eaa>
9 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e1bd>
=> __alloc_skb+0x40/0x133 <ffffffff813b5e17>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_aio_write+0xb9/0xc9 <ffffffff813af85e>
=> do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d>
=> do_readv_writev+0xa7/0x11e <ffffffff8111ea03>
=> vfs_writev+0x45/0x47 <ffffffff8111eabf>
=> sys_writev+0x4a/0x93 <ffffffff8111ebe2>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> sk_prot_alloc+0x37/0x13a <ffffffff813b27f4>
=> sk_alloc+0x2c/0x8e <ffffffff813b2955>
=> unix_create1+0x4e/0x14d <ffffffff8143cae7>
=> unix_stream_connect+0x98/0x431 <ffffffff8143ecc6>
=> sys_connect+0x98/0xd6 <ffffffff813b0437>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
5 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> alloc_file+0x1e/0xbf <ffffffff8111f3da>
=> create_read_pipe+0x2b/0x5b <ffffffff81125ffe>
=> do_pipe_flags+0x4e/0xf4 <ffffffff8112607c>
=> sys_pipe2+0x1c/0x5c <ffffffff8112613e>
=> sys_pipe+0x10/0x12 <ffffffff8112618e>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
6 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f6a>
=> e1000_alloc_rx_buffers+0x8b/0x14e [e1000e] <ffffffffa01bfb58>
=> e1000_clean_rx_irq+0x2a6/0x2df [e1000e] <ffffffffa01c05a9>
=> e1000_clean+0x73/0x228 [e1000e] <ffffffffa01c0a07>
=> net_rx_action+0xa9/0x1ac <ffffffff813bfea2>
=> __do_softirq+0xd2/0x19d <ffffffff8105abe0>
=> call_softirq+0x1c/0x30 <ffffffff8100aa9c>
=> do_softirq+0x46/0x81 <ffffffff8100c0c1>
=> irq_exit+0x49/0x8b <ffffffff8105ad64>
48 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5947>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d540c>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d6181>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6209>
=> ext4_file_write+0x1dc/0x234 <ffffffff8118e898>
=> do_sync_write+0xbf/0xff <ffffffff8111dd6f>
=> vfs_write+0xac/0xf3 <ffffffff8111e3e5>
=> sys_write+0x4a/0x6e <ffffffff8111e5d4>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76f>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
=> filemap_fault+0x17b/0x36c <ffffffff810d6ade>
=> __do_fault+0x55/0x377 <ffffffff810ee88b>
=> handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f47>
=> handle_mm_fault+0x1bb/0x1ce <ffffffff810f17bc>
=> do_page_fault+0x358/0x37a <ffffffff814741ae>
=> page_fault+0x25/0x30 <ffffffff81471455>
16 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> do_read_cache_page+0x63/0x127 <ffffffff810d59f4>
=> read_cache_page_gfp+0x1e/0x28 <ffffffff810d5ad6>
=> i915_gem_object_bind_to_gtt+0x249/0x45b [i915] <ffffffffa007b0ad>
=> i915_gem_object_pin+0xc7/0x13c [i915] <ffffffffa007d609>
=> i915_gem_execbuffer_reserve+0x1b1/0x2fc [i915] <ffffffffa007f76d>
=> i915_gem_do_execbuffer+0x45b/0x1116 [i915] <ffffffffa008005a>
=> i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe>
409 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> jbd2_journal_commit_transaction+0x6e4/0x1052 <ffffffff811c56a6>
=> kjournald2+0xc9/0x20a <ffffffff811c9764>
=> kthread+0x84/0x8c <ffffffff8106f2df>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
4 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> mpage_readpages+0xf0/0x104 <ffffffff81148e05>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> copy_vma+0xe3/0x1e9 <ffffffff810f5cd3>
=> move_vma+0x8a/0x1fb <ffffffff810f7edb>
=> do_mremap+0x40e/0x46a <ffffffff810f845a>
=> sys_mremap+0x54/0x76 <ffffffff810f850a>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
5 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> do_read_cache_page+0x63/0x127 <ffffffff810d59f4>
=> read_cache_page_gfp+0x1e/0x28 <ffffffff810d5ad6>
=> i915_gem_object_bind_to_gtt+0x249/0x45b [i915] <ffffffffa007b0ad>
=> i915_gem_fault+0x85/0x18e [i915] <ffffffffa007d287>
=> __do_fault+0x55/0x377 <ffffffff810ee88b>
=> handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f47>
=> handle_mm_fault+0x1bb/0x1ce <ffffffff810f17bc>
1 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfdf>
=> kzalloc.constprop.1+0x13/0x15 [i915] <ffffffffa007fbfd>
=> i915_gem_do_execbuffer+0x306/0x1116 [i915] <ffffffffa007ff05>
=> i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
=> sys_ioctl+0x56/0x7b <ffffffff8112c107>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
5 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> copy_process+0xc6/0x10c5 <ffffffff810533b9>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> sys_clone+0x28/0x2a <ffffffff8101150e>
=> stub_clone+0x13/0x20 <ffffffff81009ea3>
4 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> ext4_mb_init_cache+0x1e6/0x52b <ffffffff811b45c8>
=> ext4_mb_load_buddy+0x134/0x30f <ffffffff811b4b1c>
=> ext4_mb_regular_allocator+0x13c/0x275 <ffffffff811b6637>
=> ext4_mb_new_blocks+0x17f/0x3c2 <ffffffff811b7432>
=> ext4_ext_map_blocks+0x192b/0x1b5b <ffffffff811b0a81>
102 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> ll_rw_block+0x84/0xa3 <ffffffff811410f4>
=> __breadahead+0x2e/0x39 <ffffffff81141c86>
=> __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b>
=> ext4_iget+0x58/0x64c <ffffffff81192a48>
=> ext4_lookup+0x93/0xf3 <ffffffff8119a79b>
1 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> get_request+0x180/0x262 <ffffffff812124f6>
=> get_request_wait+0x35/0x193 <ffffffff81212f71>
=> __make_request+0x2b4/0x400 <ffffffff81213898>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> mpage_readpages+0xf0/0x104 <ffffffff81148e05>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
3 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> __split_vma+0x6c/0x21b <ffffffff810f50bd>
=> split_vma+0x20/0x22 <ffffffff810f59ca>
=> mprotect_fixup+0x105/0x65b <ffffffff810f70ad>
=> sys_mprotect+0x197/0x209 <ffffffff810f779a>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
14 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063>
=> scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9>
=> __scsi_get_command+0x1b/0x96 <ffffffff812f3150>
=> scsi_get_command+0x3d/0xa5 <ffffffff812f3283>
=> scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165>
=> sd_prep_fn+0x27b/0xb6d <ffffffff813026e9>
=> blk_peek_request+0xe1/0x1b6 <ffffffff81213411>
=> scsi_request_fn+0x82/0x421 <ffffffff812f9a5f>
=> __generic_unplug_device+0x34/0x38 <ffffffff81212f38>
=> generic_unplug_device+0x2d/0x3d <ffffffff812131f0>
7 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> __split_vma+0x6c/0x21b <ffffffff810f50bd>
=> split_vma+0x20/0x22 <ffffffff810f59ca>
=> mprotect_fixup+0x125/0x65b <ffffffff810f70cd>
=> sys_mprotect+0x197/0x209 <ffffffff810f779a>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> find_or_create_page+0x5a/0x8a <ffffffff810d5b6d>
=> __getblk+0x137/0x23a <ffffffff81141b55>
=> ext4_getblk+0x8e/0x153 <ffffffff811957c0>
=> ext4_bread+0x16/0x66 <ffffffff8119589b>
=> ext4_mkdir+0x159/0x33e <ffffffff8119b786>
=> vfs_mkdir+0x5f/0x9b <ffffffff81128b01>
=> sys_mkdirat+0x97/0xe8 <ffffffff8112a3dd>
4 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_sendmsg+0xa1/0xb6 <ffffffff813af90f>
=> sys_sendmsg+0x237/0x2b5 <ffffffff813b0cd9>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
2 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_aio_write+0xb9/0xc9 <ffffffff813af85e>
=> do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d>
=> do_readv_writev+0xa7/0x11e <ffffffff8111ea03>
=> vfs_writev+0x45/0x47 <ffffffff8111eabf>
=> sys_writev+0x4a/0x93 <ffffffff8111ebe2>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> ext4_mb_init_cache+0x1e6/0x52b <ffffffff811b45c8>
=> ext4_mb_init_group+0x100/0x281 <ffffffff8146b9f8>
=> ext4_mb_good_group+0x58/0xdb <ffffffff811b4965>
=> ext4_mb_regular_allocator+0x125/0x275 <ffffffff811b6620>
=> ext4_mb_new_blocks+0x17f/0x3c2 <ffffffff811b7432>
39 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> ll_rw_block+0x84/0xa3 <ffffffff811410f4>
=> __breadahead+0x2e/0x39 <ffffffff81141c86>
=> __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b>
=> ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936>
=> ext4_reserve_inode_write+0x26/0x83 <ffffffff81193de1>
66 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> scsi_sg_alloc+0x2d/0x2f <ffffffff812f9e90>
=> __sg_alloc_table+0x63/0x11c <ffffffff81230ddf>
=> scsi_alloc_sgtable+0x2a/0x4f <ffffffff812f9ebc>
=> scsi_init_sgtable+0x21/0x61 <ffffffff812f9f02>
=> scsi_init_io+0x32/0x13b <ffffffff812f9f74>
=> scsi_setup_fs_cmnd+0xb9/0xbe <ffffffff812fa1b0>
=> sd_prep_fn+0x27b/0xb6d <ffffffff813026e9>
=> blk_peek_request+0xe1/0x1b6 <ffffffff81213411>
2 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> do_read_cache_page+0x48/0x127 <ffffffff810d59d9>
=> read_cache_page_gfp+0x1e/0x28 <ffffffff810d5ad6>
=> i915_gem_object_bind_to_gtt+0x249/0x45b [i915] <ffffffffa007b0ad>
=> i915_gem_object_pin+0xc7/0x13c [i915] <ffffffffa007d609>
=> i915_gem_execbuffer_reserve+0x1b1/0x2fc [i915] <ffffffffa007f76d>
=> i915_gem_do_execbuffer+0x45b/0x1116 [i915] <ffffffffa008005a>
=> i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
=> sys_ioctl+0x56/0x7b <ffffffff8112c107>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
148 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107025>
=> khugepaged+0x543/0xf2d <ffffffff81113ce5>
=> kthread+0x84/0x8c <ffffffff8106f2df>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
526 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc>
=> alloc_inode+0x1d/0x78 <ffffffff81131809>
=> new_inode+0x1b/0x4b <ffffffff8113187f>
=> ext4_new_inode+0xa1/0xcf2 <ffffffff8118f9cb>
=> ext4_create+0xbc/0x13e <ffffffff8119b32e>
=> vfs_create+0x6c/0x8e <ffffffff81128c64>
=> do_last+0x166/0x2df <ffffffff81129993>
=> do_filp_open+0x2ac/0x60a <ffffffff81129db8>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_openat+0x11/0x13 <ffffffff8111da20>
17 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063>
=> scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9>
=> __scsi_get_command+0x1b/0x96 <ffffffff812f3150>
=> scsi_get_command+0x3d/0xa5 <ffffffff812f3283>
=> scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165>
=> sd_prep_fn+0x27b/0xb6d <ffffffff813026e9>
=> blk_peek_request+0xe1/0x1b6 <ffffffff81213411>
=> scsi_request_fn+0x82/0x421 <ffffffff812f9a5f>
=> __generic_unplug_device+0x34/0x38 <ffffffff81212f38>
=> __make_request+0x3a3/0x400 <ffffffff81213987>
43 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> ll_rw_block+0x84/0xa3 <ffffffff811410f4>
=> __breadahead+0x2e/0x39 <ffffffff81141c86>
=> __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b>
=> ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936>
=> ext4_xattr_get+0x57/0x213 <ffffffff811baf2b>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfdf>
=> drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2>
=> i915_gem_execbuffer2+0x4f/0x12e [i915] <ffffffffa0080f81>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
=> sys_ioctl+0x56/0x7b <ffffffff8112c107>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
6 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mmap_region+0x1b7/0x446 <ffffffff810f6579>
=> do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6aa0>
=> sys_mmap_pgoff+0x115/0x164 <ffffffff810f6c0f>
=> sys_mmap+0x22/0x24 <ffffffff8100da68>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
763 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> do_filp_open+0xe7/0x60a <ffffffff81129bf3>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_open+0x20/0x22 <ffffffff8111da0d>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
59 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc>
=> alloc_inode+0x1d/0x78 <ffffffff81131809>
=> new_inode+0x1b/0x4b <ffffffff8113187f>
=> ext4_new_inode+0xa1/0xcf2 <ffffffff8118f9cb>
=> ext4_mkdir+0x10b/0x33e <ffffffff8119b738>
=> vfs_mkdir+0x5f/0x9b <ffffffff81128b01>
=> sys_mkdirat+0x97/0xe8 <ffffffff8112a3dd>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
83 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> dup_mm+0x1f8/0x486 <ffffffff81053039>
=> copy_process+0x917/0x10c5 <ffffffff81053c0a>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> sys_clone+0x28/0x2a <ffffffff8101150e>
=> stub_clone+0x13/0x20 <ffffffff81009ea3>
4 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d85>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> mpage_alloc+0x2a/0x84 <ffffffff8114866b>
=> do_mpage_readpage+0x38c/0x468 <ffffffff81148bcf>
=> mpage_readpages+0xbe/0x104 <ffffffff81148dd3>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
=> ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7e>
33 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063>
=> scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9>
=> __scsi_get_command+0x1b/0x96 <ffffffff812f3150>
=> scsi_get_command+0x3d/0xa5 <ffffffff812f3283>
=> scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165>
=> sd_prep_fn+0x27b/0xb6d <ffffffff813026e9>
=> blk_peek_request+0xe1/0x1b6 <ffffffff81213411>
=> scsi_request_fn+0x82/0x421 <ffffffff812f9a5f>
=> __generic_unplug_device+0x34/0x38 <ffffffff81212f38>
=> elv_insert+0x172/0x17b <ffffffff8120dae8>
1 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107025>
=> do_wp_page+0x348/0x560 <ffffffff810ef735>
=> handle_pte_fault+0x73b/0x7a5 <ffffffff810f13db>
=> handle_mm_fault+0x1bb/0x1ce <ffffffff810f17bc>
=> do_page_fault+0x358/0x37a <ffffffff814741ae>
=> page_fault+0x25/0x30 <ffffffff81471455>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> d_alloc+0x26/0x18d <ffffffff8112e4e9>
=> d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32>
=> do_lookup+0x23a/0x287 <ffffffff81127dde>
=> link_path_walk+0x239/0x3b8 <ffffffff81128469>
=> do_path_lookup+0x4d/0xf6 <ffffffff8112881b>
=> do_filp_open+0x129/0x60a <ffffffff81129c35>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_open+0x20/0x22 <ffffffff8111da0d>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333>
=> ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936>
=> ext4_xattr_get+0x57/0x213 <ffffffff811baf2b>
=> ext4_xattr_security_get+0x2a/0x2c <ffffffff811bc92d>
=> generic_getxattr+0x57/0x60 <ffffffff81137e27>
1 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfdf>
=> kcalloc+0x38/0x3a <ffffffff811f5ff2>
=> security_get_user_sids+0x1c2/0x369 <ffffffff811f8503>
=> sel_write_user+0xd6/0x19d <ffffffff811ef5b3>
=> selinux_transaction_write+0x5b/0x77 <ffffffff811edbfc>
=> vfs_write+0xac/0xf3 <ffffffff8111e3e5>
=> sys_write+0x4a/0x6e <ffffffff8111e5d4>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
15 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> proc_alloc_inode+0x20/0x91 <ffffffff811682c0>
=> alloc_inode+0x1d/0x78 <ffffffff81131809>
=> new_inode+0x1b/0x4b <ffffffff8113187f>
=> proc_pid_make_inode+0x14/0xb7 <ffffffff811697c0>
=> proc_pident_instantiate+0x22/0xb1 <ffffffff8116c202>
=> proc_pident_lookup+0x84/0xa3 <ffffffff8116c3ce>
=> proc_tgid_base_lookup+0x1a/0x1c <ffffffff8116c43f>
=> d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b>
=> do_lookup+0x23a/0x287 <ffffffff81127dde>
=> link_path_walk+0x239/0x3b8 <ffffffff81128469>
213 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d6>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> sys_clone+0x28/0x2a <ffffffff8101150e>
=> stub_clone+0x13/0x20 <ffffffff81009ea3>
140162 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
=> block_write_full_page_endio+0x8a/0x97 <ffffffff81143671>
=> block_write_full_page+0x15/0x17 <ffffffff81143693>
=> mpage_da_submit_io+0x31a/0x395 <ffffffff811935d8>
=> mpage_da_map_and_submit+0x2ca/0x2e0 <ffffffff81196e88>
2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> d_alloc+0x26/0x18d <ffffffff8112e4e9>
=> d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32>
=> __lookup_hash+0x7f/0x88 <ffffffff81127eaa>
=> lookup_hash+0x19/0x1b <ffffffff81127ecc>
=> lookup_create+0x4e/0x94 <ffffffff81127f1c>
=> sys_mkdirat+0x43/0xe8 <ffffffff8112a389>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
53 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> submit_bh+0xe6/0x105 <ffffffff81140a6f>
=> jbd2_journal_commit_transaction+0x6e4/0x1052 <ffffffff811c56a6>
31 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063>
=> scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9>
=> __scsi_get_command+0x1b/0x96 <ffffffff812f3150>
=> scsi_get_command+0x3d/0xa5 <ffffffff812f3283>
=> scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165>
=> sd_prep_fn+0x27b/0xb6d <ffffffff813026e9>
=> blk_peek_request+0xe1/0x1b6 <ffffffff81213411>
=> scsi_request_fn+0x82/0x421 <ffffffff812f9a5f>
=> __blk_run_queue+0x4a/0x7a <ffffffff81212b10>
=> blk_run_queue+0x28/0x3c <ffffffff81212bf9>
1 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5947>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d540c>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d6181>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6209>
=> ext4_file_write+0x1dc/0x234 <ffffffff8118e898>
=> do_sync_write+0xbf/0xff <ffffffff8111dd6f>
=> vfs_write+0xac/0xf3 <ffffffff8111e3e5>
=> sys_write+0x4a/0x6e <ffffffff8111e5d4>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
8 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_aio_write+0xb9/0xc9 <ffffffff813af85e>
=> do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d>
=> do_readv_writev+0xa7/0x11e <ffffffff8111ea03>
=> vfs_writev+0x45/0x47 <ffffffff8111eabf>
=> sys_writev+0x4a/0x93 <ffffffff8111ebe2>
2 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d6>
=> __pollwait+0x5e/0xcc <ffffffff8112ca0b>
=> sock_poll_wait+0x18/0x1d <ffffffff8143cc70>
=> unix_poll+0x1c/0x9a <ffffffff8143cc91>
=> sock_poll+0x1a/0x1c <ffffffff813ad43a>
=> do_sys_poll+0x1fa/0x386 <ffffffff8112d781>
=> sys_poll+0x51/0xbb <ffffffff8112d9b0>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e1bd>
=> __alloc_skb+0x40/0x133 <ffffffff813b5e17>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_sendmsg+0xa1/0xb6 <ffffffff813af90f>
=> sys_sendmsg+0x237/0x2b5 <ffffffff813b0cd9>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
3 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> alloc_file+0x1e/0xbf <ffffffff8111f3da>
=> create_write_pipe+0x151/0x1bf <ffffffff81125f35>
=> do_pipe_flags+0x32/0xf4 <ffffffff81126060>
=> sys_pipe2+0x1c/0x5c <ffffffff8112613e>
=> sys_pipe+0x10/0x12 <ffffffff8112618e>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> dup_fd+0x33/0x25f <ffffffff81133573>
=> copy_process+0x5b8/0x10c5 <ffffffff810538ab>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> sys_clone+0x28/0x2a <ffffffff8101150e>
=> stub_clone+0x13/0x20 <ffffffff81009ea3>
2 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333>
=> ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936>
=> ext4_xattr_set_handle+0xca/0x3f0 <ffffffff811bb1b1>
=> ext4_init_security+0x61/0x82 <ffffffff811bc9d3>
=> ext4_new_inode+0xb69/0xcf2 <ffffffff81190493>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> skb_clone+0x50/0x70 <ffffffff813b5db7>
=> dev_hard_start_xmit+0xd1/0x535 <ffffffff813c1660>
=> sch_direct_xmit+0x6b/0x147 <ffffffff813d7173>
=> dev_queue_xmit+0x2ac/0x461 <ffffffff813c1d70>
=> neigh_resolve_output+0x205/0x236 <ffffffff813ca2be>
=> ip_finish_output2+0x1cc/0x20f <ffffffff813f3979>
=> ip_finish_output+0x65/0x6a <ffffffff813f42a9>
=> ip_output+0x8c/0x91 <ffffffff813f49f2>
=> dst_output+0x14/0x16 <ffffffff813f34d7>
=> ip_local_out+0x20/0x24 <ffffffff813f435f>
3 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d6>
=> __pollwait+0x5e/0xcc <ffffffff8112ca0b>
=> sock_poll_wait+0x18/0x1d <ffffffff8143cc70>
=> unix_poll+0x1c/0x9a <ffffffff8143cc91>
=> sock_poll+0x1a/0x1c <ffffffff813ad43a>
=> do_select+0x2fb/0x4f5 <ffffffff8112cf37>
=> core_sys_select+0x163/0x202 <ffffffff8112d294>
=> sys_select+0x91/0xb9 <ffffffff8112d3c4>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
2 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d85>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> mpage_alloc+0x2a/0x84 <ffffffff8114866b>
=> do_mpage_readpage+0x38c/0x468 <ffffffff81148bcf>
=> mpage_readpages+0xbe/0x104 <ffffffff81148dd3>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
=> filemap_fault+0x17b/0x36c <ffffffff810d6ade>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> do_filp_open+0xe7/0x60a <ffffffff81129bf3>
=> open_exec+0x26/0x82 <ffffffff8112265e>
=> load_elf_binary+0x1df/0x175f <ffffffff8115c1a8>
=> search_binary_handler+0xb6/0x22e <ffffffff81122ce3>
=> do_execve+0x17e/0x277 <ffffffff81123ea0>
=> sys_execve+0x43/0x5a <ffffffff81011553>
=> stub_execve+0x6c/0xc0 <ffffffff81009fdc>
210 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> grab_cache_page_write_begin+0x6d/0x9e <ffffffff810d5960>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d540c>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d6181>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6209>
=> ext4_file_write+0x1dc/0x234 <ffffffff8118e898>
=> do_sync_write+0xbf/0xff <ffffffff8111dd6f>
6 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> do_filp_open+0xe7/0x60a <ffffffff81129bf3>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_openat+0x11/0x13 <ffffffff8111da20>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
8 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107025>
=> do_huge_pmd_anonymous_page+0xbf/0x261 <ffffffff81115b8e>
=> handle_mm_fault+0x113/0x1ce <ffffffff810f1714>
=> do_page_fault+0x358/0x37a <ffffffff814741ae>
=> page_fault+0x25/0x30 <ffffffff81471455>
4 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> submit_bh+0xe6/0x105 <ffffffff81140a6f>
=> ll_rw_block+0x84/0xa3 <ffffffff811410f4>
5 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333>
=> ext4_iget+0x58/0x64c <ffffffff81192a48>
=> ext4_lookup+0x93/0xf3 <ffffffff8119a79b>
=> d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b>
=> do_lookup+0x23a/0x287 <ffffffff81127dde>
257 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
=> block_write_full_page_endio+0x8a/0x97 <ffffffff81143671>
=> block_write_full_page+0x15/0x17 <ffffffff81143693>
=> ext4_writepage+0x201/0x210 <ffffffff81193854>
=> shrink_page_list+0x3c7/0x6de <ffffffff810e22a1>
809 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> kmalloc_large_node+0x56/0x95 <ffffffff8146a57d>
=> __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff2c>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_aio_write+0xb9/0xc9 <ffffffff813af85e>
=> do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d>
=> do_readv_writev+0xa7/0x11e <ffffffff8111ea03>
=> vfs_writev+0x45/0x47 <ffffffff8111eabf>
=> sys_writev+0x4a/0x93 <ffffffff8111ebe2>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
2 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc>
=> alloc_inode+0x1d/0x78 <ffffffff81131809>
=> iget_locked+0x61/0xdd <ffffffff811323be>
=> ext4_iget+0x2c/0x64c <ffffffff81192a1c>
=> ext4_lookup+0x93/0xf3 <ffffffff8119a79b>
=> d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b>
=> do_lookup+0x23a/0x287 <ffffffff81127dde>
=> link_path_walk+0x239/0x3b8 <ffffffff81128469>
=> do_path_lookup+0x4d/0xf6 <ffffffff8112881b>
=> do_filp_open+0x129/0x60a <ffffffff81129c35>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> alloc_file+0x1e/0xbf <ffffffff8111f3da>
=> shmem_file_setup+0x125/0x160 <ffffffff810e7140>
=> drm_gem_object_init+0x34/0x61 [drm] <ffffffffa0024ffc>
=> i915_gem_alloc_object+0x41/0x111 [i915] <ffffffffa007e4cd>
=> i915_gem_create_ioctl+0x3f/0xcf [i915] <ffffffffa007e5dc>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
=> sys_ioctl+0x56/0x7b <ffffffff8112c107>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
99 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> mpage_alloc+0x2a/0x84 <ffffffff8114866b>
=> do_mpage_readpage+0x38c/0x468 <ffffffff81148bcf>
=> mpage_readpages+0xbe/0x104 <ffffffff81148dd3>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
24 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76f>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
=> ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7e>
=> page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddd08>
=> generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6514>
=> do_sync_read+0xbf/0xff <ffffffff8111de6e>
=> vfs_read+0xa9/0xf0 <ffffffff8111e4d5>
=> sys_read+0x4a/0x6e <ffffffff8111e566>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
21 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> prepare_creds+0x26/0xae <ffffffff81074d4b>
=> sys_faccessat+0x37/0x162 <ffffffff8111d279>
=> sys_access+0x18/0x1a <ffffffff8111d3bc>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_COMP|GFP_ZERO
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d6>
=> kmalloc_order_trace+0x2c/0x5e <ffffffff8110c73c>
=> __kmalloc+0x37/0x10d <ffffffff8110df65>
=> kzalloc.constprop.19+0xe/0x10 <ffffffff810b8d8c>
=> tracing_open_pipe+0x40/0x147 <ffffffff810b907a>
=> __dentry_open+0x161/0x283 <ffffffff8111c980>
=> nameidata_to_filp+0x60/0x67 <ffffffff8111d8f4>
=> finish_open+0xa1/0x17f <ffffffff8112974f>
=> do_filp_open+0x186/0x60a <ffffffff81129c92>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_open+0x20/0x22 <ffffffff8111da0d>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> do_mpage_readpage+0x34a/0x468 <ffffffff81148b8d>
=> mpage_readpages+0xbe/0x104 <ffffffff81148dd3>
2 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> find_or_create_page+0x5a/0x8a <ffffffff810d5b6d>
=> __getblk+0x137/0x23a <ffffffff81141b55>
=> __ext4_get_inode_loc+0x118/0x344 <ffffffff8119115f>
=> ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936>
=> ext4_xattr_set_handle+0xca/0x3f0 <ffffffff811bb1b1>
=> ext4_init_security+0x61/0x82 <ffffffff811bc9d3>
=> ext4_new_inode+0xb69/0xcf2 <ffffffff81190493>
8 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> ll_rw_block+0x84/0xa3 <ffffffff811410f4>
=> __breadahead+0x2e/0x39 <ffffffff81141c86>
=> __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b>
=> ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936>
=> ext4_xattr_set_handle+0xca/0x3f0 <ffffffff811bb1b1>
58 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> d_alloc+0x26/0x18d <ffffffff8112e4e9>
=> d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32>
=> __lookup_hash+0x7f/0x88 <ffffffff81127eaa>
=> lookup_hash+0x19/0x1b <ffffffff81127ecc>
=> do_last+0xcf/0x2df <ffffffff811298fc>
=> do_filp_open+0x2ac/0x60a <ffffffff81129db8>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_openat+0x11/0x13 <ffffffff8111da20>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
5 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> mpage_readpages+0x99/0x104 <ffffffff81148dae>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
=> ondemand_readahead+0x15a/0x1d8 <ffffffff810ddc0f>
=> page_cache_sync_readahead+0x3d/0x40 <ffffffff810ddd6d>
=> generic_file_aio_read+0x27d/0x5e0 <ffffffff810d64d4>
=> do_sync_read+0xbf/0xff <ffffffff8111de6e>
2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> alloc_file+0x1e/0xbf <ffffffff8111f3da>
=> create_read_pipe+0x2b/0x5b <ffffffff81125ffe>
=> do_pipe_flags+0x4e/0xf4 <ffffffff8112607c>
=> sys_pipe2+0x1c/0x5c <ffffffff8112613e>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
9 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d6>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> kernel_thread+0x75/0x77 <ffffffff81010bb6>
=> kthreadd+0xe7/0x124 <ffffffff8106f61f>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5947>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d540c>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d6181>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6209>
=> ext4_file_write+0x1dc/0x234 <ffffffff8118e898>
=> do_sync_write+0xbf/0xff <ffffffff8111dd6f>
=> vfs_write+0xac/0xf3 <ffffffff8111e3e5>
=> sys_write+0x4a/0x6e <ffffffff8111e5d4>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfdf>
=> drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2>
=> i915_gem_do_execbuffer+0x64a/0x1116 [i915] <ffffffffa0080249>
=> i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
=> sys_ioctl+0x56/0x7b <ffffffff8112c107>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> dst_alloc+0x48/0xaa <ffffffff813c6672>
=> __ip_route_output_key+0x561/0x764 <ffffffff813ee513>
=> tcp_v4_connect+0x121/0x47d <ffffffff8140becf>
=> inet_stream_connect+0xa8/0x25f <ffffffff81418e24>
=> sys_connect+0x98/0xd6 <ffffffff813b0437>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> kmalloc_large_node+0x56/0x95 <ffffffff8146a57d>
=> __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff2c>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_aio_write+0xb9/0xc9 <ffffffff813af85e>
=> do_sync_write+0xbf/0xff <ffffffff8111dd6f>
=> vfs_write+0xb3/0xf3 <ffffffff8111e3ec>
=> sys_write+0x4a/0x6e <ffffffff8111e5d4>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> __nf_conntrack_alloc+0xc9/0x1af <ffffffff813e4533>
=> nf_conntrack_in+0x2f6/0x77a <ffffffff813e490f>
=> ipv4_conntrack_local+0x46/0x4a <ffffffff8142864c>
=> nf_iterate+0x48/0x83 <ffffffff813e1cb8>
=> nf_hook_slow+0x6a/0xe9 <ffffffff813e1d5d>
=> nf_hook_thresh.constprop.8+0x42/0x44 <ffffffff813f42f0>
=> __ip_local_out+0x49/0x4d <ffffffff813f433b>
=> ip_local_out+0x13/0x24 <ffffffff813f4352>
=> ip_queue_xmit+0x2b3/0x30e <ffffffff813f4616>
=> tcp_transmit_skb+0x708/0x739 <ffffffff81407217>
1 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> dst_alloc+0x48/0xaa <ffffffff813c6672>
=> __ip_route_output_key+0x561/0x764 <ffffffff813ee513>
=> ip_route_output_flow+0x2c/0x1ea <ffffffff813ee742>
=> ip4_datagram_connect+0x18d/0x298 <ffffffff8140ede5>
=> inet_dgram_connect+0x65/0x6c <ffffffff81418b9d>
=> sys_connect+0x98/0xd6 <ffffffff813b0437>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107025>
=> do_huge_pmd_wp_page+0x15a/0x637 <ffffffff81114a25>
=> handle_mm_fault+0x169/0x1ce <ffffffff810f176a>
=> do_page_fault+0x358/0x37a <ffffffff814741ae>
=> page_fault+0x25/0x30 <ffffffff81471455>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> find_or_create_page+0x41/0x8a <ffffffff810d5b54>
=> __getblk+0x137/0x23a <ffffffff81141b55>
=> __breadahead+0x12/0x39 <ffffffff81141c6a>
=> __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b>
=> ext4_iget+0x58/0x64c <ffffffff81192a48>
=> ext4_lookup+0x93/0xf3 <ffffffff8119a79b>
=> d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b>
=> do_lookup+0x23a/0x287 <ffffffff81127dde>
=> link_path_walk+0x239/0x3b8 <ffffffff81128469>
=> do_path_lookup+0x4d/0xf6 <ffffffff8112881b>
=> user_path_at+0x57/0x94 <ffffffff811294e0>
=> sys_faccessat+0xa0/0x162 <ffffffff8111d2e2>
=> sys_access+0x18/0x1a <ffffffff8111d3bc>
2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> d_alloc+0x26/0x18d <ffffffff8112e4e9>
=> shmem_file_setup+0xb6/0x160 <ffffffff810e70d1>
=> drm_gem_object_init+0x34/0x61 [drm] <ffffffffa0024ffc>
=> i915_gem_alloc_object+0x41/0x111 [i915] <ffffffffa007e4cd>
=> i915_gem_create_ioctl+0x3f/0xcf [i915] <ffffffffa007e5dc>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
=> sys_ioctl+0x56/0x7b <ffffffff8112c107>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
416 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
=> block_write_full_page_endio+0x8a/0x97 <ffffffff81143671>
=> block_write_full_page+0x15/0x17 <ffffffff81143693>
=> blkdev_writepage+0x18/0x1a <ffffffff811460b8>
=> __writepage+0x15/0x2e <ffffffff810dca43>
13 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> mpage_readpages+0x99/0x104 <ffffffff81148dae>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
=> filemap_fault+0x17b/0x36c <ffffffff810d6ade>
=> __do_fault+0x55/0x377 <ffffffff810ee88b>
=> handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f47>
58 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d85>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> do_mpage_readpage+0x34a/0x468 <ffffffff81148b8d>
=> mpage_readpages+0xbe/0x104 <ffffffff81148dd3>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> d_alloc+0x26/0x18d <ffffffff8112e4e9>
=> d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32>
=> __lookup_hash+0x7f/0x88 <ffffffff81127eaa>
=> lookup_hash+0x19/0x1b <ffffffff81127ecc>
=> do_last+0xcf/0x2df <ffffffff811298fc>
=> do_filp_open+0x2ac/0x60a <ffffffff81129db8>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_open+0x20/0x22 <ffffffff8111da0d>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
7727 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> submit_bh+0xe6/0x105 <ffffffff81140a6f>
=> __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> __split_vma+0x6c/0x21b <ffffffff810f50bd>
=> do_munmap+0x194/0x30b <ffffffff810f57d1>
=> mmap_region+0x9b/0x446 <ffffffff810f645d>
=> do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6aa0>
=> sys_mmap_pgoff+0x115/0x164 <ffffffff810f6c0f>
=> sys_mmap+0x22/0x24 <ffffffff8100da68>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d6>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> kernel_thread+0x75/0x77 <ffffffff81010bb6>
=> __call_usermodehelper+0x43/0x76 <ffffffff81069132>
=> process_one_work+0x186/0x298 <ffffffff8106b3ca>
=> worker_thread+0xda/0x15d <ffffffff8106bf29>
=> kthread+0x84/0x8c <ffffffff8106f2df>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
18 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d85>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> mpage_readpages+0xf0/0x104 <ffffffff81148e05>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc>
=> alloc_inode+0x1d/0x78 <ffffffff81131809>
=> iget_locked+0x61/0xdd <ffffffff811323be>
=> ext4_iget+0x2c/0x64c <ffffffff81192a1c>
=> ext4_lookup+0x93/0xf3 <ffffffff8119a79b>
=> d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b>
=> do_lookup+0x23a/0x287 <ffffffff81127dde>
=> link_path_walk+0x14b/0x3b8 <ffffffff8112837b>
=> do_filp_open+0x1eb/0x60a <ffffffff81129cf7>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
1 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> find_or_create_page+0x5a/0x8a <ffffffff810d5b6d>
=> ext4_mb_load_buddy+0x1f7/0x30f <ffffffff811b4bdf>
=> ext4_free_blocks+0x41d/0x6ba <ffffffff811b7a92>
=> ext4_ext_truncate+0x361/0x78a <ffffffff811b1012>
=> ext4_truncate+0xa1/0x4d9 <ffffffff8119598c>
=> ext4_evict_inode+0x13f/0x228 <ffffffff811978e9>
=> evict+0x21/0x89 <ffffffff81131b92>
149 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> mpage_readpages+0x99/0x104 <ffffffff81148dae>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
=> ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7e>
=> page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddd08>
=> generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6514>
High-order normal allocations: 152966
High-order atomic allocations: 875
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-10 17:05 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-10 17:05 UTC (permalink / raw)
To: Mel Gorman
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Tue, 2011-05-10 at 15:57 +0000, James Bottomley wrote:
> On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote:
> > On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
> > > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> > > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> > > > > I really would like to hear if the fix makes a big difference or
> > > > > if we need to consider forcing SLUB high-order allocations bailing
> > > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> > > > > allocate_slab). Even with the fix applied, kswapd might be waking up
> > > > > less but processes will still be getting stalled in direct compaction
> > > > > and direct reclaim so it would still be jittery.
> > > >
> > > > "the fix" being this
> > > >
> > > > https://lkml.org/lkml/2011/3/5/121
> > > >
> > >
> > > Drop this for the moment. It was a long shot at best and there is little
> > > evidence the problem is in this area.
> > >
> > > I'm attaching two patches. The first is the NO_KSWAPD one to stop
> > > kswapd being woken up by SLUB using speculative high-orders. The second
> > > one is more drastic and prevents slub entering direct reclaim or
> > > compaction. It applies on top of patch 1. These are both untested and
> > > afraid are a bit rushed as well :(
> >
> > Preliminary results with both patches applied still show kswapd
> > periodically going up to 99% but it doesn't stay there, it comes back
> > down again (and, obviously, the system doesn't hang).
>
> This is a second run with the watch highorders.
>
> At the end of the run, the system hung temporarily and now comes back
> with CPU3 spinning in all system time at kswapd shrink_slab
Here's a trace in the same situation with the ftrace stack entries
bumped to 16 as requested on IRC. There was no hang for this one.
James
---
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> shmem_alloc_inode+0x1a/0x2f <ffffffff810e58c4>
=> alloc_inode+0x1d/0x78 <ffffffff81131809>
=> new_inode+0x1b/0x4b <ffffffff8113187f>
=> shmem_get_inode+0x45/0x1f6 <ffffffff810e49b3>
=> shmem_file_setup+0xe4/0x160 <ffffffff810e70ff>
=> drm_gem_object_init+0x34/0x61 [drm] <ffffffffa0024ffc>
=> i915_gem_alloc_object+0x41/0x111 [i915] <ffffffffa007e4cd>
=> i915_gem_create_ioctl+0x3f/0xcf [i915] <ffffffffa007e5dc>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
701 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> scsi_sg_alloc+0x2d/0x2f <ffffffff812f9e90>
=> __sg_alloc_table+0x63/0x11c <ffffffff81230ddf>
=> scsi_alloc_sgtable+0x2a/0x4f <ffffffff812f9ebc>
=> scsi_init_sgtable+0x21/0x61 <ffffffff812f9f02>
=> scsi_init_io+0x32/0x13b <ffffffff812f9f74>
=> scsi_setup_fs_cmnd+0xb9/0xbe <ffffffff812fa1b0>
=> sd_prep_fn+0x27b/0xb6d <ffffffff813026e9>
=> blk_peek_request+0xe1/0x1b6 <ffffffff81213411>
189 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> get_request+0x180/0x262 <ffffffff812124f6>
=> get_request_wait+0x35/0x193 <ffffffff81212f71>
=> __make_request+0x2b4/0x400 <ffffffff81213898>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> submit_bh+0xe6/0x105 <ffffffff81140a6f>
=> __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
=> block_write_full_page_endio+0x8a/0x97 <ffffffff81143671>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> inet_twsk_alloc+0x31/0x11e <ffffffff813f8a36>
=> tcp_time_wait+0xb9/0x29a <ffffffff8140e26f>
=> tcp_fin+0x7b/0x10f <ffffffff8140191a>
=> tcp_data_queue+0x1e4/0x876 <ffffffff8140207f>
=> tcp_rcv_state_process+0x880/0x8cf <ffffffff81404f67>
=> tcp_v4_do_rcv+0x312/0x363 <ffffffff8140b698>
=> tcp_v4_rcv+0x434/0x69b <ffffffff8140d209>
=> ip_local_deliver_finish+0x13d/0x1ae <ffffffff813f104d>
=> NF_HOOK.constprop.3+0x51/0x58 <ffffffff813f0eb2>
=> ip_local_deliver+0x4f/0x54 <ffffffff813f1205>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333>
=> ext4_iget+0x58/0x64c <ffffffff81192a48>
=> ext4_lookup+0x93/0xf3 <ffffffff8119a79b>
=> d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b>
=> __lookup_hash+0x7f/0x88 <ffffffff81127eaa>
9 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e1bd>
=> __alloc_skb+0x40/0x133 <ffffffff813b5e17>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_aio_write+0xb9/0xc9 <ffffffff813af85e>
=> do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d>
=> do_readv_writev+0xa7/0x11e <ffffffff8111ea03>
=> vfs_writev+0x45/0x47 <ffffffff8111eabf>
=> sys_writev+0x4a/0x93 <ffffffff8111ebe2>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> sk_prot_alloc+0x37/0x13a <ffffffff813b27f4>
=> sk_alloc+0x2c/0x8e <ffffffff813b2955>
=> unix_create1+0x4e/0x14d <ffffffff8143cae7>
=> unix_stream_connect+0x98/0x431 <ffffffff8143ecc6>
=> sys_connect+0x98/0xd6 <ffffffff813b0437>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
5 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> alloc_file+0x1e/0xbf <ffffffff8111f3da>
=> create_read_pipe+0x2b/0x5b <ffffffff81125ffe>
=> do_pipe_flags+0x4e/0xf4 <ffffffff8112607c>
=> sys_pipe2+0x1c/0x5c <ffffffff8112613e>
=> sys_pipe+0x10/0x12 <ffffffff8112618e>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
6 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> __netdev_alloc_skb+0x1f/0x3b <ffffffff813b5f6a>
=> e1000_alloc_rx_buffers+0x8b/0x14e [e1000e] <ffffffffa01bfb58>
=> e1000_clean_rx_irq+0x2a6/0x2df [e1000e] <ffffffffa01c05a9>
=> e1000_clean+0x73/0x228 [e1000e] <ffffffffa01c0a07>
=> net_rx_action+0xa9/0x1ac <ffffffff813bfea2>
=> __do_softirq+0xd2/0x19d <ffffffff8105abe0>
=> call_softirq+0x1c/0x30 <ffffffff8100aa9c>
=> do_softirq+0x46/0x81 <ffffffff8100c0c1>
=> irq_exit+0x49/0x8b <ffffffff8105ad64>
48 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5947>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d540c>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d6181>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6209>
=> ext4_file_write+0x1dc/0x234 <ffffffff8118e898>
=> do_sync_write+0xbf/0xff <ffffffff8111dd6f>
=> vfs_write+0xac/0xf3 <ffffffff8111e3e5>
=> sys_write+0x4a/0x6e <ffffffff8111e5d4>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76f>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
=> filemap_fault+0x17b/0x36c <ffffffff810d6ade>
=> __do_fault+0x55/0x377 <ffffffff810ee88b>
=> handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f47>
=> handle_mm_fault+0x1bb/0x1ce <ffffffff810f17bc>
=> do_page_fault+0x358/0x37a <ffffffff814741ae>
=> page_fault+0x25/0x30 <ffffffff81471455>
16 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> do_read_cache_page+0x63/0x127 <ffffffff810d59f4>
=> read_cache_page_gfp+0x1e/0x28 <ffffffff810d5ad6>
=> i915_gem_object_bind_to_gtt+0x249/0x45b [i915] <ffffffffa007b0ad>
=> i915_gem_object_pin+0xc7/0x13c [i915] <ffffffffa007d609>
=> i915_gem_execbuffer_reserve+0x1b1/0x2fc [i915] <ffffffffa007f76d>
=> i915_gem_do_execbuffer+0x45b/0x1116 [i915] <ffffffffa008005a>
=> i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe>
409 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> jbd2_journal_commit_transaction+0x6e4/0x1052 <ffffffff811c56a6>
=> kjournald2+0xc9/0x20a <ffffffff811c9764>
=> kthread+0x84/0x8c <ffffffff8106f2df>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
4 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> mpage_readpages+0xf0/0x104 <ffffffff81148e05>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> copy_vma+0xe3/0x1e9 <ffffffff810f5cd3>
=> move_vma+0x8a/0x1fb <ffffffff810f7edb>
=> do_mremap+0x40e/0x46a <ffffffff810f845a>
=> sys_mremap+0x54/0x76 <ffffffff810f850a>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
5 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> do_read_cache_page+0x63/0x127 <ffffffff810d59f4>
=> read_cache_page_gfp+0x1e/0x28 <ffffffff810d5ad6>
=> i915_gem_object_bind_to_gtt+0x249/0x45b [i915] <ffffffffa007b0ad>
=> i915_gem_fault+0x85/0x18e [i915] <ffffffffa007d287>
=> __do_fault+0x55/0x377 <ffffffff810ee88b>
=> handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f47>
=> handle_mm_fault+0x1bb/0x1ce <ffffffff810f17bc>
1 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfdf>
=> kzalloc.constprop.1+0x13/0x15 [i915] <ffffffffa007fbfd>
=> i915_gem_do_execbuffer+0x306/0x1116 [i915] <ffffffffa007ff05>
=> i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
=> sys_ioctl+0x56/0x7b <ffffffff8112c107>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
5 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> copy_process+0xc6/0x10c5 <ffffffff810533b9>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> sys_clone+0x28/0x2a <ffffffff8101150e>
=> stub_clone+0x13/0x20 <ffffffff81009ea3>
4 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> ext4_mb_init_cache+0x1e6/0x52b <ffffffff811b45c8>
=> ext4_mb_load_buddy+0x134/0x30f <ffffffff811b4b1c>
=> ext4_mb_regular_allocator+0x13c/0x275 <ffffffff811b6637>
=> ext4_mb_new_blocks+0x17f/0x3c2 <ffffffff811b7432>
=> ext4_ext_map_blocks+0x192b/0x1b5b <ffffffff811b0a81>
102 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> ll_rw_block+0x84/0xa3 <ffffffff811410f4>
=> __breadahead+0x2e/0x39 <ffffffff81141c86>
=> __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b>
=> ext4_iget+0x58/0x64c <ffffffff81192a48>
=> ext4_lookup+0x93/0xf3 <ffffffff8119a79b>
1 instances order=2 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> get_request+0x180/0x262 <ffffffff812124f6>
=> get_request_wait+0x35/0x193 <ffffffff81212f71>
=> __make_request+0x2b4/0x400 <ffffffff81213898>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> mpage_readpages+0xf0/0x104 <ffffffff81148e05>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
3 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> __split_vma+0x6c/0x21b <ffffffff810f50bd>
=> split_vma+0x20/0x22 <ffffffff810f59ca>
=> mprotect_fixup+0x105/0x65b <ffffffff810f70ad>
=> sys_mprotect+0x197/0x209 <ffffffff810f779a>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
14 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063>
=> scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9>
=> __scsi_get_command+0x1b/0x96 <ffffffff812f3150>
=> scsi_get_command+0x3d/0xa5 <ffffffff812f3283>
=> scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165>
=> sd_prep_fn+0x27b/0xb6d <ffffffff813026e9>
=> blk_peek_request+0xe1/0x1b6 <ffffffff81213411>
=> scsi_request_fn+0x82/0x421 <ffffffff812f9a5f>
=> __generic_unplug_device+0x34/0x38 <ffffffff81212f38>
=> generic_unplug_device+0x2d/0x3d <ffffffff812131f0>
7 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> __split_vma+0x6c/0x21b <ffffffff810f50bd>
=> split_vma+0x20/0x22 <ffffffff810f59ca>
=> mprotect_fixup+0x125/0x65b <ffffffff810f70cd>
=> sys_mprotect+0x197/0x209 <ffffffff810f779a>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> find_or_create_page+0x5a/0x8a <ffffffff810d5b6d>
=> __getblk+0x137/0x23a <ffffffff81141b55>
=> ext4_getblk+0x8e/0x153 <ffffffff811957c0>
=> ext4_bread+0x16/0x66 <ffffffff8119589b>
=> ext4_mkdir+0x159/0x33e <ffffffff8119b786>
=> vfs_mkdir+0x5f/0x9b <ffffffff81128b01>
=> sys_mkdirat+0x97/0xe8 <ffffffff8112a3dd>
4 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_sendmsg+0xa1/0xb6 <ffffffff813af90f>
=> sys_sendmsg+0x237/0x2b5 <ffffffff813b0cd9>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
2 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_aio_write+0xb9/0xc9 <ffffffff813af85e>
=> do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d>
=> do_readv_writev+0xa7/0x11e <ffffffff8111ea03>
=> vfs_writev+0x45/0x47 <ffffffff8111eabf>
=> sys_writev+0x4a/0x93 <ffffffff8111ebe2>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> ext4_mb_init_cache+0x1e6/0x52b <ffffffff811b45c8>
=> ext4_mb_init_group+0x100/0x281 <ffffffff8146b9f8>
=> ext4_mb_good_group+0x58/0xdb <ffffffff811b4965>
=> ext4_mb_regular_allocator+0x125/0x275 <ffffffff811b6620>
=> ext4_mb_new_blocks+0x17f/0x3c2 <ffffffff811b7432>
39 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> ll_rw_block+0x84/0xa3 <ffffffff811410f4>
=> __breadahead+0x2e/0x39 <ffffffff81141c86>
=> __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b>
=> ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936>
=> ext4_reserve_inode_write+0x26/0x83 <ffffffff81193de1>
66 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> scsi_sg_alloc+0x2d/0x2f <ffffffff812f9e90>
=> __sg_alloc_table+0x63/0x11c <ffffffff81230ddf>
=> scsi_alloc_sgtable+0x2a/0x4f <ffffffff812f9ebc>
=> scsi_init_sgtable+0x21/0x61 <ffffffff812f9f02>
=> scsi_init_io+0x32/0x13b <ffffffff812f9f74>
=> scsi_setup_fs_cmnd+0xb9/0xbe <ffffffff812fa1b0>
=> sd_prep_fn+0x27b/0xb6d <ffffffff813026e9>
=> blk_peek_request+0xe1/0x1b6 <ffffffff81213411>
2 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> do_read_cache_page+0x48/0x127 <ffffffff810d59d9>
=> read_cache_page_gfp+0x1e/0x28 <ffffffff810d5ad6>
=> i915_gem_object_bind_to_gtt+0x249/0x45b [i915] <ffffffffa007b0ad>
=> i915_gem_object_pin+0xc7/0x13c [i915] <ffffffffa007d609>
=> i915_gem_execbuffer_reserve+0x1b1/0x2fc [i915] <ffffffffa007f76d>
=> i915_gem_do_execbuffer+0x45b/0x1116 [i915] <ffffffffa008005a>
=> i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
=> sys_ioctl+0x56/0x7b <ffffffff8112c107>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
148 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107025>
=> khugepaged+0x543/0xf2d <ffffffff81113ce5>
=> kthread+0x84/0x8c <ffffffff8106f2df>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
526 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc>
=> alloc_inode+0x1d/0x78 <ffffffff81131809>
=> new_inode+0x1b/0x4b <ffffffff8113187f>
=> ext4_new_inode+0xa1/0xcf2 <ffffffff8118f9cb>
=> ext4_create+0xbc/0x13e <ffffffff8119b32e>
=> vfs_create+0x6c/0x8e <ffffffff81128c64>
=> do_last+0x166/0x2df <ffffffff81129993>
=> do_filp_open+0x2ac/0x60a <ffffffff81129db8>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_openat+0x11/0x13 <ffffffff8111da20>
17 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063>
=> scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9>
=> __scsi_get_command+0x1b/0x96 <ffffffff812f3150>
=> scsi_get_command+0x3d/0xa5 <ffffffff812f3283>
=> scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165>
=> sd_prep_fn+0x27b/0xb6d <ffffffff813026e9>
=> blk_peek_request+0xe1/0x1b6 <ffffffff81213411>
=> scsi_request_fn+0x82/0x421 <ffffffff812f9a5f>
=> __generic_unplug_device+0x34/0x38 <ffffffff81212f38>
=> __make_request+0x3a3/0x400 <ffffffff81213987>
43 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> ll_rw_block+0x84/0xa3 <ffffffff811410f4>
=> __breadahead+0x2e/0x39 <ffffffff81141c86>
=> __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b>
=> ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936>
=> ext4_xattr_get+0x57/0x213 <ffffffff811baf2b>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfdf>
=> drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2>
=> i915_gem_execbuffer2+0x4f/0x12e [i915] <ffffffffa0080f81>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
=> sys_ioctl+0x56/0x7b <ffffffff8112c107>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
6 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mmap_region+0x1b7/0x446 <ffffffff810f6579>
=> do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6aa0>
=> sys_mmap_pgoff+0x115/0x164 <ffffffff810f6c0f>
=> sys_mmap+0x22/0x24 <ffffffff8100da68>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
763 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> do_filp_open+0xe7/0x60a <ffffffff81129bf3>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_open+0x20/0x22 <ffffffff8111da0d>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
59 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc>
=> alloc_inode+0x1d/0x78 <ffffffff81131809>
=> new_inode+0x1b/0x4b <ffffffff8113187f>
=> ext4_new_inode+0xa1/0xcf2 <ffffffff8118f9cb>
=> ext4_mkdir+0x10b/0x33e <ffffffff8119b738>
=> vfs_mkdir+0x5f/0x9b <ffffffff81128b01>
=> sys_mkdirat+0x97/0xe8 <ffffffff8112a3dd>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
83 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> dup_mm+0x1f8/0x486 <ffffffff81053039>
=> copy_process+0x917/0x10c5 <ffffffff81053c0a>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> sys_clone+0x28/0x2a <ffffffff8101150e>
=> stub_clone+0x13/0x20 <ffffffff81009ea3>
4 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d85>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> mpage_alloc+0x2a/0x84 <ffffffff8114866b>
=> do_mpage_readpage+0x38c/0x468 <ffffffff81148bcf>
=> mpage_readpages+0xbe/0x104 <ffffffff81148dd3>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
=> ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7e>
33 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063>
=> scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9>
=> __scsi_get_command+0x1b/0x96 <ffffffff812f3150>
=> scsi_get_command+0x3d/0xa5 <ffffffff812f3283>
=> scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165>
=> sd_prep_fn+0x27b/0xb6d <ffffffff813026e9>
=> blk_peek_request+0xe1/0x1b6 <ffffffff81213411>
=> scsi_request_fn+0x82/0x421 <ffffffff812f9a5f>
=> __generic_unplug_device+0x34/0x38 <ffffffff81212f38>
=> elv_insert+0x172/0x17b <ffffffff8120dae8>
1 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107025>
=> do_wp_page+0x348/0x560 <ffffffff810ef735>
=> handle_pte_fault+0x73b/0x7a5 <ffffffff810f13db>
=> handle_mm_fault+0x1bb/0x1ce <ffffffff810f17bc>
=> do_page_fault+0x358/0x37a <ffffffff814741ae>
=> page_fault+0x25/0x30 <ffffffff81471455>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> d_alloc+0x26/0x18d <ffffffff8112e4e9>
=> d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32>
=> do_lookup+0x23a/0x287 <ffffffff81127dde>
=> link_path_walk+0x239/0x3b8 <ffffffff81128469>
=> do_path_lookup+0x4d/0xf6 <ffffffff8112881b>
=> do_filp_open+0x129/0x60a <ffffffff81129c35>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_open+0x20/0x22 <ffffffff8111da0d>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333>
=> ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936>
=> ext4_xattr_get+0x57/0x213 <ffffffff811baf2b>
=> ext4_xattr_security_get+0x2a/0x2c <ffffffff811bc92d>
=> generic_getxattr+0x57/0x60 <ffffffff81137e27>
1 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfdf>
=> kcalloc+0x38/0x3a <ffffffff811f5ff2>
=> security_get_user_sids+0x1c2/0x369 <ffffffff811f8503>
=> sel_write_user+0xd6/0x19d <ffffffff811ef5b3>
=> selinux_transaction_write+0x5b/0x77 <ffffffff811edbfc>
=> vfs_write+0xac/0xf3 <ffffffff8111e3e5>
=> sys_write+0x4a/0x6e <ffffffff8111e5d4>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
15 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> proc_alloc_inode+0x20/0x91 <ffffffff811682c0>
=> alloc_inode+0x1d/0x78 <ffffffff81131809>
=> new_inode+0x1b/0x4b <ffffffff8113187f>
=> proc_pid_make_inode+0x14/0xb7 <ffffffff811697c0>
=> proc_pident_instantiate+0x22/0xb1 <ffffffff8116c202>
=> proc_pident_lookup+0x84/0xa3 <ffffffff8116c3ce>
=> proc_tgid_base_lookup+0x1a/0x1c <ffffffff8116c43f>
=> d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b>
=> do_lookup+0x23a/0x287 <ffffffff81127dde>
=> link_path_walk+0x239/0x3b8 <ffffffff81128469>
213 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d6>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> sys_clone+0x28/0x2a <ffffffff8101150e>
=> stub_clone+0x13/0x20 <ffffffff81009ea3>
140162 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
=> block_write_full_page_endio+0x8a/0x97 <ffffffff81143671>
=> block_write_full_page+0x15/0x17 <ffffffff81143693>
=> mpage_da_submit_io+0x31a/0x395 <ffffffff811935d8>
=> mpage_da_map_and_submit+0x2ca/0x2e0 <ffffffff81196e88>
2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> d_alloc+0x26/0x18d <ffffffff8112e4e9>
=> d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32>
=> __lookup_hash+0x7f/0x88 <ffffffff81127eaa>
=> lookup_hash+0x19/0x1b <ffffffff81127ecc>
=> lookup_create+0x4e/0x94 <ffffffff81127f1c>
=> sys_mkdirat+0x43/0xe8 <ffffffff8112a389>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
53 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> submit_bh+0xe6/0x105 <ffffffff81140a6f>
=> jbd2_journal_commit_transaction+0x6e4/0x1052 <ffffffff811c56a6>
31 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> scsi_pool_alloc_command+0x29/0x68 <ffffffff812f3063>
=> scsi_host_alloc_command+0x1f/0x6b <ffffffff812f30e9>
=> __scsi_get_command+0x1b/0x96 <ffffffff812f3150>
=> scsi_get_command+0x3d/0xa5 <ffffffff812f3283>
=> scsi_setup_fs_cmnd+0x6e/0xbe <ffffffff812fa165>
=> sd_prep_fn+0x27b/0xb6d <ffffffff813026e9>
=> blk_peek_request+0xe1/0x1b6 <ffffffff81213411>
=> scsi_request_fn+0x82/0x421 <ffffffff812f9a5f>
=> __blk_run_queue+0x4a/0x7a <ffffffff81212b10>
=> blk_run_queue+0x28/0x3c <ffffffff81212bf9>
1 instances order=3 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5947>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d540c>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d6181>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6209>
=> ext4_file_write+0x1dc/0x234 <ffffffff8118e898>
=> do_sync_write+0xbf/0xff <ffffffff8111dd6f>
=> vfs_write+0xac/0xf3 <ffffffff8111e3e5>
=> sys_write+0x4a/0x6e <ffffffff8111e5d4>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
8 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc_node_track_caller+0xcf/0x131 <ffffffff8110ffca>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_aio_write+0xb9/0xc9 <ffffffff813af85e>
=> do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d>
=> do_readv_writev+0xa7/0x11e <ffffffff8111ea03>
=> vfs_writev+0x45/0x47 <ffffffff8111eabf>
=> sys_writev+0x4a/0x93 <ffffffff8111ebe2>
2 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d6>
=> __pollwait+0x5e/0xcc <ffffffff8112ca0b>
=> sock_poll_wait+0x18/0x1d <ffffffff8143cc70>
=> unix_poll+0x1c/0x9a <ffffffff8143cc91>
=> sock_poll+0x1a/0x1c <ffffffff813ad43a>
=> do_sys_poll+0x1fa/0x386 <ffffffff8112d781>
=> sys_poll+0x51/0xbb <ffffffff8112d9b0>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_REPEAT|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc_node+0x93/0x12d <ffffffff8110e1bd>
=> __alloc_skb+0x40/0x133 <ffffffff813b5e17>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_sendmsg+0xa1/0xb6 <ffffffff813af90f>
=> sys_sendmsg+0x237/0x2b5 <ffffffff813b0cd9>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
3 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> alloc_file+0x1e/0xbf <ffffffff8111f3da>
=> create_write_pipe+0x151/0x1bf <ffffffff81125f35>
=> do_pipe_flags+0x32/0xf4 <ffffffff81126060>
=> sys_pipe2+0x1c/0x5c <ffffffff8112613e>
=> sys_pipe+0x10/0x12 <ffffffff8112618e>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> dup_fd+0x33/0x25f <ffffffff81133573>
=> copy_process+0x5b8/0x10c5 <ffffffff810538ab>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> sys_clone+0x28/0x2a <ffffffff8101150e>
=> stub_clone+0x13/0x20 <ffffffff81009ea3>
2 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333>
=> ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936>
=> ext4_xattr_set_handle+0xca/0x3f0 <ffffffff811bb1b1>
=> ext4_init_security+0x61/0x82 <ffffffff811bc9d3>
=> ext4_new_inode+0xb69/0xcf2 <ffffffff81190493>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> skb_clone+0x50/0x70 <ffffffff813b5db7>
=> dev_hard_start_xmit+0xd1/0x535 <ffffffff813c1660>
=> sch_direct_xmit+0x6b/0x147 <ffffffff813d7173>
=> dev_queue_xmit+0x2ac/0x461 <ffffffff813c1d70>
=> neigh_resolve_output+0x205/0x236 <ffffffff813ca2be>
=> ip_finish_output2+0x1cc/0x20f <ffffffff813f3979>
=> ip_finish_output+0x65/0x6a <ffffffff813f42a9>
=> ip_output+0x8c/0x91 <ffffffff813f49f2>
=> dst_output+0x14/0x16 <ffffffff813f34d7>
=> ip_local_out+0x20/0x24 <ffffffff813f435f>
3 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d6>
=> __pollwait+0x5e/0xcc <ffffffff8112ca0b>
=> sock_poll_wait+0x18/0x1d <ffffffff8143cc70>
=> unix_poll+0x1c/0x9a <ffffffff8143cc91>
=> sock_poll+0x1a/0x1c <ffffffff813ad43a>
=> do_select+0x2fb/0x4f5 <ffffffff8112cf37>
=> core_sys_select+0x163/0x202 <ffffffff8112d294>
=> sys_select+0x91/0xb9 <ffffffff8112d3c4>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
2 instances order=3 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d85>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> mpage_alloc+0x2a/0x84 <ffffffff8114866b>
=> do_mpage_readpage+0x38c/0x468 <ffffffff81148bcf>
=> mpage_readpages+0xbe/0x104 <ffffffff81148dd3>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
=> filemap_fault+0x17b/0x36c <ffffffff810d6ade>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> do_filp_open+0xe7/0x60a <ffffffff81129bf3>
=> open_exec+0x26/0x82 <ffffffff8112265e>
=> load_elf_binary+0x1df/0x175f <ffffffff8115c1a8>
=> search_binary_handler+0xb6/0x22e <ffffffff81122ce3>
=> do_execve+0x17e/0x277 <ffffffff81123ea0>
=> sys_execve+0x43/0x5a <ffffffff81011553>
=> stub_execve+0x6c/0xc0 <ffffffff81009fdc>
210 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> grab_cache_page_write_begin+0x6d/0x9e <ffffffff810d5960>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d540c>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d6181>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6209>
=> ext4_file_write+0x1dc/0x234 <ffffffff8118e898>
=> do_sync_write+0xbf/0xff <ffffffff8111dd6f>
6 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> do_filp_open+0xe7/0x60a <ffffffff81129bf3>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_openat+0x11/0x13 <ffffffff8111da20>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
8 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107025>
=> do_huge_pmd_anonymous_page+0xbf/0x261 <ffffffff81115b8e>
=> handle_mm_fault+0x113/0x1ce <ffffffff810f1714>
=> do_page_fault+0x358/0x37a <ffffffff814741ae>
=> page_fault+0x25/0x30 <ffffffff81471455>
4 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> submit_bh+0xe6/0x105 <ffffffff81140a6f>
=> ll_rw_block+0x84/0xa3 <ffffffff811410f4>
5 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __ext4_get_inode_loc+0x2ec/0x344 <ffffffff81191333>
=> ext4_iget+0x58/0x64c <ffffffff81192a48>
=> ext4_lookup+0x93/0xf3 <ffffffff8119a79b>
=> d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b>
=> do_lookup+0x23a/0x287 <ffffffff81127dde>
257 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
=> block_write_full_page_endio+0x8a/0x97 <ffffffff81143671>
=> block_write_full_page+0x15/0x17 <ffffffff81143693>
=> ext4_writepage+0x201/0x210 <ffffffff81193854>
=> shrink_page_list+0x3c7/0x6de <ffffffff810e22a1>
809 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> kmalloc_large_node+0x56/0x95 <ffffffff8146a57d>
=> __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff2c>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_aio_write+0xb9/0xc9 <ffffffff813af85e>
=> do_sync_readv_writev+0xb9/0xfe <ffffffff8111e79d>
=> do_readv_writev+0xa7/0x11e <ffffffff8111ea03>
=> vfs_writev+0x45/0x47 <ffffffff8111eabf>
=> sys_writev+0x4a/0x93 <ffffffff8111ebe2>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
2 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc>
=> alloc_inode+0x1d/0x78 <ffffffff81131809>
=> iget_locked+0x61/0xdd <ffffffff811323be>
=> ext4_iget+0x2c/0x64c <ffffffff81192a1c>
=> ext4_lookup+0x93/0xf3 <ffffffff8119a79b>
=> d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b>
=> do_lookup+0x23a/0x287 <ffffffff81127dde>
=> link_path_walk+0x239/0x3b8 <ffffffff81128469>
=> do_path_lookup+0x4d/0xf6 <ffffffff8112881b>
=> do_filp_open+0x129/0x60a <ffffffff81129c35>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> alloc_file+0x1e/0xbf <ffffffff8111f3da>
=> shmem_file_setup+0x125/0x160 <ffffffff810e7140>
=> drm_gem_object_init+0x34/0x61 [drm] <ffffffffa0024ffc>
=> i915_gem_alloc_object+0x41/0x111 [i915] <ffffffffa007e4cd>
=> i915_gem_create_ioctl+0x3f/0xcf [i915] <ffffffffa007e5dc>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
=> sys_ioctl+0x56/0x7b <ffffffff8112c107>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
99 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> mpage_alloc+0x2a/0x84 <ffffffff8114866b>
=> do_mpage_readpage+0x38c/0x468 <ffffffff81148bcf>
=> mpage_readpages+0xbe/0x104 <ffffffff81148dd3>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
24 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> __do_page_cache_readahead+0x9b/0x177 <ffffffff810dd76f>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
=> ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7e>
=> page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddd08>
=> generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6514>
=> do_sync_read+0xbf/0xff <ffffffff8111de6e>
=> vfs_read+0xa9/0xf0 <ffffffff8111e4d5>
=> sys_read+0x4a/0x6e <ffffffff8111e566>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
21 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> prepare_creds+0x26/0xae <ffffffff81074d4b>
=> sys_faccessat+0x37/0x162 <ffffffff8111d279>
=> sys_access+0x18/0x1a <ffffffff8111d3bc>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_COMP|GFP_ZERO
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d6>
=> kmalloc_order_trace+0x2c/0x5e <ffffffff8110c73c>
=> __kmalloc+0x37/0x10d <ffffffff8110df65>
=> kzalloc.constprop.19+0xe/0x10 <ffffffff810b8d8c>
=> tracing_open_pipe+0x40/0x147 <ffffffff810b907a>
=> __dentry_open+0x161/0x283 <ffffffff8111c980>
=> nameidata_to_filp+0x60/0x67 <ffffffff8111d8f4>
=> finish_open+0xa1/0x17f <ffffffff8112974f>
=> do_filp_open+0x186/0x60a <ffffffff81129c92>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_open+0x20/0x22 <ffffffff8111da0d>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> do_mpage_readpage+0x34a/0x468 <ffffffff81148b8d>
=> mpage_readpages+0xbe/0x104 <ffffffff81148dd3>
2 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> find_or_create_page+0x5a/0x8a <ffffffff810d5b6d>
=> __getblk+0x137/0x23a <ffffffff81141b55>
=> __ext4_get_inode_loc+0x118/0x344 <ffffffff8119115f>
=> ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936>
=> ext4_xattr_set_handle+0xca/0x3f0 <ffffffff811bb1b1>
=> ext4_init_security+0x61/0x82 <ffffffff811bc9d3>
=> ext4_new_inode+0xb69/0xcf2 <ffffffff81190493>
8 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> ll_rw_block+0x84/0xa3 <ffffffff811410f4>
=> __breadahead+0x2e/0x39 <ffffffff81141c86>
=> __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b>
=> ext4_get_inode_loc+0x2e/0x33 <ffffffff81192936>
=> ext4_xattr_set_handle+0xca/0x3f0 <ffffffff811bb1b1>
58 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> d_alloc+0x26/0x18d <ffffffff8112e4e9>
=> d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32>
=> __lookup_hash+0x7f/0x88 <ffffffff81127eaa>
=> lookup_hash+0x19/0x1b <ffffffff81127ecc>
=> do_last+0xcf/0x2df <ffffffff811298fc>
=> do_filp_open+0x2ac/0x60a <ffffffff81129db8>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_openat+0x11/0x13 <ffffffff8111da20>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
5 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> mpage_readpages+0x99/0x104 <ffffffff81148dae>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
=> ondemand_readahead+0x15a/0x1d8 <ffffffff810ddc0f>
=> page_cache_sync_readahead+0x3d/0x40 <ffffffff810ddd6d>
=> generic_file_aio_read+0x27d/0x5e0 <ffffffff810d64d4>
=> do_sync_read+0xbf/0xff <ffffffff8111de6e>
2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> get_empty_filp+0x7a/0x141 <ffffffff8111f2f5>
=> alloc_file+0x1e/0xbf <ffffffff8111f3da>
=> create_read_pipe+0x2b/0x5b <ffffffff81125ffe>
=> do_pipe_flags+0x4e/0xf4 <ffffffff8112607c>
=> sys_pipe2+0x1c/0x5c <ffffffff8112613e>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
9 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d6>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> kernel_thread+0x75/0x77 <ffffffff81010bb6>
=> kthreadd+0xe7/0x124 <ffffffff8106f61f>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> grab_cache_page_write_begin+0x54/0x9e <ffffffff810d5947>
=> ext4_da_write_begin+0x13f/0x20e <ffffffff8119639a>
=> generic_file_buffered_write+0x109/0x23a <ffffffff810d540c>
=> __generic_file_aio_write+0x242/0x272 <ffffffff810d6181>
=> generic_file_aio_write+0x58/0xa6 <ffffffff810d6209>
=> ext4_file_write+0x1dc/0x234 <ffffffff8118e898>
=> do_sync_write+0xbf/0xff <ffffffff8111dd6f>
=> vfs_write+0xac/0xf3 <ffffffff8111e3e5>
=> sys_write+0x4a/0x6e <ffffffff8111e5d4>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> __kmalloc+0xb1/0x10d <ffffffff8110dfdf>
=> drm_malloc_ab+0x3b/0x53 [i915] <ffffffffa007fbd2>
=> i915_gem_do_execbuffer+0x64a/0x1116 [i915] <ffffffffa0080249>
=> i915_gem_execbuffer2+0xcc/0x12e [i915] <ffffffffa0080ffe>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
=> sys_ioctl+0x56/0x7b <ffffffff8112c107>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> dst_alloc+0x48/0xaa <ffffffff813c6672>
=> __ip_route_output_key+0x561/0x764 <ffffffff813ee513>
=> tcp_v4_connect+0x121/0x47d <ffffffff8140becf>
=> inet_stream_connect+0xa8/0x25f <ffffffff81418e24>
=> sys_connect+0x98/0xd6 <ffffffff813b0437>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=2 normal gfp_flags=GFP_KERNEL|GFP_REPEAT|GFP_COMP
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> kmalloc_large_node+0x56/0x95 <ffffffff8146a57d>
=> __kmalloc_node_track_caller+0x31/0x131 <ffffffff8110ff2c>
=> __alloc_skb+0x75/0x133 <ffffffff813b5e4c>
=> sock_alloc_send_pskb+0xb4/0x2d7 <ffffffff813b23aa>
=> sock_alloc_send_skb+0x15/0x17 <ffffffff813b25e2>
=> unix_stream_sendmsg+0x11e/0x2ec <ffffffff8143d237>
=> __sock_sendmsg+0x69/0x76 <ffffffff813af798>
=> sock_aio_write+0xb9/0xc9 <ffffffff813af85e>
=> do_sync_write+0xbf/0xff <ffffffff8111dd6f>
=> vfs_write+0xb3/0xf3 <ffffffff8111e3ec>
=> sys_write+0x4a/0x6e <ffffffff8111e5d4>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=1 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> __nf_conntrack_alloc+0xc9/0x1af <ffffffff813e4533>
=> nf_conntrack_in+0x2f6/0x77a <ffffffff813e490f>
=> ipv4_conntrack_local+0x46/0x4a <ffffffff8142864c>
=> nf_iterate+0x48/0x83 <ffffffff813e1cb8>
=> nf_hook_slow+0x6a/0xe9 <ffffffff813e1d5d>
=> nf_hook_thresh.constprop.8+0x42/0x44 <ffffffff813f42f0>
=> __ip_local_out+0x49/0x4d <ffffffff813f433b>
=> ip_local_out+0x13/0x24 <ffffffff813f4352>
=> ip_queue_xmit+0x2b3/0x30e <ffffffff813f4616>
=> tcp_transmit_skb+0x708/0x739 <ffffffff81407217>
1 instances order=2 atomic gfp_flags=GFP_ATOMIC|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> dst_alloc+0x48/0xaa <ffffffff813c6672>
=> __ip_route_output_key+0x561/0x764 <ffffffff813ee513>
=> ip_route_output_flow+0x2c/0x1ea <ffffffff813ee742>
=> ip4_datagram_connect+0x18d/0x298 <ffffffff8140ede5>
=> inet_dgram_connect+0x65/0x6c <ffffffff81418b9d>
=> sys_connect+0x98/0xd6 <ffffffff813b0437>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=9 normal gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_vma+0xf5/0xfa <ffffffff81107025>
=> do_huge_pmd_wp_page+0x15a/0x637 <ffffffff81114a25>
=> handle_mm_fault+0x169/0x1ce <ffffffff810f176a>
=> do_page_fault+0x358/0x37a <ffffffff814741ae>
=> page_fault+0x25/0x30 <ffffffff81471455>
1 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __page_cache_alloc+0x77/0x7e <ffffffff810d577c>
=> find_or_create_page+0x41/0x8a <ffffffff810d5b54>
=> __getblk+0x137/0x23a <ffffffff81141b55>
=> __breadahead+0x12/0x39 <ffffffff81141c6a>
=> __ext4_get_inode_loc+0x2c4/0x344 <ffffffff8119130b>
=> ext4_iget+0x58/0x64c <ffffffff81192a48>
=> ext4_lookup+0x93/0xf3 <ffffffff8119a79b>
=> d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b>
=> do_lookup+0x23a/0x287 <ffffffff81127dde>
=> link_path_walk+0x239/0x3b8 <ffffffff81128469>
=> do_path_lookup+0x4d/0xf6 <ffffffff8112881b>
=> user_path_at+0x57/0x94 <ffffffff811294e0>
=> sys_faccessat+0xa0/0x162 <ffffffff8111d2e2>
=> sys_access+0x18/0x1a <ffffffff8111d3bc>
2 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> d_alloc+0x26/0x18d <ffffffff8112e4e9>
=> shmem_file_setup+0xb6/0x160 <ffffffff810e70d1>
=> drm_gem_object_init+0x34/0x61 [drm] <ffffffffa0024ffc>
=> i915_gem_alloc_object+0x41/0x111 [i915] <ffffffffa007e4cd>
=> i915_gem_create_ioctl+0x3f/0xcf [i915] <ffffffffa007e5dc>
=> drm_ioctl+0x29e/0x37b [drm] <ffffffffa0023815>
=> do_vfs_ioctl+0x47e/0x4bf <ffffffff8112c070>
=> sys_ioctl+0x56/0x7b <ffffffff8112c107>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
416 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> bio_alloc+0x15/0x24 <ffffffff81144ef5>
=> submit_bh+0x6d/0x105 <ffffffff811409f6>
=> __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
=> block_write_full_page_endio+0x8a/0x97 <ffffffff81143671>
=> block_write_full_page+0x15/0x17 <ffffffff81143693>
=> blkdev_writepage+0x18/0x1a <ffffffff811460b8>
=> __writepage+0x15/0x2e <ffffffff810dca43>
13 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> mpage_readpages+0x99/0x104 <ffffffff81148dae>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
=> filemap_fault+0x17b/0x36c <ffffffff810d6ade>
=> __do_fault+0x55/0x377 <ffffffff810ee88b>
=> handle_pte_fault+0x2a7/0x7a5 <ffffffff810f0f47>
58 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d85>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> do_mpage_readpage+0x34a/0x468 <ffffffff81148b8d>
=> mpage_readpages+0xbe/0x104 <ffffffff81148dd3>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> d_alloc+0x26/0x18d <ffffffff8112e4e9>
=> d_alloc_and_lookup+0x2c/0x6b <ffffffff81126d32>
=> __lookup_hash+0x7f/0x88 <ffffffff81127eaa>
=> lookup_hash+0x19/0x1b <ffffffff81127ecc>
=> do_last+0xcf/0x2df <ffffffff811298fc>
=> do_filp_open+0x2ac/0x60a <ffffffff81129db8>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
=> sys_open+0x20/0x22 <ffffffff8111da0d>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
7727 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
=> mempool_alloc+0x68/0x116 <ffffffff810d70fa>
=> bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> submit_bh+0xe6/0x105 <ffffffff81140a6f>
=> __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
1 instances order=1 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> __split_vma+0x6c/0x21b <ffffffff810f50bd>
=> do_munmap+0x194/0x30b <ffffffff810f57d1>
=> mmap_region+0x9b/0x446 <ffffffff810f645d>
=> do_mmap_pgoff+0x298/0x2f2 <ffffffff810f6aa0>
=> sys_mmap_pgoff+0x115/0x164 <ffffffff810f6c0f>
=> sys_mmap+0x22/0x24 <ffffffff8100da68>
=> system_call_fastpath+0x16/0x1b <ffffffff81009b82>
1 instances order=1 normal gfp_flags=GFP_KERNEL
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> __get_free_pages+0xe/0x4a <ffffffff810d89d6>
=> copy_process+0xe1/0x10c5 <ffffffff810533d4>
=> do_fork+0x104/0x286 <ffffffff810544f7>
=> kernel_thread+0x75/0x77 <ffffffff81010bb6>
=> __call_usermodehelper+0x43/0x76 <ffffffff81069132>
=> process_one_work+0x186/0x298 <ffffffff8106b3ca>
=> worker_thread+0xda/0x15d <ffffffff8106bf29>
=> kthread+0x84/0x8c <ffffffff8106f2df>
=> kernel_thread_helper+0x4/0x10 <ffffffff8100a9a4>
18 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> bvec_alloc_bs+0xae/0xcc <ffffffff81144d85>
=> bio_alloc_bioset+0x75/0xc3 <ffffffff81144e18>
=> clone_bio+0x41/0xdb <ffffffff81382d3d>
=> __split_and_process_bio+0x23d/0x4ea <ffffffff813837d0>
=> dm_request+0x160/0x172 <ffffffff81383bdd>
=> generic_make_request+0x2ae/0x328 <ffffffff8121212f>
=> submit_bio+0xde/0xfd <ffffffff81212287>
=> mpage_readpages+0xf0/0x104 <ffffffff81148e05>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
1 instances order=3 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> ext4_alloc_inode+0x1a/0x111 <ffffffff8119f4bc>
=> alloc_inode+0x1d/0x78 <ffffffff81131809>
=> iget_locked+0x61/0xdd <ffffffff811323be>
=> ext4_iget+0x2c/0x64c <ffffffff81192a1c>
=> ext4_lookup+0x93/0xf3 <ffffffff8119a79b>
=> d_alloc_and_lookup+0x45/0x6b <ffffffff81126d4b>
=> do_lookup+0x23a/0x287 <ffffffff81127dde>
=> link_path_walk+0x14b/0x3b8 <ffffffff8112837b>
=> do_filp_open+0x1eb/0x60a <ffffffff81129cf7>
=> do_sys_open+0x60/0xf2 <ffffffff8111d95b>
1 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> find_or_create_page+0x5a/0x8a <ffffffff810d5b6d>
=> ext4_mb_load_buddy+0x1f7/0x30f <ffffffff811b4bdf>
=> ext4_free_blocks+0x41d/0x6ba <ffffffff811b7a92>
=> ext4_ext_truncate+0x361/0x78a <ffffffff811b1012>
=> ext4_truncate+0xa1/0x4d9 <ffffffff8119598c>
=> ext4_evict_inode+0x13f/0x228 <ffffffff811978e9>
=> evict+0x21/0x89 <ffffffff81131b92>
149 instances order=2 normal gfp_flags=GFP_IO|GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_RECLAIMABLE|
=> __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
=> alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
=> alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
=> new_slab+0x50/0x199 <ffffffff8110dc48>
=> __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
=> kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
=> radix_tree_preload+0x31/0x81 <ffffffff812293bd>
=> add_to_page_cache_locked+0x56/0x118 <ffffffff810d57d9>
=> add_to_page_cache_lru+0x2a/0x58 <ffffffff810d58c5>
=> mpage_readpages+0x99/0x104 <ffffffff81148dae>
=> ext4_readpages+0x1d/0x1f <ffffffff8119174d>
=> __do_page_cache_readahead+0x100/0x177 <ffffffff810dd7d4>
=> ra_submit+0x21/0x25 <ffffffff810ddab1>
=> ondemand_readahead+0x1c9/0x1d8 <ffffffff810ddc7e>
=> page_cache_async_readahead+0x7b/0xa3 <ffffffff810ddd08>
=> generic_file_aio_read+0x2bd/0x5e0 <ffffffff810d6514>
High-order normal allocations: 152966
High-order atomic allocations: 875
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-10 17:05 ` James Bottomley
@ 2011-05-10 17:17 ` Mel Gorman
-1 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-05-10 17:17 UTC (permalink / raw)
To: James Bottomley
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Tue, May 10, 2011 at 12:05:54PM -0500, James Bottomley wrote:
> On Tue, 2011-05-10 at 15:57 +0000, James Bottomley wrote:
> > On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote:
> > > On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
> > > > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> > > > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> > > > > > I really would like to hear if the fix makes a big difference or
> > > > > > if we need to consider forcing SLUB high-order allocations bailing
> > > > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> > > > > > allocate_slab). Even with the fix applied, kswapd might be waking up
> > > > > > less but processes will still be getting stalled in direct compaction
> > > > > > and direct reclaim so it would still be jittery.
> > > > >
> > > > > "the fix" being this
> > > > >
> > > > > https://lkml.org/lkml/2011/3/5/121
> > > > >
> > > >
> > > > Drop this for the moment. It was a long shot at best and there is little
> > > > evidence the problem is in this area.
> > > >
> > > > I'm attaching two patches. The first is the NO_KSWAPD one to stop
> > > > kswapd being woken up by SLUB using speculative high-orders. The second
> > > > one is more drastic and prevents slub entering direct reclaim or
> > > > compaction. It applies on top of patch 1. These are both untested and
> > > > afraid are a bit rushed as well :(
> > >
> > > Preliminary results with both patches applied still show kswapd
> > > periodically going up to 99% but it doesn't stay there, it comes back
> > > down again (and, obviously, the system doesn't hang).
> >
> > This is a second run with the watch highorders.
> >
> > At the end of the run, the system hung temporarily and now comes back
> > with CPU3 spinning in all system time at kswapd shrink_slab
>
> Here's a trace in the same situation with the ftrace stack entries
> bumped to 16 as requested on IRC. There was no hang for this one.
>
Ok, so the bulk of the high-order allocs are coming from
> 140162 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
> => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
> => new_slab+0x50/0x199 <ffffffff8110dc48>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
> => mempool_alloc+0x68/0x116 <ffffffff810d70fa>
> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
> => bio_alloc+0x15/0x24 <ffffffff81144ef5>
> => submit_bh+0x6d/0x105 <ffffffff811409f6>
> => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
> => block_write_full_page_endio+0x8a/0x97 <ffffffff81143671>
> => block_write_full_page+0x15/0x17 <ffffffff81143693>
> => mpage_da_submit_io+0x31a/0x395 <ffffffff811935d8>
> => mpage_da_map_and_submit+0x2ca/0x2e0 <ffffffff81196e88>
>
That at least is in line with the large untar and absolves i915
from being the main cause of trouble. The lack of the hang implies
that SLUB doing high order allocations is stressing the system too
much and needs to be more willing to fall back to order-0 although
it does not adequately explain why it hung as opposed to just being
incredible slow.
I'm also still concerned with the reports of getting stuck in a heavy
loop on the i915 shrinker so will try again reproducing this locally
with a greater focus on something X related happening at the same time.
One thing at a time though, SLUB needs to be less aggressive so I'll
prepare a series in the morning, have another go at generating data
and see what shakes out.
Thanks.
--
Mel Gorman
SUSE Labs
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-10 17:17 ` Mel Gorman
0 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-05-10 17:17 UTC (permalink / raw)
To: James Bottomley
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Tue, May 10, 2011 at 12:05:54PM -0500, James Bottomley wrote:
> On Tue, 2011-05-10 at 15:57 +0000, James Bottomley wrote:
> > On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote:
> > > On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
> > > > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> > > > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> > > > > > I really would like to hear if the fix makes a big difference or
> > > > > > if we need to consider forcing SLUB high-order allocations bailing
> > > > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> > > > > > allocate_slab). Even with the fix applied, kswapd might be waking up
> > > > > > less but processes will still be getting stalled in direct compaction
> > > > > > and direct reclaim so it would still be jittery.
> > > > >
> > > > > "the fix" being this
> > > > >
> > > > > https://lkml.org/lkml/2011/3/5/121
> > > > >
> > > >
> > > > Drop this for the moment. It was a long shot at best and there is little
> > > > evidence the problem is in this area.
> > > >
> > > > I'm attaching two patches. The first is the NO_KSWAPD one to stop
> > > > kswapd being woken up by SLUB using speculative high-orders. The second
> > > > one is more drastic and prevents slub entering direct reclaim or
> > > > compaction. It applies on top of patch 1. These are both untested and
> > > > afraid are a bit rushed as well :(
> > >
> > > Preliminary results with both patches applied still show kswapd
> > > periodically going up to 99% but it doesn't stay there, it comes back
> > > down again (and, obviously, the system doesn't hang).
> >
> > This is a second run with the watch highorders.
> >
> > At the end of the run, the system hung temporarily and now comes back
> > with CPU3 spinning in all system time at kswapd shrink_slab
>
> Here's a trace in the same situation with the ftrace stack entries
> bumped to 16 as requested on IRC. There was no hang for this one.
>
Ok, so the bulk of the high-order allocs are coming from
> 140162 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
> => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
> => alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
> => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
> => new_slab+0x50/0x199 <ffffffff8110dc48>
> => __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
> => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
> => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
> => mempool_alloc+0x68/0x116 <ffffffff810d70fa>
> => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
> => bio_alloc+0x15/0x24 <ffffffff81144ef5>
> => submit_bh+0x6d/0x105 <ffffffff811409f6>
> => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
> => block_write_full_page_endio+0x8a/0x97 <ffffffff81143671>
> => block_write_full_page+0x15/0x17 <ffffffff81143693>
> => mpage_da_submit_io+0x31a/0x395 <ffffffff811935d8>
> => mpage_da_map_and_submit+0x2ca/0x2e0 <ffffffff81196e88>
>
That at least is in line with the large untar and absolves i915
from being the main cause of trouble. The lack of the hang implies
that SLUB doing high order allocations is stressing the system too
much and needs to be more willing to fall back to order-0 although
it does not adequately explain why it hung as opposed to just being
incredible slow.
I'm also still concerned with the reports of getting stuck in a heavy
loop on the i915 shrinker so will try again reproducing this locally
with a greater focus on something X related happening at the same time.
One thing at a time though, SLUB needs to be less aggressive so I'll
prepare a series in the morning, have another go at generating data
and see what shakes out.
Thanks.
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-10 17:17 ` Mel Gorman
@ 2011-05-10 17:29 ` James Bottomley
-1 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-10 17:29 UTC (permalink / raw)
To: Mel Gorman
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Tue, 2011-05-10 at 18:17 +0100, Mel Gorman wrote:
> On Tue, May 10, 2011 at 12:05:54PM -0500, James Bottomley wrote:
> > On Tue, 2011-05-10 at 15:57 +0000, James Bottomley wrote:
> > > On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote:
> > > > On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
> > > > > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> > > > > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> > > > > > > I really would like to hear if the fix makes a big difference or
> > > > > > > if we need to consider forcing SLUB high-order allocations bailing
> > > > > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> > > > > > > allocate_slab). Even with the fix applied, kswapd might be waking up
> > > > > > > less but processes will still be getting stalled in direct compaction
> > > > > > > and direct reclaim so it would still be jittery.
> > > > > >
> > > > > > "the fix" being this
> > > > > >
> > > > > > https://lkml.org/lkml/2011/3/5/121
> > > > > >
> > > > >
> > > > > Drop this for the moment. It was a long shot at best and there is little
> > > > > evidence the problem is in this area.
> > > > >
> > > > > I'm attaching two patches. The first is the NO_KSWAPD one to stop
> > > > > kswapd being woken up by SLUB using speculative high-orders. The second
> > > > > one is more drastic and prevents slub entering direct reclaim or
> > > > > compaction. It applies on top of patch 1. These are both untested and
> > > > > afraid are a bit rushed as well :(
> > > >
> > > > Preliminary results with both patches applied still show kswapd
> > > > periodically going up to 99% but it doesn't stay there, it comes back
> > > > down again (and, obviously, the system doesn't hang).
> > >
> > > This is a second run with the watch highorders.
> > >
> > > At the end of the run, the system hung temporarily and now comes back
> > > with CPU3 spinning in all system time at kswapd shrink_slab
> >
> > Here's a trace in the same situation with the ftrace stack entries
> > bumped to 16 as requested on IRC. There was no hang for this one.
> >
>
> Ok, so the bulk of the high-order allocs are coming from
>
> > 140162 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
> > => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
> > => new_slab+0x50/0x199 <ffffffff8110dc48>
> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
> > => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
> > => mempool_alloc+0x68/0x116 <ffffffff810d70fa>
> > => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
> > => bio_alloc+0x15/0x24 <ffffffff81144ef5>
> > => submit_bh+0x6d/0x105 <ffffffff811409f6>
> > => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
> > => block_write_full_page_endio+0x8a/0x97 <ffffffff81143671>
> > => block_write_full_page+0x15/0x17 <ffffffff81143693>
> > => mpage_da_submit_io+0x31a/0x395 <ffffffff811935d8>
> > => mpage_da_map_and_submit+0x2ca/0x2e0 <ffffffff81196e88>
> >
>
> That at least is in line with the large untar and absolves i915
> from being the main cause of trouble. The lack of the hang implies
> that SLUB doing high order allocations is stressing the system too
> much and needs to be more willing to fall back to order-0 although
> it does not adequately explain why it hung as opposed to just being
> incredible slow.
>
> I'm also still concerned with the reports of getting stuck in a heavy
> loop on the i915 shrinker so will try again reproducing this locally
> with a greater focus on something X related happening at the same time.
>
> One thing at a time though, SLUB needs to be less aggressive so I'll
> prepare a series in the morning, have another go at generating data
> and see what shakes out.
As a final data point, I tried slub_max_order=0. kswapd is much more
quiescent (it calms down to nothing). It shows spikes of up to 70% and
once up to 99% though, but it doesn't go up to 99% and stay there like
it did previously. This is just with top, so it's a bit inaccurate.
James
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-10 17:29 ` James Bottomley
0 siblings, 0 replies; 138+ messages in thread
From: James Bottomley @ 2011-05-10 17:29 UTC (permalink / raw)
To: Mel Gorman
Cc: Mel Gorman, Jan Kara, colin.king, Chris Mason, linux-fsdevel,
linux-mm, linux-kernel, linux-ext4
On Tue, 2011-05-10 at 18:17 +0100, Mel Gorman wrote:
> On Tue, May 10, 2011 at 12:05:54PM -0500, James Bottomley wrote:
> > On Tue, 2011-05-10 at 15:57 +0000, James Bottomley wrote:
> > > On Tue, 2011-05-10 at 10:29 -0500, James Bottomley wrote:
> > > > On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
> > > > > On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> > > > > > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> > > > > > > I really would like to hear if the fix makes a big difference or
> > > > > > > if we need to consider forcing SLUB high-order allocations bailing
> > > > > > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> > > > > > > allocate_slab). Even with the fix applied, kswapd might be waking up
> > > > > > > less but processes will still be getting stalled in direct compaction
> > > > > > > and direct reclaim so it would still be jittery.
> > > > > >
> > > > > > "the fix" being this
> > > > > >
> > > > > > https://lkml.org/lkml/2011/3/5/121
> > > > > >
> > > > >
> > > > > Drop this for the moment. It was a long shot at best and there is little
> > > > > evidence the problem is in this area.
> > > > >
> > > > > I'm attaching two patches. The first is the NO_KSWAPD one to stop
> > > > > kswapd being woken up by SLUB using speculative high-orders. The second
> > > > > one is more drastic and prevents slub entering direct reclaim or
> > > > > compaction. It applies on top of patch 1. These are both untested and
> > > > > afraid are a bit rushed as well :(
> > > >
> > > > Preliminary results with both patches applied still show kswapd
> > > > periodically going up to 99% but it doesn't stay there, it comes back
> > > > down again (and, obviously, the system doesn't hang).
> > >
> > > This is a second run with the watch highorders.
> > >
> > > At the end of the run, the system hung temporarily and now comes back
> > > with CPU3 spinning in all system time at kswapd shrink_slab
> >
> > Here's a trace in the same situation with the ftrace stack entries
> > bumped to 16 as requested on IRC. There was no hang for this one.
> >
>
> Ok, so the bulk of the high-order allocs are coming from
>
> > 140162 instances order=1 normal gfp_flags=GFP_NOWARN|GFP_NORETRY|GFP_COMP|GFP_NOMEMALLOC|
> > => __alloc_pages_nodemask+0x754/0x792 <ffffffff810dc0de>
> > => alloc_pages_current+0xbe/0xd8 <ffffffff81105459>
> > => alloc_slab_page+0x1c/0x4d <ffffffff8110c5fe>
> > => new_slab+0x50/0x199 <ffffffff8110dc48>
> > => __slab_alloc+0x24a/0x328 <ffffffff8146ab86>
> > => kmem_cache_alloc+0x77/0x105 <ffffffff8110e450>
> > => mempool_alloc_slab+0x15/0x17 <ffffffff810d6e85>
> > => mempool_alloc+0x68/0x116 <ffffffff810d70fa>
> > => bio_alloc_bioset+0x35/0xc3 <ffffffff81144dd8>
> > => bio_alloc+0x15/0x24 <ffffffff81144ef5>
> > => submit_bh+0x6d/0x105 <ffffffff811409f6>
> > => __block_write_full_page+0x1e7/0x2d7 <ffffffff81141fac>
> > => block_write_full_page_endio+0x8a/0x97 <ffffffff81143671>
> > => block_write_full_page+0x15/0x17 <ffffffff81143693>
> > => mpage_da_submit_io+0x31a/0x395 <ffffffff811935d8>
> > => mpage_da_map_and_submit+0x2ca/0x2e0 <ffffffff81196e88>
> >
>
> That at least is in line with the large untar and absolves i915
> from being the main cause of trouble. The lack of the hang implies
> that SLUB doing high order allocations is stressing the system too
> much and needs to be more willing to fall back to order-0 although
> it does not adequately explain why it hung as opposed to just being
> incredible slow.
>
> I'm also still concerned with the reports of getting stuck in a heavy
> loop on the i915 shrinker so will try again reproducing this locally
> with a greater focus on something X related happening at the same time.
>
> One thing at a time though, SLUB needs to be less aggressive so I'll
> prepare a series in the morning, have another go at generating data
> and see what shakes out.
As a final data point, I tried slub_max_order=0. kswapd is much more
quiescent (it calms down to nothing). It shows spikes of up to 70% and
once up to 99% though, but it doesn't go up to 99% and stay there like
it did previously. This is just with top, so it's a bit inaccurate.
James
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-10 15:29 ` James Bottomley
(?)
(?)
@ 2011-05-10 21:08 ` Raghavendra D Prabhu
2011-05-11 9:16 ` Mel Gorman
-1 siblings, 1 reply; 138+ messages in thread
From: Raghavendra D Prabhu @ 2011-05-10 21:08 UTC (permalink / raw)
To: James Bottomley
Cc: Mel Gorman, Mel Gorman, Jan Kara, colin.king, Chris Mason,
linux-fsdevel, linux-mm, linux-kernel, linux-ext4
[-- Attachment #1: Type: text/plain, Size: 22814 bytes --]
* On Tue, May 10, 2011 at 10:29:57AM -0500, James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
>On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
>> On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
>> > On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
>> > > I really would like to hear if the fix makes a big difference or
>> > > if we need to consider forcing SLUB high-order allocations bailing
>> > > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
>> > > allocate_slab). Even with the fix applied, kswapd might be waking up
>> > > less but processes will still be getting stalled in direct compaction
>> > > and direct reclaim so it would still be jittery.
>> > "the fix" being this
>> > https://lkml.org/lkml/2011/3/5/121
>> Drop this for the moment. It was a long shot at best and there is little
>> evidence the problem is in this area.
>> I'm attaching two patches. The first is the NO_KSWAPD one to stop
>> kswapd being woken up by SLUB using speculative high-orders. The second
>> one is more drastic and prevents slub entering direct reclaim or
>> compaction. It applies on top of patch 1. These are both untested and
>> afraid are a bit rushed as well :(
>
>Preliminary results with both patches applied still show kswapd
>periodically going up to 99% but it doesn't stay there, it comes back
>down again (and, obviously, the system doesn't hang).
>
>This is sysrq-M from a couple of times when it went up there:
>
>[ 426.736958] SysRq : Show Memory
>[ 426.736974] Mem-Info:
>[ 426.736977] Node 0 DMA per-cpu:
>[ 426.736983] CPU 0: hi: 0, btch: 1 usd: 0
>[ 426.736986] CPU 1: hi: 0, btch: 1 usd: 0
>[ 426.736989] CPU 2: hi: 0, btch: 1 usd: 0
>[ 426.736993] CPU 3: hi: 0, btch: 1 usd: 0
>[ 426.736996] Node 0 DMA32 per-cpu:
>[ 426.737002] CPU 0: hi: 186, btch: 31 usd: 169
>[ 426.737005] CPU 1: hi: 186, btch: 31 usd: 40
>[ 426.737009] CPU 2: hi: 186, btch: 31 usd: 166
>[ 426.737012] CPU 3: hi: 186, btch: 31 usd: 168
>[ 426.737015] Node 0 Normal per-cpu:
>[ 426.737020] CPU 0: hi: 0, btch: 1 usd: 0
>[ 426.737024] CPU 1: hi: 0, btch: 1 usd: 0
>[ 426.737027] CPU 2: hi: 0, btch: 1 usd: 0
>[ 426.737030] CPU 3: hi: 0, btch: 1 usd: 0
>[ 426.737036] active_anon:108658 inactive_anon:37031 isolated_anon:0
>[ 426.737037] active_file:32006 inactive_file:41051 isolated_file:32
>[ 426.737038] unevictable:8 dirty:41202 writeback:204 unstable:0
>[ 426.737039] free:191520 slab_reclaimable:8490 slab_unreclaimable:27477
>[ 426.737039] mapped:9176 shmem:26412 pagetables:5427 bounce:0
>[ 426.737051] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
>[ 426.737064] lowmem_reserve[]: 0 1856 1862 1862
>[ 426.737078] Node 0 DMA32 free:757892kB min:66820kB low:83524kB high:100228kB active_anon:434632kB inactive_anon:148112kB active_file:127784kB inactive_file:156924kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:1901408kB mlocked:32kB dirty:164808kB writeback:816kB mapped:36704kB shmem:105648kB slab_reclaimable:33676kB slab_unreclaimable:108372kB kernel_stack:2304kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
>[ 426.737092] lowmem_reserve[]: 0 0 5 5
>[ 426.737106] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
>[ 426.737119] lowmem_reserve[]: 0 0 0 0
>[ 426.737132] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB
>[ 426.737174] Node 0 DMA32: 945*4kB 585*8kB 6469*16kB 4871*32kB 3189*64kB 1338*128kB 228*256kB 58*512kB 24*1024kB 1*2048kB 0*4096kB = 757884kB
>[ 426.737227] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB
>[ 426.737257] 99595 total pagecache pages
>[ 426.737260] 0 pages in swap cache
>[ 426.737263] Swap cache stats: add 0, delete 0, find 0/0
>[ 426.737266] Free swap = 3768316kB
>[ 426.737268] Total swap = 3768316kB
>[ 426.744603] 525808 pages RAM
>[ 426.744612] 57618 pages reserved
>[ 426.744614] 141551 pages shared
>[ 426.744617] 186065 pages non-shared
>[ 472.301810] SysRq : Show Memory
>[ 472.301826] Mem-Info:
>[ 472.301829] Node 0 DMA per-cpu:
>[ 472.301835] CPU 0: hi: 0, btch: 1 usd: 0
>[ 472.301839] CPU 1: hi: 0, btch: 1 usd: 0
>[ 472.301842] CPU 2: hi: 0, btch: 1 usd: 0
>[ 472.301845] CPU 3: hi: 0, btch: 1 usd: 0
>[ 472.301848] Node 0 DMA32 per-cpu:
>[ 472.301854] CPU 0: hi: 186, btch: 31 usd: 184
>[ 472.301857] CPU 1: hi: 186, btch: 31 usd: 46
>[ 472.301860] CPU 2: hi: 186, btch: 31 usd: 158
>[ 472.301863] CPU 3: hi: 186, btch: 31 usd: 163
>[ 472.301866] Node 0 Normal per-cpu:
>[ 472.301871] CPU 0: hi: 0, btch: 1 usd: 0
>[ 472.301874] CPU 1: hi: 0, btch: 1 usd: 0
>[ 472.301878] CPU 2: hi: 0, btch: 1 usd: 0
>[ 472.301881] CPU 3: hi: 0, btch: 1 usd: 0
>[ 472.301886] active_anon:107673 inactive_anon:37031 isolated_anon:0
>[ 472.301887] active_file:31533 inactive_file:33323 isolated_file:32
>[ 472.301888] unevictable:8 dirty:26256 writeback:6475 unstable:0
>[ 472.301889] free:198742 slab_reclaimable:9347 slab_unreclaimable:28647
>[ 472.301889] mapped:8307 shmem:26412 pagetables:5427 bounce:0
>[ 472.301901] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
>[ 472.301915] lowmem_reserve[]: 0 1856 1862 1862
>[ 472.301928] Node 0 DMA32 free:786780kB min:66820kB low:83524kB high:100228kB active_anon:430692kB inactive_anon:148112kB active_file:125892kB inactive_file:126012kB unevictable:32kB isolated(anon):0kB isolated(file):128kB present:1901408kB mlocked:32kB dirty:105024kB writeback:25900kB mapped:33228kB shmem:105648kB slab_reclaimable:37104kB slab_unreclaimable:113052kB kernel_stack:2288kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3196 all_unreclaimable? no
>[ 472.301943] lowmem_reserve[]: 0 0 5 5
>[ 472.301956] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
>[ 472.301968] lowmem_reserve[]: 0 0 0 0
>[ 472.301982] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB
>[ 472.302015] Node 0 DMA32: 6121*4kB 1912*8kB 5094*16kB 4920*32kB 3168*64kB 1381*128kB 262*256kB 68*512kB 24*1024kB 1*2048kB 0*4096kB = 786756kB
>[ 472.302048] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB
>[ 472.302090] 91312 total pagecache pages
>[ 472.302093] 0 pages in swap cache
>[ 472.302096] Swap cache stats: add 0, delete 0, find 0/0
>[ 472.302099] Free swap = 3768316kB
>[ 472.302102] Total swap = 3768316kB
>[ 472.309521] 525808 pages RAM
>[ 472.309529] 57618 pages reserved
>[ 472.309548] 142496 pages shared
>[ 472.309551] 177182 pages non-shared
>
>I'll finish off this verification, and then re-run with the watch-highorder script running.
>
>James
Hi,
I saw similar lockups while doing a heavy copy routine (basically
copying an entire partition). I don't think this may be related to
ext4, since the source and destination filesystems were btrfs and
xfs.
===============================================================================================================
9 00:04:58 localhost kernel: [39824.230679] INFO: task kswapd0:528 blocked for more than 120 seconds.
May 9 00:04:58 localhost kernel: [39824.230684] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 9 00:04:58 localhost kernel: [39824.230690] kswapd0 D ffff8800bb90d740 0 528 2 0x00000000
May 9 00:04:58 localhost kernel: [39824.230699] ffff88013f6a3750 0000000000000046 0000000000000246 ffff88013b290040
May 9 00:04:58 localhost kernel: [39824.230708] ffff88013b290000 0000000000000001 ffff88013f6ea210 ffff88013f6a3fd8
May 9 00:04:58 localhost kernel: [39824.230716] ffff88013f6a3fd8 ffff88013f6a3fd8 0000000000000004 ffff88013f6ea470
May 9 00:04:58 localhost kernel: [39824.230724] Call Trace:
May 9 00:04:58 localhost kernel: [39824.230738] [<ffffffff81548aa3>] io_schedule+0x73/0xc0
May 9 00:04:58 localhost kernel: [39824.230747] [<ffffffff813343bb>] get_request_wait+0xdb/0x1e0
May 9 00:04:58 localhost kernel: [39824.230757] [<ffffffff810608f0>] ? autoremove_wake_function+0x0/0x40
May 9 00:05:22 localhost kernel: [39824.230766] [<ffffffff81334cb6>] __make_request+0x76/0x560
May 9 00:05:22 localhost kernel: [39824.230774] [<ffffffff810e1518>] ? account_page_writeback+0x28/0x30
May 9 00:05:22 localhost kernel: [39824.230782] [<ffffffff81332bce>] generic_make_request+0x28e/0x560
May 9 00:05:22 localhost kernel: [39824.230790] [<ffffffff81332f25>] submit_bio+0x85/0x110
May 9 00:05:22 localhost kernel: [39824.230797] [<ffffffff811552b2>] ? bio_alloc_bioset+0xb2/0xf0
May 9 00:05:22 localhost kernel: [39824.230806] [<ffffffff8127da17>] xfs_submit_ioend_bio.isra.11+0x57/0x80
May 9 00:05:22 localhost kernel: [39824.230816] [<ffffffff8127dafb>] xfs_submit_ioend+0xbb/0x110
May 9 00:05:22 localhost kernel: [39824.230818] [<ffffffff8127ed47>] xfs_vm_writepage+0x3f7/0x540
May 9 00:05:22 localhost kernel: [39824.230821] [<ffffffff810e90b0>] shrink_page_list+0x650/0x8f0
May 9 00:05:22 localhost kernel: [39824.230823] [<ffffffff810e9758>] shrink_inactive_list+0x178/0x410
May 9 00:05:22 localhost kernel: [39824.230826] [<ffffffff8104e89a>] ? del_timer_sync+0x3a/0x60
May 9 00:05:22 localhost kernel: [39824.230828] [<ffffffff810e9e6e>] shrink_zone+0x47e/0x540
May 9 00:05:22 localhost kernel: [39824.230831] [<ffffffff810e7a25>] ? shrink_slab+0x145/0x180
*May 9 00:05:22 localhost kernel: [39824.230833] [<ffffffff810ea5bd>] kswapd+0x68d/0xa20
*May 9 00:05:22 localhost kernel: [39824.230835] [<ffffffff810e9f30>] ? kswapd+0x0/0xa20
May 9 00:05:22 localhost kernel: [39824.230837] [<ffffffff8105fffc>] kthread+0x8c/0xa0
May 9 00:05:22 localhost kernel: [39824.230841] [<ffffffff81039a7c>] ? schedule_tail+0x4c/0xf0
May 9 00:05:22 localhost kernel: [39824.230843] [<ffffffff81003c94>] kernel_thread_helper+0x4/0x10
May 9 00:05:22 localhost kernel: [39824.230846] [<ffffffff8105ff70>] ? kthread+0x0/0xa0
May 9 00:05:22 localhost kernel: [39824.230848] [<ffffffff81003c90>] ? kernel_thread_helper+0x0/0x10
May 9 00:05:22 localhost kernel: [39824.230853] INFO: task xfsbufd/sda9:3757 blocked for more than 120 seconds.
May 9 00:05:22 localhost kernel: [39824.230854] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 9 00:05:22 localhost kernel: [39824.230856] xfsbufd/sda9 D ffff8800bb84d740 0 3757 2 0x00000000
May 9 00:05:22 localhost kernel: [39824.230858] ffff88013b12fb10 0000000000000046 0000000000000246 ffff88013f6736c0
May 9 00:05:22 localhost kernel: [39824.230861] ffff88013f673680 0000000000000001 ffff88013cec2210 ffff88013b12ffd8
May 9 00:05:22 localhost kernel: [39824.230863] ffff88013b12ffd8 ffff88013b12ffd8 0000000000000001 ffff88013cec2470
May 9 00:05:22 localhost kernel: [39824.230866] Call Trace:
May 9 00:05:22 localhost kernel: [39824.230868] [<ffffffff81548aa3>] io_schedule+0x73/0xc0
May 9 00:05:22 localhost kernel: [39824.230870] [<ffffffff813343bb>] get_request_wait+0xdb/0x1e0
May 9 00:05:22 localhost kernel: [39824.230873] [<ffffffff810608f0>] ? autoremove_wake_function+0x0/0x40
May 9 00:05:22 localhost kernel: [39824.230875] [<ffffffff81334cb6>] __make_request+0x76/0x560
May 9 00:05:22 localhost kernel: [39824.230878] [<ffffffff81037434>] ? resched_best_mask.isra.60+0x114/0x130
May 9 00:05:22 localhost kernel: [39824.230880] [<ffffffff81332bce>] generic_make_request+0x28e/0x560
May 9 00:05:22 localhost kernel: [39824.230883] [<ffffffff8103640c>] ? update_cpu_clock+0x19c/0x3d0
May 9 00:05:22 localhost kernel: [39824.230885] [<ffffffff81332f25>] submit_bio+0x85/0x110
May 9 00:05:22 localhost kernel: [39824.230887] [<ffffffff8115525a>] ? bio_alloc_bioset+0x5a/0xf0
May 9 00:05:22 localhost kernel: [39824.230889] [<ffffffff8128159c>] _xfs_buf_ioapply+0x19c/0x300
May 9 00:05:22 localhost kernel: [39824.230892] [<ffffffff812825cd>] ? xfs_bdstrat_cb+0x5d/0x100
May 9 00:05:22 localhost kernel: [39824.230894] [<ffffffff8128194e>] xfs_buf_iorequest+0x5e/0x140
May 9 00:05:22 localhost kernel: [39824.230896] [<ffffffff812825cd>] xfs_bdstrat_cb+0x5d/0x100
May 9 00:05:22 localhost kernel: [39824.230899] [<ffffffff81282776>] xfsbufd+0x106/0x180
May 9 00:05:22 localhost kernel: [39824.230901] [<ffffffff81282670>] ? xfsbufd+0x0/0x180
May 9 00:05:22 localhost kernel: [39824.230903] [<ffffffff8105fffc>] kthread+0x8c/0xa0
May 9 00:05:22 localhost kernel: [39824.230905] [<ffffffff81039a7c>] ? schedule_tail+0x4c/0xf0
May 9 00:05:22 localhost kernel: [39824.230907] [<ffffffff81003c94>] kernel_thread_helper+0x4/0x10
May 9 00:05:22 localhost kernel: [39824.230909] [<ffffffff8105ff70>] ? kthread+0x0/0xa0
May 9 00:05:22 localhost kernel: [39824.230911] [<ffffffff81003c90>] ? kernel_thread_helper+0x0/0x10
May 9 00:05:22 localhost kernel: [39824.230913] INFO: task xfsbufd/sda11:3769 blocked for more than 120 seconds.
May 9 00:05:22 localhost kernel: [39824.230914] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 9 00:05:22 localhost kernel: [39824.230915] xfsbufd/sda11 D ffff8800bb84d740 0 3769 2 0x00000000
May 9 00:05:22 localhost kernel: [39824.230918] ffff88013b183b10 0000000000000046 0000000000000246 ffff88013f6736c0
May 9 00:05:22 localhost kernel: [39824.230920] ffff88013f673680 0000000000000001 ffff88013b299b40 ffff88013b183fd8
May 9 00:05:22 localhost kernel: [39824.230923] ffff88013b183fd8 ffff88013b183fd8 0000000000000001 ffff88013b299da0
May 9 00:05:22 localhost kernel: [39824.230925] Call Trace:
May 9 00:05:22 localhost kernel: [39824.230928] [<ffffffff81548aa3>] io_schedule+0x73/0xc0
May 9 00:05:22 localhost kernel: [39824.230930] [<ffffffff813343bb>] get_request_wait+0xdb/0x1e0
May 9 00:05:22 localhost kernel: [39824.230932] [<ffffffff810608f0>] ? autoremove_wake_function+0x0/0x40
May 9 00:05:22 localhost kernel: [39824.230935] [<ffffffff81334cb6>] __make_request+0x76/0x560
May 9 00:05:22 localhost kernel: [39824.230938] [<ffffffff8135d7af>] ? __sg_alloc_table+0x7f/0x140
May 9 00:05:22 localhost kernel: [39824.230941] [<ffffffff81332bce>] generic_make_request+0x28e/0x560
May 9 00:05:22 localhost kernel: [39824.230943] [<ffffffff8103640c>] ? update_cpu_clock+0x19c/0x3d0
May 9 00:05:22 localhost kernel: [39824.230946] [<ffffffff81431a60>] ? ata_scsi_rw_xlat+0x0/0x210
May 9 00:05:22 localhost kernel: [39824.230948] [<ffffffff81332f25>] submit_bio+0x85/0x110
May 9 00:05:22 localhost kernel: [39824.230950] [<ffffffff8115525a>] ? bio_alloc_bioset+0x5a/0xf0
May 9 00:05:22 localhost kernel: [39824.230953] [<ffffffff8128159c>] _xfs_buf_ioapply+0x19c/0x300
May 9 00:05:22 localhost kernel: [39824.230955] [<ffffffff812825cd>] ? xfs_bdstrat_cb+0x5d/0x100
May 9 00:05:22 localhost kernel: [39824.230957] [<ffffffff8128194e>] xfs_buf_iorequest+0x5e/0x140
May 9 00:05:22 localhost kernel: [39824.230959] [<ffffffff812825cd>] xfs_bdstrat_cb+0x5d/0x100
May 9 00:05:22 localhost kernel: [39824.230962] [<ffffffff81282776>] xfsbufd+0x106/0x180
May 9 00:05:22 localhost kernel: [39824.230964] [<ffffffff81282670>] ? xfsbufd+0x0/0x180
May 9 00:05:22 localhost kernel: [39824.230966] [<ffffffff8105fffc>] kthread+0x8c/0xa0
May 9 00:05:22 localhost kernel: [39824.230968] [<ffffffff81039a7c>] ? schedule_tail+0x4c/0xf0
May 9 00:05:22 localhost kernel: [39824.230970] [<ffffffff81003c94>] kernel_thread_helper+0x4/0x10
May 9 00:05:22 localhost kernel: [39824.230972] [<ffffffff8105ff70>] ? kthread+0x0/0xa0
May 9 00:05:22 localhost kernel: [39824.230974] [<ffffffff81003c90>] ? kernel_thread_helper+0x0/0x10
May 9 00:05:22 localhost kernel: [39824.230989] INFO: task logrotate:1749 blocked for more than 120 seconds.
May 9 00:05:22 localhost kernel: [39824.230990] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 9 00:05:22 localhost kernel: [39824.230992] logrotate D ffff8800bb90d740 0 1749 1748 0x00000000
May 9 00:05:22 localhost kernel: [39824.230994] ffff88000c6f3788 0000000000000082 ffff88000c6f3738 ffff88013f475890
May 9 00:05:22 localhost kernel: [39824.230997] 0000000000000001 0000000000000001 0000000000000001 ffff88000c6f3fd8
May 9 00:05:22 localhost kernel: [39824.230999] ffff88000c6f3fd8 ffff88000c6f3fd8 0000000000000004 ffff88013f475af0
May 9 00:05:22 localhost kernel: [39824.231002] Call Trace:
May 9 00:05:22 localhost kernel: [39824.231004] [<ffffffff81548aa3>] io_schedule+0x73/0xc0
May 9 00:05:22 localhost kernel: [39824.231006] [<ffffffff813343bb>] get_request_wait+0xdb/0x1e0
May 9 00:05:22 localhost kernel: [39824.231008] [<ffffffff810608f0>] ? autoremove_wake_function+0x0/0x40
May 9 00:05:22 localhost kernel: [39824.231011] [<ffffffff81334cb6>] __make_request+0x76/0x560
May 9 00:05:22 localhost kernel: [39824.231014] [<ffffffff81150af4>] ? __getblk+0x24/0x280
May 9 00:05:22 localhost kernel: [39824.231016] [<ffffffff81332bce>] generic_make_request+0x28e/0x560
May 9 00:05:22 localhost kernel: [39824.231020] [<ffffffff811ae4fc>] ? ext4_mark_iloc_dirty+0x39c/0x5a0
May 9 00:05:22 localhost kernel: [39824.231022] [<ffffffff81332f25>] submit_bio+0x85/0x110
May 9 00:05:22 localhost kernel: [39824.231024] [<ffffffff8115525a>] ? bio_alloc_bioset+0x5a/0xf0
May 9 00:05:22 localhost kernel: [39824.231027] [<ffffffff8114f66b>] submit_bh+0xeb/0x120
May 9 00:05:22 localhost kernel: [39824.231029] [<ffffffff81151720>] __block_write_full_page+0x210/0x390
May 9 00:05:22 localhost kernel: [39824.231031] [<ffffffff810605de>] ? wake_up_bit+0x2e/0x40
May 9 00:05:22 localhost kernel: [39824.231034] [<ffffffff811518a0>] ? end_buffer_async_write+0x0/0x200
May 9 00:05:22 localhost kernel: [39824.231036] [<ffffffff811b00c0>] ? noalloc_get_block_write+0x0/0x30
May 9 00:05:22 localhost kernel: [39824.231038] [<ffffffff811518a0>] ? end_buffer_async_write+0x0/0x200
May 9 00:05:22 localhost kernel: [39824.231041] [<ffffffff81151f65>] block_write_full_page_endio+0xf5/0x140
May 9 00:05:22 localhost kernel: [39824.231043] [<ffffffff811b00c0>] ? noalloc_get_block_write+0x0/0x30
May 9 00:05:22 localhost kernel: [39824.231045] [<ffffffff81151fc5>] block_write_full_page+0x15/0x20
May 9 00:05:22 localhost kernel: [39824.231048] [<ffffffff811add81>] mpage_da_submit_io+0x451/0x510
May 9 00:05:22 localhost kernel: [39824.231050] [<ffffffff811ae4fc>] ? ext4_mark_iloc_dirty+0x39c/0x5a0
May 9 00:05:22 localhost kernel: [39824.231053] [<ffffffff811ae822>] ? ext4_mark_inode_dirty+0x82/0x230
May 9 00:05:22 localhost kernel: [39824.231055] [<ffffffff811b26a6>] mpage_da_map_and_submit+0x1d6/0x410
May 9 00:05:22 localhost kernel: [39824.231059] [<ffffffff811e8983>] ? jbd2_journal_start+0x13/0x20
May 9 00:05:22 localhost kernel: [39824.231061] [<ffffffff811b31b1>] ext4_da_writepages+0x3e1/0x750
May 9 00:05:22 localhost kernel: [39824.231064] [<ffffffff810e2e91>] do_writepages+0x21/0x40
May 9 00:05:22 localhost kernel: [39824.231066] [<ffffffff810da00b>] __filemap_fdatawrite_range+0x5b/0x60
May 9 00:05:22 localhost kernel: [39824.231068] [<ffffffff810da74c>] filemap_flush+0x1c/0x20
May 9 00:05:22 localhost kernel: [39824.231070] [<ffffffff811acafc>] ext4_alloc_da_blocks+0x4c/0xe0
May 9 00:05:22 localhost kernel: [39824.231073] [<ffffffff811a78e9>] ext4_release_file+0x79/0xc0
May 9 00:05:22 localhost kernel: [39824.231075] [<ffffffff81124d7a>] fput+0xda/0x200
May 9 00:05:22 localhost kernel: [39824.231077] [<ffffffff811214e5>] filp_close+0x55/0x80
May 9 00:05:22 localhost kernel: [39824.231079] [<ffffffff811215c6>] sys_close+0xb6/0x120
May 9 00:05:22 localhost kernel: [39824.231082] [<ffffffff81002eeb>] system_call_fastpath+0x16/0x1b
===========================================================================================================================================================
The mv was basically frozen though after sometime it resumed. I also noticed that levels of dirty data were really high (=~ 1G) which I presumed may have caused this.
The kernel is
========================================================
Linux Xye 2.6.39-rc6 #78 SMP PREEMPT Mon May 9 14:38:17 IST 2011 x86_64 Intel(R) Core(TM) i7 CPU Q 740 @ 1.73GHz GenuineIntel GNU/Linux
======================================================
Also, I am not using cgroups or systemd at the moment. Also, this is not a Fedora system. (Arch linux).
This happened only once. After which it has been fine, though I have not done such high throughput copying after that.
---------------------
Raghavendra D Prabhu
[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
2011-05-10 21:08 ` Raghavendra D Prabhu
@ 2011-05-11 9:16 ` Mel Gorman
0 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-05-11 9:16 UTC (permalink / raw)
To: Raghavendra D Prabhu
Cc: James Bottomley, Mel Gorman, Jan Kara, colin.king, Chris Mason,
linux-fsdevel, linux-mm, linux-kernel, linux-ext4
On Wed, May 11, 2011 at 02:38:09AM +0530, Raghavendra D Prabhu wrote:
> * On Tue, May 10, 2011 at 10:29:57AM -0500, James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> >On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
> >>On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> >>> On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> >>> > I really would like to hear if the fix makes a big difference or
> >>> > if we need to consider forcing SLUB high-order allocations bailing
> >>> > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> >>> > allocate_slab). Even with the fix applied, kswapd might be waking up
> >>> > less but processes will still be getting stalled in direct compaction
> >>> > and direct reclaim so it would still be jittery.
>
> >>> "the fix" being this
>
> >>> https://lkml.org/lkml/2011/3/5/121
>
>
> >>Drop this for the moment. It was a long shot at best and there is little
> >>evidence the problem is in this area.
>
> >>I'm attaching two patches. The first is the NO_KSWAPD one to stop
> >>kswapd being woken up by SLUB using speculative high-orders. The second
> >>one is more drastic and prevents slub entering direct reclaim or
> >>compaction. It applies on top of patch 1. These are both untested and
> >>afraid are a bit rushed as well :(
> >
> >Preliminary results with both patches applied still show kswapd
> >periodically going up to 99% but it doesn't stay there, it comes back
> >down again (and, obviously, the system doesn't hang).
> >
> >This is sysrq-M from a couple of times when it went up there:
> >
> >[ 426.736958] SysRq : Show Memory
> >[ 426.736974] Mem-Info:
> >[ 426.736977] Node 0 DMA per-cpu:
> >[ 426.736983] CPU 0: hi: 0, btch: 1 usd: 0
> >[ 426.736986] CPU 1: hi: 0, btch: 1 usd: 0
> >[ 426.736989] CPU 2: hi: 0, btch: 1 usd: 0
> >[ 426.736993] CPU 3: hi: 0, btch: 1 usd: 0
> >[ 426.736996] Node 0 DMA32 per-cpu:
> >[ 426.737002] CPU 0: hi: 186, btch: 31 usd: 169
> >[ 426.737005] CPU 1: hi: 186, btch: 31 usd: 40
> >[ 426.737009] CPU 2: hi: 186, btch: 31 usd: 166
> >[ 426.737012] CPU 3: hi: 186, btch: 31 usd: 168
> >[ 426.737015] Node 0 Normal per-cpu:
> >[ 426.737020] CPU 0: hi: 0, btch: 1 usd: 0
> >[ 426.737024] CPU 1: hi: 0, btch: 1 usd: 0
> >[ 426.737027] CPU 2: hi: 0, btch: 1 usd: 0
> >[ 426.737030] CPU 3: hi: 0, btch: 1 usd: 0
> >[ 426.737036] active_anon:108658 inactive_anon:37031 isolated_anon:0
> >[ 426.737037] active_file:32006 inactive_file:41051 isolated_file:32
> >[ 426.737038] unevictable:8 dirty:41202 writeback:204 unstable:0
> >[ 426.737039] free:191520 slab_reclaimable:8490 slab_unreclaimable:27477
> >[ 426.737039] mapped:9176 shmem:26412 pagetables:5427 bounce:0
> >[ 426.737051] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> >[ 426.737064] lowmem_reserve[]: 0 1856 1862 1862
> >[ 426.737078] Node 0 DMA32 free:757892kB min:66820kB low:83524kB high:100228kB active_anon:434632kB inactive_anon:148112kB active_file:127784kB inactive_file:156924kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:1901408kB mlocked:32kB dirty:164808kB writeback:816kB mapped:36704kB shmem:105648kB slab_reclaimable:33676kB slab_unreclaimable:108372kB kernel_stack:2304kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
> >[ 426.737092] lowmem_reserve[]: 0 0 5 5
> >[ 426.737106] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> >[ 426.737119] lowmem_reserve[]: 0 0 0 0
> >[ 426.737132] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB
> >[ 426.737174] Node 0 DMA32: 945*4kB 585*8kB 6469*16kB 4871*32kB 3189*64kB 1338*128kB 228*256kB 58*512kB 24*1024kB 1*2048kB 0*4096kB = 757884kB
> >[ 426.737227] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB
> >[ 426.737257] 99595 total pagecache pages
> >[ 426.737260] 0 pages in swap cache
> >[ 426.737263] Swap cache stats: add 0, delete 0, find 0/0
> >[ 426.737266] Free swap = 3768316kB
> >[ 426.737268] Total swap = 3768316kB
> >[ 426.744603] 525808 pages RAM
> >[ 426.744612] 57618 pages reserved
> >[ 426.744614] 141551 pages shared
> >[ 426.744617] 186065 pages non-shared
> >[ 472.301810] SysRq : Show Memory
> >[ 472.301826] Mem-Info:
> >[ 472.301829] Node 0 DMA per-cpu:
> >[ 472.301835] CPU 0: hi: 0, btch: 1 usd: 0
> >[ 472.301839] CPU 1: hi: 0, btch: 1 usd: 0
> >[ 472.301842] CPU 2: hi: 0, btch: 1 usd: 0
> >[ 472.301845] CPU 3: hi: 0, btch: 1 usd: 0
> >[ 472.301848] Node 0 DMA32 per-cpu:
> >[ 472.301854] CPU 0: hi: 186, btch: 31 usd: 184
> >[ 472.301857] CPU 1: hi: 186, btch: 31 usd: 46
> >[ 472.301860] CPU 2: hi: 186, btch: 31 usd: 158
> >[ 472.301863] CPU 3: hi: 186, btch: 31 usd: 163
> >[ 472.301866] Node 0 Normal per-cpu:
> >[ 472.301871] CPU 0: hi: 0, btch: 1 usd: 0
> >[ 472.301874] CPU 1: hi: 0, btch: 1 usd: 0
> >[ 472.301878] CPU 2: hi: 0, btch: 1 usd: 0
> >[ 472.301881] CPU 3: hi: 0, btch: 1 usd: 0
> >[ 472.301886] active_anon:107673 inactive_anon:37031 isolated_anon:0
> >[ 472.301887] active_file:31533 inactive_file:33323 isolated_file:32
> >[ 472.301888] unevictable:8 dirty:26256 writeback:6475 unstable:0
> >[ 472.301889] free:198742 slab_reclaimable:9347 slab_unreclaimable:28647
> >[ 472.301889] mapped:8307 shmem:26412 pagetables:5427 bounce:0
> >[ 472.301901] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> >[ 472.301915] lowmem_reserve[]: 0 1856 1862 1862
> >[ 472.301928] Node 0 DMA32 free:786780kB min:66820kB low:83524kB high:100228kB active_anon:430692kB inactive_anon:148112kB active_file:125892kB inactive_file:126012kB unevictable:32kB isolated(anon):0kB isolated(file):128kB present:1901408kB mlocked:32kB dirty:105024kB writeback:25900kB mapped:33228kB shmem:105648kB slab_reclaimable:37104kB slab_unreclaimable:113052kB kernel_stack:2288kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3196 all_unreclaimable? no
> >[ 472.301943] lowmem_reserve[]: 0 0 5 5
> >[ 472.301956] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> >[ 472.301968] lowmem_reserve[]: 0 0 0 0
> >[ 472.301982] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB
> >[ 472.302015] Node 0 DMA32: 6121*4kB 1912*8kB 5094*16kB 4920*32kB 3168*64kB 1381*128kB 262*256kB 68*512kB 24*1024kB 1*2048kB 0*4096kB = 786756kB
> >[ 472.302048] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB
> >[ 472.302090] 91312 total pagecache pages
> >[ 472.302093] 0 pages in swap cache
> >[ 472.302096] Swap cache stats: add 0, delete 0, find 0/0
> >[ 472.302099] Free swap = 3768316kB
> >[ 472.302102] Total swap = 3768316kB
> >[ 472.309521] 525808 pages RAM
> >[ 472.309529] 57618 pages reserved
> >[ 472.309548] 142496 pages shared
> >[ 472.309551] 177182 pages non-shared
> >
> >I'll finish off this verification, and then re-run with the watch-highorder script running.
> >
> >James
> Hi,
>
> I saw similar lockups while doing a heavy copy routine (basically
> copying an entire partition). I don't think this may be related to
> ext4, since the source and destination filesystems were btrfs and
> xfs.
>
If you have SLUB configured, can you try again with the same two patches
I attached earlier in the thread please?
> <SNIP>
>
> The mv was basically frozen though after sometime it resumed. I also noticed that levels of dirty data were really high (=~ 1G) which I presumed may have caused this.
>
Is 1G higher than what /proc/sys/vm/dirty_ratio should allow?
--
Mel Gorman
SUSE Labs
^ permalink raw reply [flat|nested] 138+ messages in thread
* Re: [BUG] fatal hang untarring 90GB file, possibly writeback related.
@ 2011-05-11 9:16 ` Mel Gorman
0 siblings, 0 replies; 138+ messages in thread
From: Mel Gorman @ 2011-05-11 9:16 UTC (permalink / raw)
To: Raghavendra D Prabhu
Cc: James Bottomley, Mel Gorman, Jan Kara, colin.king, Chris Mason,
linux-fsdevel, linux-mm, linux-kernel, linux-ext4
On Wed, May 11, 2011 at 02:38:09AM +0530, Raghavendra D Prabhu wrote:
> * On Tue, May 10, 2011 at 10:29:57AM -0500, James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> >On Tue, 2011-05-10 at 15:35 +0100, Mel Gorman wrote:
> >>On Tue, May 10, 2011 at 09:01:04AM -0500, James Bottomley wrote:
> >>> On Tue, 2011-05-10 at 11:21 +0100, Mel Gorman wrote:
> >>> > I really would like to hear if the fix makes a big difference or
> >>> > if we need to consider forcing SLUB high-order allocations bailing
> >>> > at the first sign of trouble (e.g. by masking out __GFP_WAIT in
> >>> > allocate_slab). Even with the fix applied, kswapd might be waking up
> >>> > less but processes will still be getting stalled in direct compaction
> >>> > and direct reclaim so it would still be jittery.
>
> >>> "the fix" being this
>
> >>> https://lkml.org/lkml/2011/3/5/121
>
>
> >>Drop this for the moment. It was a long shot at best and there is little
> >>evidence the problem is in this area.
>
> >>I'm attaching two patches. The first is the NO_KSWAPD one to stop
> >>kswapd being woken up by SLUB using speculative high-orders. The second
> >>one is more drastic and prevents slub entering direct reclaim or
> >>compaction. It applies on top of patch 1. These are both untested and
> >>afraid are a bit rushed as well :(
> >
> >Preliminary results with both patches applied still show kswapd
> >periodically going up to 99% but it doesn't stay there, it comes back
> >down again (and, obviously, the system doesn't hang).
> >
> >This is sysrq-M from a couple of times when it went up there:
> >
> >[ 426.736958] SysRq : Show Memory
> >[ 426.736974] Mem-Info:
> >[ 426.736977] Node 0 DMA per-cpu:
> >[ 426.736983] CPU 0: hi: 0, btch: 1 usd: 0
> >[ 426.736986] CPU 1: hi: 0, btch: 1 usd: 0
> >[ 426.736989] CPU 2: hi: 0, btch: 1 usd: 0
> >[ 426.736993] CPU 3: hi: 0, btch: 1 usd: 0
> >[ 426.736996] Node 0 DMA32 per-cpu:
> >[ 426.737002] CPU 0: hi: 186, btch: 31 usd: 169
> >[ 426.737005] CPU 1: hi: 186, btch: 31 usd: 40
> >[ 426.737009] CPU 2: hi: 186, btch: 31 usd: 166
> >[ 426.737012] CPU 3: hi: 186, btch: 31 usd: 168
> >[ 426.737015] Node 0 Normal per-cpu:
> >[ 426.737020] CPU 0: hi: 0, btch: 1 usd: 0
> >[ 426.737024] CPU 1: hi: 0, btch: 1 usd: 0
> >[ 426.737027] CPU 2: hi: 0, btch: 1 usd: 0
> >[ 426.737030] CPU 3: hi: 0, btch: 1 usd: 0
> >[ 426.737036] active_anon:108658 inactive_anon:37031 isolated_anon:0
> >[ 426.737037] active_file:32006 inactive_file:41051 isolated_file:32
> >[ 426.737038] unevictable:8 dirty:41202 writeback:204 unstable:0
> >[ 426.737039] free:191520 slab_reclaimable:8490 slab_unreclaimable:27477
> >[ 426.737039] mapped:9176 shmem:26412 pagetables:5427 bounce:0
> >[ 426.737051] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> >[ 426.737064] lowmem_reserve[]: 0 1856 1862 1862
> >[ 426.737078] Node 0 DMA32 free:757892kB min:66820kB low:83524kB high:100228kB active_anon:434632kB inactive_anon:148112kB active_file:127784kB inactive_file:156924kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:1901408kB mlocked:32kB dirty:164808kB writeback:816kB mapped:36704kB shmem:105648kB slab_reclaimable:33676kB slab_unreclaimable:108372kB kernel_stack:2304kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no
> >[ 426.737092] lowmem_reserve[]: 0 0 5 5
> >[ 426.737106] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> >[ 426.737119] lowmem_reserve[]: 0 0 0 0
> >[ 426.737132] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB
> >[ 426.737174] Node 0 DMA32: 945*4kB 585*8kB 6469*16kB 4871*32kB 3189*64kB 1338*128kB 228*256kB 58*512kB 24*1024kB 1*2048kB 0*4096kB = 757884kB
> >[ 426.737227] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB
> >[ 426.737257] 99595 total pagecache pages
> >[ 426.737260] 0 pages in swap cache
> >[ 426.737263] Swap cache stats: add 0, delete 0, find 0/0
> >[ 426.737266] Free swap = 3768316kB
> >[ 426.737268] Total swap = 3768316kB
> >[ 426.744603] 525808 pages RAM
> >[ 426.744612] 57618 pages reserved
> >[ 426.744614] 141551 pages shared
> >[ 426.744617] 186065 pages non-shared
> >[ 472.301810] SysRq : Show Memory
> >[ 472.301826] Mem-Info:
> >[ 472.301829] Node 0 DMA per-cpu:
> >[ 472.301835] CPU 0: hi: 0, btch: 1 usd: 0
> >[ 472.301839] CPU 1: hi: 0, btch: 1 usd: 0
> >[ 472.301842] CPU 2: hi: 0, btch: 1 usd: 0
> >[ 472.301845] CPU 3: hi: 0, btch: 1 usd: 0
> >[ 472.301848] Node 0 DMA32 per-cpu:
> >[ 472.301854] CPU 0: hi: 186, btch: 31 usd: 184
> >[ 472.301857] CPU 1: hi: 186, btch: 31 usd: 46
> >[ 472.301860] CPU 2: hi: 186, btch: 31 usd: 158
> >[ 472.301863] CPU 3: hi: 186, btch: 31 usd: 163
> >[ 472.301866] Node 0 Normal per-cpu:
> >[ 472.301871] CPU 0: hi: 0, btch: 1 usd: 0
> >[ 472.301874] CPU 1: hi: 0, btch: 1 usd: 0
> >[ 472.301878] CPU 2: hi: 0, btch: 1 usd: 0
> >[ 472.301881] CPU 3: hi: 0, btch: 1 usd: 0
> >[ 472.301886] active_anon:107673 inactive_anon:37031 isolated_anon:0
> >[ 472.301887] active_file:31533 inactive_file:33323 isolated_file:32
> >[ 472.301888] unevictable:8 dirty:26256 writeback:6475 unstable:0
> >[ 472.301889] free:198742 slab_reclaimable:9347 slab_unreclaimable:28647
> >[ 472.301889] mapped:8307 shmem:26412 pagetables:5427 bounce:0
> >[ 472.301901] Node 0 DMA free:8140kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:12kB active_file:240kB inactive_file:7280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:180kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> >[ 472.301915] lowmem_reserve[]: 0 1856 1862 1862
> >[ 472.301928] Node 0 DMA32 free:786780kB min:66820kB low:83524kB high:100228kB active_anon:430692kB inactive_anon:148112kB active_file:125892kB inactive_file:126012kB unevictable:32kB isolated(anon):0kB isolated(file):128kB present:1901408kB mlocked:32kB dirty:105024kB writeback:25900kB mapped:33228kB shmem:105648kB slab_reclaimable:37104kB slab_unreclaimable:113052kB kernel_stack:2288kB pagetables:21708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3196 all_unreclaimable? no
> >[ 472.301943] lowmem_reserve[]: 0 0 5 5
> >[ 472.301956] Node 0 Normal free:48kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:104kB slab_unreclaimable:1520kB kernel_stack:176kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> >[ 472.301968] lowmem_reserve[]: 0 0 0 0
> >[ 472.301982] Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8140kB
> >[ 472.302015] Node 0 DMA32: 6121*4kB 1912*8kB 5094*16kB 4920*32kB 3168*64kB 1381*128kB 262*256kB 68*512kB 24*1024kB 1*2048kB 0*4096kB = 786756kB
> >[ 472.302048] Node 0 Normal: 2*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48kB
> >[ 472.302090] 91312 total pagecache pages
> >[ 472.302093] 0 pages in swap cache
> >[ 472.302096] Swap cache stats: add 0, delete 0, find 0/0
> >[ 472.302099] Free swap = 3768316kB
> >[ 472.302102] Total swap = 3768316kB
> >[ 472.309521] 525808 pages RAM
> >[ 472.309529] 57618 pages reserved
> >[ 472.309548] 142496 pages shared
> >[ 472.309551] 177182 pages non-shared
> >
> >I'll finish off this verification, and then re-run with the watch-highorder script running.
> >
> >James
> Hi,
>
> I saw similar lockups while doing a heavy copy routine (basically
> copying an entire partition). I don't think this may be related to
> ext4, since the source and destination filesystems were btrfs and
> xfs.
>
If you have SLUB configured, can you try again with the same two patches
I attached earlier in the thread please?
> <SNIP>
>
> The mv was basically frozen though after sometime it resumed. I also noticed that levels of dirty data were really high (=~ 1G) which I presumed may have caused this.
>
Is 1G higher than what /proc/sys/vm/dirty_ratio should allow?
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 138+ messages in thread
end of thread, other threads:[~2011-05-11 16:47 UTC | newest]
Thread overview: 138+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-04-27 16:09 [BUG] fatal hang untarring 90GB file, possibly writeback related James Bottomley
2011-04-27 16:09 ` James Bottomley
2011-04-27 16:33 ` Chris Mason
2011-04-27 16:33 ` Chris Mason
2011-04-27 16:50 ` James Bottomley
2011-04-27 16:50 ` James Bottomley
2011-04-27 16:50 ` James Bottomley
2011-04-27 16:54 ` Chris Mason
2011-04-27 16:54 ` Chris Mason
2011-04-27 17:21 ` James Bottomley
2011-04-27 17:21 ` James Bottomley
2011-04-27 17:21 ` James Bottomley
2011-04-27 17:34 ` Chris Mason
2011-04-27 17:34 ` Chris Mason
2011-04-27 17:50 ` James Bottomley
2011-04-27 17:50 ` James Bottomley
2011-04-27 18:25 ` Colin Ian King
2011-04-27 18:25 ` Colin Ian King
2011-04-28 15:57 ` James Bottomley
2011-04-28 15:57 ` James Bottomley
2011-04-27 20:05 ` James Bottomley
2011-04-27 20:05 ` James Bottomley
2011-04-28 11:36 ` Colin Ian King
2011-04-28 11:36 ` Colin Ian King
2011-04-28 12:29 ` Chris Mason
2011-04-28 12:29 ` Chris Mason
2011-04-28 13:42 ` Colin Ian King
2011-04-28 13:42 ` Colin Ian King
2011-04-28 13:45 ` Chris Mason
2011-04-28 13:45 ` Chris Mason
2011-04-28 14:01 ` Colin Ian King
2011-04-28 14:04 ` Chris Mason
2011-04-28 14:04 ` Chris Mason
2011-04-28 15:23 ` Colin Ian King
2011-04-28 14:25 ` Jan Kara
2011-04-28 14:25 ` Jan Kara
2011-04-28 14:33 ` Jan Kara
2011-04-28 14:33 ` Jan Kara
2011-04-28 14:58 ` Colin Ian King
2011-04-28 22:40 ` Jan Kara
2011-04-28 22:40 ` Jan Kara
2011-04-28 22:44 ` James Bottomley
2011-04-28 22:44 ` James Bottomley
2011-05-03 18:55 ` Colin Ian King
2011-05-03 18:55 ` Colin Ian King
2011-04-28 16:11 ` James Bottomley
2011-04-28 16:11 ` James Bottomley
2011-04-28 14:49 ` James Bottomley
2011-04-28 14:49 ` James Bottomley
2011-04-28 13:52 ` Jan Kara
2011-04-28 13:52 ` Jan Kara
2011-04-28 14:07 ` Mel Gorman
2011-04-28 14:07 ` Mel Gorman
2011-04-28 14:25 ` James Bottomley
2011-04-28 14:25 ` James Bottomley
2011-04-28 15:08 ` Mel Gorman
2011-04-28 15:08 ` Mel Gorman
2011-04-28 16:01 ` James Bottomley
2011-04-28 16:01 ` James Bottomley
2011-04-28 16:50 ` James Bottomley
2011-04-28 16:50 ` James Bottomley
2011-04-28 16:56 ` James Bottomley
2011-04-28 16:56 ` James Bottomley
2011-04-28 17:18 ` Mel Gorman
2011-04-28 17:18 ` Mel Gorman
2011-04-28 18:30 ` James Bottomley
2011-04-28 18:30 ` James Bottomley
2011-04-28 19:21 ` Mel Gorman
2011-04-28 19:21 ` Mel Gorman
2011-04-28 19:59 ` James Bottomley
2011-04-28 19:59 ` James Bottomley
2011-04-28 20:27 ` Mel Gorman
2011-04-28 20:27 ` Mel Gorman
2011-04-29 15:02 ` James Bottomley
2011-04-29 15:02 ` James Bottomley
2011-04-28 21:12 ` James Bottomley
2011-04-28 21:12 ` James Bottomley
2011-04-28 22:43 ` James Bottomley
2011-04-28 22:43 ` James Bottomley
2011-05-03 9:13 ` Mel Gorman
2011-05-03 9:13 ` Mel Gorman
2011-05-03 14:13 ` James Bottomley
2011-05-03 14:13 ` James Bottomley
2011-05-03 14:22 ` James Bottomley
2011-05-06 7:42 ` Mel Gorman
2011-05-06 7:42 ` Mel Gorman
2011-05-06 8:07 ` Mel Gorman
2011-05-09 18:16 ` James Bottomley
2011-05-09 18:16 ` James Bottomley
2011-05-10 10:21 ` Mel Gorman
2011-05-10 10:21 ` Mel Gorman
2011-05-10 10:33 ` Pekka Enberg
2011-05-10 10:33 ` Pekka Enberg
2011-05-10 14:01 ` James Bottomley
2011-05-10 14:01 ` James Bottomley
2011-05-10 14:35 ` Mel Gorman
2011-05-10 14:35 ` Mel Gorman
2011-05-10 15:29 ` James Bottomley
2011-05-10 15:29 ` James Bottomley
2011-05-10 15:57 ` James Bottomley
2011-05-10 15:57 ` James Bottomley
2011-05-10 17:05 ` James Bottomley
2011-05-10 17:05 ` James Bottomley
2011-05-10 17:17 ` Mel Gorman
2011-05-10 17:17 ` Mel Gorman
2011-05-10 17:29 ` James Bottomley
2011-05-10 17:29 ` James Bottomley
2011-05-10 21:08 ` Raghavendra D Prabhu
2011-05-11 9:16 ` Mel Gorman
2011-05-11 9:16 ` Mel Gorman
2011-05-06 11:42 ` Mel Gorman
2011-05-06 11:42 ` Mel Gorman
2011-05-06 15:44 ` Mel Gorman
2011-05-06 15:44 ` Mel Gorman
2011-05-06 19:14 ` James Bottomley
2011-05-06 19:14 ` James Bottomley
2011-05-06 19:37 ` Mel Gorman
2011-05-06 19:37 ` Mel Gorman
2011-05-10 5:37 ` Colin Ian King
2011-05-10 5:37 ` Colin Ian King
2011-05-10 5:37 ` Colin Ian King
2011-05-06 15:58 ` James Bottomley
2011-05-03 9:54 ` Colin Ian King
2011-05-03 9:54 ` Colin Ian King
2011-04-28 17:10 ` Colin Ian King
2011-04-28 17:10 ` Colin Ian King
2011-04-28 0:37 ` Dave Chinner
2011-04-28 0:37 ` Dave Chinner
2011-04-29 10:23 ` Sedat Dilek
2011-04-29 10:23 ` Sedat Dilek
2011-04-29 15:37 ` James Bottomley
2011-04-29 15:37 ` James Bottomley
2011-04-29 16:31 ` James Bottomley
2011-04-29 16:31 ` James Bottomley
2011-04-29 18:02 ` James Bottomley
2011-04-29 18:02 ` James Bottomley
2011-05-02 20:04 ` James Bottomley
2011-05-02 20:04 ` James Bottomley
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.